ウェブアプリケーションフレームワークをベンチマークしてみた

なんとなくやってみます。ベンチマークツールとウェブアプリケーションを同一ホストで実行しているし、結果しか掲載してないので、結構イマイチかもしれませんが参考程度に....

# 環境

OS: Mac OS X Yosemite(10.10.1)
CPU: Intel Core i7 (3GHz)
Memory: 16GB (8GB DIMM(DDR3/1600MHz) * 2)
Storage: SSD

# 実行/計測方法

apache bench を使う。

```
$ ab -n 10000 -c 100 -rk "http://127.0.0.1:8888/"
```

計測に使用したアプリケーションは "Benchmarking..." という文字列を返すだけのもの。
source codeはこちら。
https://github.com/TakesxiSximada/benchmarks

# 結果

## rails + unicorn (workers=8) 【Ruby】

### バージョン

ruby: 2.1.6
rails: 4.2.1

### 結果

#### 1回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   12.624 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    792.13 [#/sec] (mean)
Time per request:       126.242 [ms] (mean)
Time per request:       1.262 [ms] (mean, across all concurrent requests)
Transfer rate:          317.16 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   5.4      0     540
Processing:     7  125  58.5    113     649
Waiting:        7  125  58.2    113     648
Total:         10  126  58.7    113     649

Percentage of the requests served within a certain time (ms)
  50%    113
  66%    119
  75%    124
  80%    127
  90%    140
  95%    156
  98%    328
  99%    541
 100%    649 (longest request)
```

#### 2回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   14.515 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    688.95 [#/sec] (mean)
Time per request:       145.149 [ms] (mean)
Time per request:       1.451 [ms] (mean, across all concurrent requests)
Transfer rate:          275.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      21
Processing:     8  144  38.2    132     373
Waiting:        8  144  38.1    132     372
Total:         10  144  38.1    132     373

```

#### 3回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   12.725 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      4100000 bytes
HTML transferred:       150000 bytes
Requests per second:    785.83 [#/sec] (mean)
Time per request:       127.253 [ms] (mean)
Time per request:       1.273 [ms] (mean, across all concurrent requests)
Transfer rate:          314.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       5
Processing:     7  126  33.4    120     514
Waiting:        7  126  33.1    120     514
Total:         11  127  33.3    120     514
```

## revel 【Go】

### バージョン

Go: 1.4
revel: 0.12.0

### 結果

#### 1回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   5.768 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    1733.65 [#/sec] (mean)
Time per request:       57.682 [ms] (mean)
Time per request:       0.577 [ms] (mean, across all concurrent requests)
Transfer rate:          3895.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     4   58 222.0     33    2275
Waiting:        4   58 222.0     33    2275
Total:          4   58 222.2     33    2276
```

#### 2回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   4.821 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    2074.45 [#/sec] (mean)
Time per request:       48.205 [ms] (mean)
Time per request:       0.482 [ms] (mean, across all concurrent requests)
Transfer rate:          4661.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     6   48 147.7     32    1544
Waiting:        6   48 147.7     32    1544
Total:          6   48 147.9     32    1545
```

#### 3回目

```
Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        2030 bytes

Concurrency Level:      100
Time taken for tests:   5.442 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      23010000 bytes
HTML transferred:       20300000 bytes
Requests per second:    1837.60 [#/sec] (mean)
Time per request:       54.419 [ms] (mean)
Time per request:       0.544 [ms] (mean, across all concurrent requests)
Transfer rate:          4129.22 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      13
Processing:     4   54 195.3     32    2013
Waiting:        4   54 195.3     32    2013
Total:          4   54 196.0     32    2020
```

## Tornado 【Python】

### バージョン

Python: 3.4.2
Tornado: 4.1.dev1

### 結果

#### 1回目

```
Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   5.214 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    1917.78 [#/sec] (mean)
Time per request:       52.144 [ms] (mean)
Time per request:       0.521 [ms] (mean, across all concurrent requests)
Transfer rate:          436.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.1      0      23
Processing:     1   37  23.6     33     315
Waiting:        1   37  23.6     33     315
Total:          1   37  23.7     33     315
```

#### 2回目

```
Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   4.846 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    2063.60 [#/sec] (mean)
Time per request:       48.459 [ms] (mean)
Time per request:       0.485 [ms] (mean, across all concurrent requests)
Transfer rate:          469.55 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     1   48   5.6     47     166
Waiting:        1   48   5.6     47     166
Total:          4   48   5.6     47     169
```

#### 3回目

```
Server Software:        TornadoServer/4.1.dev1
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   5.551 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      2330000 bytes
HTML transferred:       110000 bytes
Requests per second:    1801.52 [#/sec] (mean)
Time per request:       55.509 [ms] (mean)
Time per request:       0.555 [ms] (mean, across all concurrent requests)
Transfer rate:          409.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   55  19.6     48     217
Waiting:        2   55  19.6     48     217
Total:          4   55  19.6     48     217
```


## Pyramid + Gunicorn (workers=8) 【Python】

### バージョン

Python: 3.4.2
Pyramid: 1.5.7
Gunicorn: 19.3.0


### 結果

#### 1回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   9.700 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    1030.92 [#/sec] (mean)
Time per request:       97.000 [ms] (mean)
Time per request:       0.970 [ms] (mean, across all concurrent requests)
Transfer rate:          176.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:    32   96  13.8     92     183
Waiting:       32   95  13.5     92     182
Total:         35   97  13.7     92     183
```

#### 2回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   24.306 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    411.42 [#/sec] (mean)
Time per request:       243.063 [ms] (mean)
Time per request:       2.431 [ms] (mean, across all concurrent requests)
Transfer rate:          70.31 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  134 1334.6      0   13421
Processing:     2  108  51.1     96     403
Waiting:        2  107  50.5     95     403
Total:          2  243 1329.2     97   13503
```

#### 3回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   9.781 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1750000 bytes
HTML transferred:       150000 bytes
Requests per second:    1022.40 [#/sec] (mean)
Time per request:       97.809 [ms] (mean)
Time per request:       0.978 [ms] (mean, across all concurrent requests)
Transfer rate:          174.73 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0      35
Processing:     5   96  21.3     91     254
Waiting:        5   95  21.1     90     254
Total:          8   96  21.1     91     255
```


## Django + Gunicorn (workers=8) 【Python】

### バージョン

Python: 3.4.2
Django: 1.8.2
Gunicorn: 19.3.0

### 結果

#### 1回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   14.071 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    710.69 [#/sec] (mean)
Time per request:       140.708 [ms] (mean)
Time per request:       1.407 [ms] (mean, across all concurrent requests)
Transfer rate:          127.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:    58  140  16.5    138     221
Waiting:       58  139  16.4    136     219
Total:         62  140  16.5    138     221
```

#### 2回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   17.237 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    580.14 [#/sec] (mean)
Time per request:       172.373 [ms] (mean)
Time per request:       1.724 [ms] (mean, across all concurrent requests)
Transfer rate:          104.24 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   38 376.3      0    3890
Processing:     2  134  30.2    134     305
Waiting:        1  132  30.2    133     304
Total:          2  172 370.8    134    3905
```

#### 3回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   24.894 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1840000 bytes
HTML transferred:       150000 bytes
Requests per second:    401.70 [#/sec] (mean)
Time per request:       248.942 [ms] (mean)
Time per request:       2.489 [ms] (mean, across all concurrent requests)
Transfer rate:          72.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   42 416.2      0    4449
Processing:     1  206  82.0    201     683
Waiting:        1  204  81.4    199     675
Total:          2  248 416.8    204    4565

```


## Bottle + Gunicorn (workers=8) 【Python】

### バージョン

Python: 3.4.2
Bottle: 0.12.8
Gunicorn: 19.3.0

### 結果

#### 1回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   5.422 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1844.34 [#/sec] (mean)
Time per request:       54.220 [ms] (mean)
Time per request:       0.542 [ms] (mean, across all concurrent requests)
Transfer rate:          324.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      11
Processing:     2   54  46.9     51    1089
Waiting:        1   53  46.9     51    1089
Total:          4   54  47.1     51    1091

```

#### 2回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   5.615 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1780.82 [#/sec] (mean)
Time per request:       56.154 [ms] (mean)
Time per request:       0.562 [ms] (mean, across all concurrent requests)
Transfer rate:          313.03 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     2   56  52.5     52    1246
Waiting:        1   55  52.5     52    1246
Total:          4   56  52.6     52    1248
```

#### 3回目

```
Server Software:        gunicorn/19.3.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      100
Time taken for tests:   6.429 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1800000 bytes
HTML transferred:       150000 bytes
Requests per second:    1555.50 [#/sec] (mean)
Time per request:       64.288 [ms] (mean)
Time per request:       0.643 [ms] (mean, across all concurrent requests)
Transfer rate:          273.43 [Kbytes/sec] received
```

## CherryPy 【Python】

CherryPyは -c が100だとerrorになってしまうので計測不能。
とりあえず-c 20で実施した。ただしそれでもいくつかのrequestはerrorしている。

### バージョン

Python: 3.4.2
CherryPy: 3.7.0

### 結果

#### 1回目

```
Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   19.241 seconds
Complete requests:      10000
Failed requests:        28
   (Connect: 3, Receive: 11, Length: 14, Exceptions: 0)
Keep-Alive requests:    9984
Total transferred:      1777152 bytes
HTML transferred:       149760 bytes
Requests per second:    519.72 [#/sec] (mean)
Time per request:       38.483 [ms] (mean)
Time per request:       1.924 [ms] (mean, across all concurrent requests)
Transfer rate:          90.20 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   4.6      0     125
Processing:     0   20 399.2      8   19115
Waiting:        0   14 330.9      7   19114
Total:          1   20 400.4      8   19239
```

#### 2回目

```
Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   19.076 seconds
Complete requests:      10000
Failed requests:        19
   (Connect: 1, Receive: 5, Length: 13, Exceptions: 0)
Keep-Alive requests:    9986
Total transferred:      1777671 bytes
HTML transferred:       149790 bytes
Requests per second:    524.23 [#/sec] (mean)
Time per request:       38.151 [ms] (mean)
Time per request:       1.908 [ms] (mean, across all concurrent requests)
Transfer rate:          91.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.5      0     126
Processing:     0   18 299.8      8   10006
Waiting:        0    8   2.7      7      39
Total:          1   18 299.8      8   10006
```

#### 3回目

```
Server Software:        CherryPy/3.7.0
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        15 bytes

Concurrency Level:      20
Time taken for tests:   18.978 seconds
Complete requests:      10000
Failed requests:        24
   (Connect: 0, Receive: 7, Length: 17, Exceptions: 0)
Keep-Alive requests:    9983
Total transferred:      1777137 bytes
HTML transferred:       149745 bytes
Requests per second:    526.91 [#/sec] (mean)
Time per request:       37.957 [ms] (mean)
Time per request:       1.898 [ms] (mean, across all concurrent requests)
Transfer rate:          91.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   3.9      0     129
Processing:     0   19 316.0      8   10007
Waiting:        0    8   2.6      7      26
Total:          1   19 316.0      8   10007

```


# 番外編

## Nginx 【笑】

いや、むっちゃ速い...というか当たり前か。

### バージョン

Nginx: 1.6.2

### 結果

#### 1回目

```
Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.384 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9911
Total transferred:      2509555 bytes
HTML transferred:       160000 bytes
Requests per second:    26072.36 [#/sec] (mean)
Time per request:       0.767 [ms] (mean)
Time per request:       0.038 [ms] (mean, across all concurrent requests)
Transfer rate:          6389.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.6      1       5
Waiting:        0    1   0.6      1       5
Total:          0    1   0.6      1       5

```

#### 2回目

```
Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.346 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9910
Total transferred:      2509550 bytes
HTML transferred:       160000 bytes
Requests per second:    28939.79 [#/sec] (mean)
Time per request:       0.691 [ms] (mean)
Time per request:       0.035 [ms] (mean, across all concurrent requests)
Transfer rate:          7092.37 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.5      1       6
Waiting:        0    1   0.5      1       6
Total:          0    1   0.5      1       6

```

#### 3回目

```
Server Software:        nginx/1.6.2
Server Hostname:        127.0.0.1
Server Port:            8888

Document Path:          /
Document Length:        16 bytes

Concurrency Level:      20
Time taken for tests:   0.346 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    9909
Total transferred:      2509545 bytes
HTML transferred:       160000 bytes
Requests per second:    28860.28 [#/sec] (mean)
Time per request:       0.693 [ms] (mean)
Time per request:       0.035 [ms] (mean, across all concurrent requests)
Transfer rate:          7072.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     0    1   0.5      1       4
Waiting:        0    1   0.5      1       4
Total:          0    1   0.5      1       4

```