続・Provisioned Concurrency for Lambda Functions ちょっと試してみた。

AWSの一番好きなサービスはAWS Lambda な人です(再掲)

---

[Provisioned Concurrency for Lambda Functions ちょっと試してみた。](https://qiita.com/keni_w/items/f284a1bac64830a826dd)
の続きです。

Apache Bench使って、負荷かけてみたので、その辺まとめます。

あと、12/20にAWS Loft Tokyoで[AWS re:Invent 2019 サーバーレス re:Cap](https://awsreinvent2019recap.splashthat.com/) (以下サーバーレスre:Cap)があり、そこで聞いた話を織り交ぜます。

あと、クラスメソッドさんのブログでも検証されていますので、こちらも。

[LambdaのProvisioned Concurrencyを使って、コールドスタート対策をしてみた #reinvent](https://dev.classmethod.jp/cloud/aws/lambda-provisioned-concurrency-coldstart/)

[安い？それとも高い？Provisioned Concurrencyを有効化したLambdaのコストに関する考察 #reinvent](https://dev.classmethod.jp/cloud/aws/simulate-provisioned-concurrency-cost/)

課金でいうと、ほぼ一ヶ月間、設定だけして、起動していたLambdaがあるんですが、$15ぐらいになってました(おおっ、課金されてる！その2)。

<img width="1379" alt="lambda課金" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/72752/d93d1a1d-2c47-2462-291c-ff749fdeeaa0.png">

ご利用は計画的にwって感じでしょうか。

## 実行パターン

以下のパターンのLambdaを準備して、Apache Benchを使って、10並列1000リクエストと20並列2000リクエストを3回ずつ実行しています。

1. 何も設定してないLambda関数(Non VPC)
2. Provisioned Concurrency=10を設定したLambda関数(Non VPC)
3. Reserved Concurrency=10、Provisioned Concurrency=10を設定したLambda関数(Non VPC)
4. 何も設定してないLambda関数(VPC)
5. Provisioned Concurrency=10を設定したLambda関数(VPC)
6. Reserved Concurrency=20、Provisioned Concurrency=10を設定したLambda関数(VPC)
    ※下記記載の3. の事象があったので、数変更

### Provisioned Concurrency設定時
設定すると、ログストリームが多めに作られる、つまりコンテナが余計に起動するのは、一緒です。
サーバーレスre:CapでSAの方に聞いてみたのですが、どうも多めに起動させて、徐々に減らして、適正な数にしてるんじゃないかとのことでした。

## 実行

まず先に書くと、3. Reserved Concurrency=10、Provisioned Concurrency=10を設定したLambda関数(Non VPC)ですが、APIから実行すると、全てのリクエストがスロットリングになりました。

<img width="423" alt="case3-1-lambda.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/72752/5062fc9f-c50c-90fa-a314-076d3fea42a9.png">

API Gatewayは500になってますね。
<img width="249" alt="case3-1-apigw.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/72752/ddab2d9b-1d82-790e-4386-6e22d5581466.png">
ログ抜粋

```
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Extended Request Id: FMkOVExVtjMFmzQ=
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Verifying Usage Plan for request: 79217ec3-8791-4e43-b667-54a35612b6d7. API Key: API Stage: aabbccddee/dev
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) API Key authorized because method 'GET /api3' does not require API Key. Request will not contribute to throttle or quota limits
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Usage Plan check succeeded for API Key and API Stage aabbccddee/dev
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Starting execution for request: 79217ec3-8791-4e43-b667-54a35612b6d7
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) HTTP Method: GET, Resource Path: /api3
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request path: {}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request query string: {count=10000}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request headers: {Accept=*/*, CloudFront-Viewer-Country=JP, CloudFront-Forwarded-Proto=https, CloudFront-Is-Tablet-Viewer=false, CloudFront-Is-Mobile-Viewer=false, User-Agent=ApacheBench/2.3, X-Forwarded-Proto=https, CloudFront-Is-SmartTV-Viewer=false, Host=aabbccddee.execute-api.ap-northeast-1.amazonaws.com, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method request body before transformations:
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request URI: https://lambda.ap-northeast-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:123456789012:function:usual-lambda-dev-index3:pbLatest/invocations
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request headers: {x-amzn-lambda-integration-tag=79217ec3-8791-4e43-b667-54a35612b6d7...
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint request body after transformations: {"resource":"/api3","path":"/api3","httpMethod":"GET","headers":{"Accept":"*/*","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-Country":"JP","Host":"vr05c
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Sending request to https://lambda.ap-northeast-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:ap-northeast-1:123456789012:function:usual-lambda-dev-index3:pbLatest/invocations
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Received response. Status: 429, Integration latency: 33 ms
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint response headers: {Date=Tue, 24 Dec 2019 06:45:09 GMT, Content-Type=application/json, Content-Length=104, Connection=keep-alive, x-amzn-RequestId=988d5bb2-6e0d-4b81-beb0-650aae7e5713, x-amzn-ErrorType=TooManyRequestsException}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Endpoint response body before transformations: {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."}
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Lambda invocation failed with status: 429. Lambda request id: 988d5bb2-6e0d-4b81-beb0-650aae7e5713
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Execution failed due to configuration error: Rate Exceeded.
15:45:09(79217ec3-8791-4e43-b667-54a35612b6d7) Method completed with status: 500
```

自分な勝手な想像としては、普通にリクエストを受け付けてくれると思ったんですけどね。10以上きたら、スロットリング起きるのはしょうがないとして。
まさか全リクエストエラーになるとは思いませんでした。。。

メトリクスを見たところ、Invocations = **ProvisionedConcurrencySpilloverInvocations** になっていました。
`ProvisionedConcurrencySpilloverInvocations` ってなんだろと探したところ、
[AWS Lambda のメトリクス](https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/monitoring-functions-metrics.html)
に以下の記載がありました。

```
すべてのプロビジョニングされた同時実行数が使用中のときに、プロビジョニングされていない同時実行数を使用して実行される呼び出しの数。プロビジョニングされた同時実行数を使用するように設定されたバージョンまたはエイリアスの場合、Lambda は、プロビジョニングされていない同時実行数を使用して実行される呼び出しごとに、1 回カウントをインクリメントします。
```

別途コンテナを起動して処理した数っていう認識（違っていたらすみません）ですが、
すでに予約済み10で、10個（おそらく10個以上）コンテナが確保済みなので、別途起動しようとしたんですが、
予約済みなので、受け付けない状態になってるんでしょうか。。。とりあえず謎です。
仕様通りっていう話であれば、全然OKなんですが。。。

Provisioned Concurrency=9など、Reserved Concurrency以下(そもそも設定時にReserved Concurrency設定値以上は登録不可)にすれば、とりあえずリクエストは受け付けてくれます。

なので、併用する場合は、`Reserved Concurrency > Provisioned Concurrency` にする必要がありそうです。

同じくサーバーレスre:Capでも聞いてみたんですけどね。こういうことする人がいないwのか、そういう話は聞いたことがないとのことで、
とりあえず書いておいてくださいってことなので、書いておきますw

---

12パターンについてログと、一部メトリクスを添付します。
ごめんなさい。抜粋してますけど、長いです。

ログの見方は以下を参照してください。
[Apache Benchでサクッと性能テスト](https://qiita.com/flexfirm/items/ac5a2f53cfa933a37192)

### パターン1 10並列
これが今までのLambdaです。

```bash
# 1回目
Concurrency Level:      10
Time taken for tests:   53.154 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    18.81 [#/sec] (mean)
Time per request:       531.537 [ms] (mean)
Time per request:       53.154 [ms] (mean, across all concurrent requests)
Transfer rate:          9.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     11      20
Processing:   393  507  94.9    514    1243
Waiting:      393  506  94.8    514    1242
Total:        404  517  95.1    524    1253

Percentage of the requests served within a certain time (ms)
  50%    524
  66%    560
  75%    571
  80%    578
  90%    584
  95%    618
  98%    675
  99%   1115
 100%   1253 (longest request)


# 2回目
Concurrency Level:      10
Time taken for tests:   51.709 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.34 [#/sec] (mean)
Time per request:       517.092 [ms] (mean)
Time per request:       51.709 [ms] (mean, across all concurrent requests)
Transfer rate:          9.75 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   0.7     11      19
Processing:   389  499  82.1    508    1154
Waiting:      389  499  82.1    507    1153
Total:        399  510  82.2    518    1167

Percentage of the requests served within a certain time (ms)
 50%    518
 66%    558
 75%    568
 80%    574
 90%    584
 95%    603
 98%    649
 99%    757
100%   1167 (longest request)

# 3回目
Server Software:
Server Hostname:        vr05cvqij1.execute-api.ap-northeast-1.amazonaws.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key:        ECDH P-256 256 bits
TLS Server Name:        vr05cvqij1.execute-api.ap-northeast-1.amazonaws.com

Document Path:          /dev/api?count=10000
Document Length:        33 bytes

Concurrency Level:      10
Time taken for tests:   51.894 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.27 [#/sec] (mean)
Time per request:       518.940 [ms] (mean)
Time per request:       51.894 [ms] (mean, across all concurrent requests)
Transfer rate:          9.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   7.2     11     237
Processing:   332  502  82.3    510    1143
Waiting:      331  502  82.3    509    1142
Total:        342  513  82.9    520    1157

Percentage of the requests served within a certain time (ms)
  50%    520
  66%    560
  75%    571
  80%    577
  90%    583
  95%    600
  98%    650
  99%    730
 100%   1157 (longest request)
```

### パターン1 20並列
安定して処理してますねw

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   53.007 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    37.73 [#/sec] (mean)
Time per request:       530.066 [ms] (mean)
Time per request:       26.503 [ms] (mean, across all concurrent requests)
Transfer rate:          19.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.1     11      22
Processing:   352  505  94.6    517    1195
Waiting:      352  505  94.6    516    1195
Total:        363  516  95.1    527    1213

Percentage of the requests served within a certain time (ms)
  50%    527
  66%    561
  75%    568
  80%    574
  90%    587
  95%    616
  98%    673
  99%   1111
 100%   1213 (longest request)

# 2回目
Concurrency Level:      20
Time taken for tests:   51.902 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    38.53 [#/sec] (mean)
Time per request:       519.025 [ms] (mean)
Time per request:       25.951 [ms] (mean, across all concurrent requests)
Transfer rate:          19.42 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.2     10      23
Processing:   387  501  82.1    510    1172
Waiting:      387  501  82.1    509    1171
Total:        398  511  82.6    520    1192

Percentage of the requests served within a certain time (ms)
 50%    520
 66%    559
 75%    569
 80%    574
 90%    583
 95%    598
 98%    647
 99%    698
100%   1192 (longest request)

# 3回目
Concurrency Level:      20
Time taken for tests:   52.264 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    38.27 [#/sec] (mean)
Time per request:       522.641 [ms] (mean)
Time per request:       26.132 [ms] (mean, across all concurrent requests)
Transfer rate:          19.28 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      32
Processing:   317  504  83.4    515    1189
Waiting:      317  504  83.4    515    1189
Total:        327  515  83.8    526    1204

Percentage of the requests served within a certain time (ms)
  50%    526
  66%    563
  75%    573
  80%    578
  90%    590
  95%    611
  98%    658
  99%    736
 100%   1204 (longest request)

```

### パターン2 10並列
若干ではありますが、MAXの値が高くなってますね。

```bash
# 1回目
Concurrency Level:      10
Time taken for tests:   53.185 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    18.80 [#/sec] (mean)
Time per request:       531.846 [ms] (mean)
Time per request:       53.185 [ms] (mean, across all concurrent requests)
Transfer rate:          9.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     10      25
Processing:   391  509  84.2    517    1022
Waiting:      391  509  84.2    517    1022
Total:        401  520  84.5    527    1036
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%    527
  66%    562
  75%    575
  80%    580
  90%    593
  95%    618
  98%    683
  99%    986
 100%   1036 (longest request)

# 2回目
Concurrency Level:      10
Time taken for tests:   52.592 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.01 [#/sec] (mean)
Time per request:       525.920 [ms] (mean)
Time per request:       52.592 [ms] (mean, across all concurrent requests)
Transfer rate:          9.58 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.7     11      21
Processing:   396  505  76.7    517    1014
Waiting:      395  505  76.7    517    1014
Total:        406  516  76.8    528    1028

Percentage of the requests served within a certain time (ms)
  50%    528
  66%    563
  75%    576
  80%    579
  90%    588
  95%    602
  98%    638
  99%    770
 100%   1028 (longest request)


# 3回目
Concurrency Level:      10
Time taken for tests:   52.384 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.09 [#/sec] (mean)
Time per request:       523.835 [ms] (mean)
Time per request:       52.384 [ms] (mean, across all concurrent requests)
Transfer rate:          9.62 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.3     11      38
Processing:   329  504  74.7    513     983
Waiting:      329  503  74.7    512     983
Total:        340  514  74.9    523     999

Percentage of the requests served within a certain time (ms)
 50%    523
 66%    561
 75%    576
 80%    580
 90%    587
 95%    600
 98%    634
 99%    732
100%    999 (longest request)
```

### パターン2 20並列
足りない分はコンテナ起動が走り、あと、スロットリングも発生してます。

1回目のメトリクスですが、
<img width="395" alt="case5-1-lambda.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/72752/6042655c-f570-1d91-5612-6ad6060b81ce.png">

これを見ると、
ProvisionedConcurrencyInvocations(プロビジョニングされた同時実行数を使用して実行される呼び出しの数)が744、
ProvisionedConcurrencySpilloverInvocationsが697
なので、起動済みコンテナで744、別途起動したコンテナで697リクエストそれぞれ呼び出され、残り559がスロットリングになっている感じですね。
何も設定してないLambdaではスロットリング発生しなかったので、この辺は謎ですが、ちゃんと負荷試験して、適切な数を設定しないとダメだよっていう戒めでしょうかw

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   39.292 seconds
Complete requests:      2000
Failed requests:        559
   (Connect: 0, Receive: 0, Length: 559, Exceptions: 0)
Non-2xx responses:      559
Total transferred:      1006286 bytes
HTML transferred:       67677 bytes
Requests per second:    50.90 [#/sec] (mean)
Time per request:       392.923 [ms] (mean)
Time per request:       19.646 [ms] (mean, across all concurrent requests)
Transfer rate:          25.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      43
Processing:    13  373 231.9    434    1642
Waiting:       12  373 231.9    434    1642
Total:         23  384 232.1    445    1662

Percentage of the requests served within a certain time (ms)
  50%    445
  66%    538
  75%    560
  80%    567
  90%    584
  95%    601
  98%    658
  99%    762
 100%   1662 (longest request)

# 2回目
Concurrency Level:      20
Time taken for tests:   44.707 seconds
Complete requests:      2000
Failed requests:        358
  (Connect: 0, Receive: 0, Length: 358, Exceptions: 0)
Non-2xx responses:      358
Total transferred:      1015532 bytes
HTML transferred:       67074 bytes
Requests per second:    44.74 [#/sec] (mean)
Time per request:       447.070 [ms] (mean)
Time per request:       22.353 [ms] (mean, across all concurrent requests)
Transfer rate:          22.18 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.6     11      48
Processing:    13  423 205.6    461    1173
Waiting:       13  423 205.6    461    1173
Total:         24  434 206.0    472    1195

Percentage of the requests served within a certain time (ms)
 50%    472
 66%    557
 75%    568
 80%    574
 90%    586
 95%    613
 98%    681
 99%    988
100%   1195 (longest request)

# 3回目
Concurrency Level:      20
Time taken for tests:   44.839 seconds
Complete requests:      2000
Failed requests:        311
   (Connect: 0, Receive: 0, Length: 311, Exceptions: 0)
Non-2xx responses:      311
Total transferred:      1017694 bytes
HTML transferred:       66933 bytes
Requests per second:    44.60 [#/sec] (mean)
Time per request:       448.390 [ms] (mean)
Time per request:       22.419 [ms] (mean, across all concurrent requests)
Transfer rate:          22.16 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.8     11      65
Processing:    13  430 187.8    477    1004
Waiting:       13  430 187.8    477    1004
Total:         24  441 187.9    488    1024

Percentage of the requests served within a certain time (ms)
  50%    488
  66%    553
  75%    566
  80%    574
  90%    584
  95%    598
  98%    647
  99%    723
 100%   1024 (longest request)

```


### パターン3 10並列
上記した通り全リクエストエラーになりました。

```bash
Concurrency Level:      10
Time taken for tests:   3.179 seconds
Complete requests:      1000
Failed requests:        0
Non-2xx responses:      1000
Total transferred:      470000 bytes
HTML transferred:       36000 bytes
Requests per second:    314.59 [#/sec] (mean)
Time per request:       31.787 [ms] (mean)
Time per request:       3.179 [ms] (mean, across all concurrent requests)
Transfer rate:          144.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     11      19
Processing:    10   20   5.7     19      90
Waiting:        9   20   5.7     19      90
Total:         21   31   5.8     30     101

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     32
  75%     33
  80%     35
  90%     39
  95%     41
  98%     45
  99%     46
 100%    101 (longest request)
```

### パターン3 20並列
これも全滅です。

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   3.258 seconds
Complete requests:      2000
Failed requests:        0
Non-2xx responses:      2000
Total transferred:      940000 bytes
HTML transferred:       72000 bytes
Requests per second:    613.92 [#/sec] (mean)
Time per request:       32.577 [ms] (mean)
Time per request:       1.629 [ms] (mean, across all concurrent requests)
Transfer rate:          281.78 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   12   1.6     12      30
Processing:    10   20   5.5     19      91
Waiting:        9   19   5.5     19      91
Total:         20   32   5.9     31     101

Percentage of the requests served within a certain time (ms)
 50%     31
 66%     32
 75%     34
 80%     35
 90%     38
 95%     42
 98%     46
 99%     49
100%    101 (longest request)
```

次からは真価を発揮するはずのVPC Lambdaです（こっちだけでいいんじゃね説あり）
### パターン4 10並列
これが今までのLambdaです。
初回はちょっと時間かかってるかな。
とはいえ、ms単位なので、ほんと速くなりました！

```bash
# 1回目
Concurrency Level:      10
Time taken for tests:   53.362 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    18.74 [#/sec] (mean)
Time per request:       533.621 [ms] (mean)
Time per request:       53.362 [ms] (mean, across all concurrent requests)
Transfer rate:          9.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.0     11      34
Processing:   389  507 107.8    517    1413
Waiting:      389  507 107.8    517    1413
Total:        400  518 108.1    528    1424

Percentage of the requests served within a certain time (ms)
  50%    528
  66%    560
  75%    570
  80%    577
  90%    587
  95%    619
  98%    673
  99%   1263
 100%   1424 (longest request)

# 2回目
Concurrency Level:      10
Time taken for tests:   51.580 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.39 [#/sec] (mean)
Time per request:       515.800 [ms] (mean)
Time per request:       51.580 [ms] (mean, across all concurrent requests)
Transfer rate:          9.77 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     10      24
Processing:   356  497  69.0    507     809
Waiting:      355  497  69.0    507     809
Total:        366  508  69.0    518     819
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%    518
  66%    559
  75%    565
  80%    571
  90%    581
  95%    590
  98%    619
  99%    709
 100%    819 (longest request)

# 3回目
Concurrency Level:      10
Time taken for tests:   51.879 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.28 [#/sec] (mean)
Time per request:       518.788 [ms] (mean)
Time per request:       51.879 [ms] (mean, across all concurrent requests)
Transfer rate:          9.71 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   0.7     11      16
Processing:   359  500  71.4    509     937
Waiting:      359  499  71.4    509     937
Total:        370  510  71.4    520     948

Percentage of the requests served within a certain time (ms)
 50%    520
 66%    560
 75%    571
 80%    578
 90%    586
 95%    598
 98%    625
 99%    693
100%    948 (longest request)

```

### パターン4 20並列
こちらも安定して処理してますねw

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   53.248 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    37.56 [#/sec] (mean)
Time per request:       532.478 [ms] (mean)
Time per request:       26.624 [ms] (mean, across all concurrent requests)
Transfer rate:          18.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      22
Processing:   376  506 107.8    517    1374
Waiting:      375  506 107.8    517    1374
Total:        386  517 108.5    528    1394

Percentage of the requests served within a certain time (ms)
  50%    528
  66%    560
  75%    568
  80%    573
  90%    587
  95%    613
  98%    673
  99%   1299
 100%   1394 (longest request)

# 2回目
Concurrency Level:      20
Time taken for tests:   51.571 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    38.78 [#/sec] (mean)
Time per request:       515.709 [ms] (mean)
Time per request:       25.785 [ms] (mean, across all concurrent requests)
Transfer rate:          19.54 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.2     10      24
Processing:   382  498  70.4    505     980
Waiting:      381  498  70.4    505     980
Total:        392  509  70.4    515    1001

Percentage of the requests served within a certain time (ms)
 50%    515
 66%    558
 75%    567
 80%    574
 90%    583
 95%    593
 98%    616
 99%    750
100%   1001 (longest request)

# 3回目
Concurrency Level:      20
Time taken for tests:   51.633 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      1032000 bytes
HTML transferred:       66000 bytes
Requests per second:    38.74 [#/sec] (mean)
Time per request:       516.326 [ms] (mean)
Time per request:       25.816 [ms] (mean, across all concurrent requests)
Transfer rate:          19.52 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      25
Processing:   376  498  70.6    510     818
Waiting:      375  498  70.5    509     818
Total:        386  509  70.7    520     838

Percentage of the requests served within a certain time (ms)
 50%    520
 66%    557
 75%    567
 80%    574
 90%    586
 95%    595
 98%    616
 99%    670
100%    838 (longest request)

```

### パターン5 10並列
3リクエストとも似たような値で、真価発揮しまくりですね。

```bash
# 1回目
Concurrency Level:      10
Time taken for tests:   53.120 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    18.83 [#/sec] (mean)
Time per request:       531.199 [ms] (mean)
Time per request:       53.120 [ms] (mean, across all concurrent requests)
Transfer rate:          9.49 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   0.8     11      26
Processing:   392  508  86.0    516    1042
Waiting:      391  508  85.9    515    1042
Total:        402  519  86.2    527    1053

Percentage of the requests served within a certain time (ms)
 50%    527
 66%    564
 75%    576
 80%    580
 90%    594
 95%    616
 98%    697
 99%    976
100%   1053 (longest request)

# 2回目
Concurrency Level:      10
Time taken for tests:   51.891 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.27 [#/sec] (mean)
Time per request:       518.911 [ms] (mean)
Time per request:       51.891 [ms] (mean, across all concurrent requests)
Transfer rate:          9.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     11      18
Processing:   391  501  69.0    513     815
Waiting:      390  500  69.0    513     814
Total:        401  511  69.1    524     827

Percentage of the requests served within a certain time (ms)
  50%    524
  66%    559
  75%    569
  80%    573
  90%    582
  95%    595
  98%    611
  99%    771
 100%    827 (longest request)

#3回目
Concurrency Level:      10
Time taken for tests:   52.026 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.22 [#/sec] (mean)
Time per request:       520.264 [ms] (mean)
Time per request:       52.026 [ms] (mean, across all concurrent requests)
Transfer rate:          9.69 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   0.8     10      23
Processing:   389  502  72.4    517    1003
Waiting:      389  502  72.4    517    1003
Total:        400  513  72.5    528    1017
WARNING: The median and mean for the initial connection time are not within a normal deviation
       These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
 50%    528
 66%    560
 75%    572
 80%    578
 90%    590
 95%    601
 98%    635
 99%    683
100%   1017 (longest request)

```

### パターン5 20並列
やはりスロットリングが発生してますね。
1回目はLambdaの起動が走っているので、若干MAXの値が高いです。

ごめんなさい。3回目のログ欠落してます。。。
メトリクスは以下になります。
<img width="419" alt="case11-3-lambda.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/72752/62bc6a51-44f5-6b18-ee71-9bbbaf072884.png">

ProvisionedConcurrencyInvocations(プロビジョニングされた同時実行数を使用して実行される呼び出しの数)が833、
ProvisionedConcurrencySpilloverInvocationsが805
なので、起動済みコンテナで833、別途起動したコンテナで805リクエストそれぞれ呼び出され、残り362がスロットリングになっている感じですね。

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   44.721 seconds
Complete requests:      2000
Failed requests:        362
  (Connect: 0, Receive: 0, Length: 362, Exceptions: 0)
Non-2xx responses:      362
Total transferred:      1015348 bytes
HTML transferred:       67086 bytes
Requests per second:    44.72 [#/sec] (mean)
Time per request:       447.212 [ms] (mean)
Time per request:       22.361 [ms] (mean, across all concurrent requests)
Transfer rate:          22.17 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      27
Processing:    12  424 208.9    463    1714
Waiting:       12  424 208.9    463    1714
Total:         23  435 209.1    473    1733

Percentage of the requests served within a certain time (ms)
 50%    473
 66%    556
 75%    569
 80%    578
 90%    589
 95%    609
 98%    673
 99%    737
100%   1733 (longest request)


# 2回目
Concurrency Level:      20
Time taken for tests:   44.973 seconds
Complete requests:      2000
Failed requests:        305
   (Connect: 0, Receive: 0, Length: 305, Exceptions: 0)
Non-2xx responses:      305
Total transferred:      1017970 bytes
HTML transferred:       66915 bytes
Requests per second:    44.47 [#/sec] (mean)
Time per request:       449.731 [ms] (mean)
Time per request:       22.487 [ms] (mean, across all concurrent requests)
Transfer rate:          22.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      23
Processing:    13  430 183.8    474     799
Waiting:       13  430 183.9    473     799
Total:         24  441 183.9    485     820

Percentage of the requests served within a certain time (ms)
  50%    485
  66%    556
  75%    569
  80%    576
  90%    587
  95%    600
  98%    617
  99%    658
 100%    820 (longest request)

```

### パターン6 10並列
値を変えたので、安定して処理しています。
パターン5と傾向似てます。

```bash
# 1回目
Concurrency Level:      10
Time taken for tests:   53.350 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    18.74 [#/sec] (mean)
Time per request:       533.498 [ms] (mean)
Time per request:       53.350 [ms] (mean, across all concurrent requests)
Transfer rate:          9.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.6     10      19
Processing:   397  510  85.3    519    1059
Waiting:      397  510  85.3    519    1059
Total:        407  521  85.5    529    1070
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%    529
  66%    567
  75%    577
  80%    580
  90%    595
  95%    617
  98%    677
  99%    981
 100%   1070 (longest request)

# 2回目
Concurrency Level:      10
Time taken for tests:   52.095 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.20 [#/sec] (mean)
Time per request:       520.947 [ms] (mean)
Time per request:       52.095 [ms] (mean, across all concurrent requests)
Transfer rate:          9.67 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   0.8     10      25
Processing:   393  503  69.5    513     879
Waiting:      393  503  69.5    513     878
Total:        403  514  69.6    523     889
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%    523
  66%    561
  75%    572
  80%    578
  90%    586
  95%    598
  98%    619
  99%    680
 100%    889 (longest request)


# 3回目
Concurrency Level:      10
Time taken for tests:   52.404 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      516000 bytes
HTML transferred:       33000 bytes
Requests per second:    19.08 [#/sec] (mean)
Time per request:       524.041 [ms] (mean)
Time per request:       52.404 [ms] (mean, across all concurrent requests)
Transfer rate:          9.62 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   0.8     11      23
Processing:   396  506  71.8    517     817
Waiting:      395  506  71.8    517     817
Total:        406  516  71.8    527     831

Percentage of the requests served within a certain time (ms)
 50%    527
 66%    565
 75%    578
 80%    581
 90%    597
 95%    603
 98%    622
 99%    752
100%    831 (longest request)

```

### パターン6 20並列
これもスロットリングおきてますね。。。

```bash
# 1回目
Concurrency Level:      20
Time taken for tests:   42.204 seconds
Complete requests:      2000
Failed requests:        442
   (Connect: 0, Receive: 0, Length: 442, Exceptions: 0)
Non-2xx responses:      442
Total transferred:      1011668 bytes
HTML transferred:       67326 bytes
Requests per second:    47.39 [#/sec] (mean)
Time per request:       422.038 [ms] (mean)
Time per request:       21.102 [ms] (mean, across all concurrent requests)
Transfer rate:          23.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      24
Processing:    12  404 221.6    445    1386
Waiting:       12  404 221.6    445    1386
Total:         23  415 221.9    456    1409

Percentage of the requests served within a certain time (ms)
  50%    456
  66%    550
  75%    569
  80%    576
  90%    590
  95%    609
  98%    673
  99%    761
 100%   1409 (longest request)

# 2回目
Concurrency Level:      20
Time taken for tests:   42.836 seconds
Complete requests:      2000
Failed requests:        395
   (Connect: 0, Receive: 0, Length: 395, Exceptions: 0)
Non-2xx responses:      395
Total transferred:      1013830 bytes
HTML transferred:       67185 bytes
Requests per second:    46.69 [#/sec] (mean)
Time per request:       428.356 [ms] (mean)
Time per request:       21.418 [ms] (mean, across all concurrent requests)
Transfer rate:          23.11 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   11   1.4     10      30
Processing:    13  410 201.1    452     837
Waiting:       13  409 201.2    452     837
Total:         24  420 201.2    463     858

Percentage of the requests served within a certain time (ms)
  50%    463
  66%    549
  75%    566
  80%    573
  90%    587
  95%    599
  98%    620
  99%    667
 100%    858 (longest request)


# 3回目
Concurrency Level:      20
Time taken for tests:   44.347 seconds
Complete requests:      2000
Failed requests:        328
  (Connect: 0, Receive: 0, Length: 328, Exceptions: 0)
Non-2xx responses:      328
Total transferred:      1016912 bytes
HTML transferred:       66984 bytes
Requests per second:    45.10 [#/sec] (mean)
Time per request:       443.467 [ms] (mean)
Time per request:       22.173 [ms] (mean, across all concurrent requests)
Transfer rate:          22.39 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:       10   11   1.2     11      27
Processing:    13  426 189.0    469     949
Waiting:       13  425 189.0    469     949
Total:         24  436 189.1    481     959

Percentage of the requests served within a certain time (ms)
 50%    481
 66%    554
 75%    570
 80%    576
 90%    589
 95%    601
 98%    623
 99%    711
100%    959 (longest request)
```

---

VPC Lambda速くなったなーというのは素直な感想。
また、Provisioned concurrencyを設定すると、さらに安定して処理してくれる感じですね。
ただ、Provisioned concurrency設定以上の同時リクエストがあった場合にスロットリング起きるのがちょっと意外でした。
未設定なLambdaではおきなかったので。
もしかしたら、設定してすぐにテストを実施したので、
「うも多めに起動させて、徐々に減らして、適正な数にしてるんじゃないか」っていうあたりが絡んでたりするのかな。。。
来年早々に、設定して、しばらく放置してから、どうなるかを検証してみたいと思います。

--- 

サーバーレスre:Capでも説明あったんですが、
Provisioned concurrencyは、定量的に同時にリクエストがあるならば、Lambdaを起動しておいてくれるのはもちろん、
事前にバーストさせておくことができるのが真価の一つとのこと。
つまり、
同時接続数を上限緩和して、1500等にした場合、従来だと、
ベースの同時接続数の1000までは即時バーストしますが、
そのあとの上限までは30秒に250ずつ（１分間に500ずつ）スケールする仕様ですが、
Provisioned concurrencyは1500って指定すれば、設定時に、1500個のコンテナ(+α)が起動します。
※同時接続数割り当てのうち、100は他のLambdaの起動用に必要らしいので、　1500を指定したい場合は、同時接続数は1600以上にしておく必要があります。

このテストしたいんですけど、個人検証用アカウントなんで上限緩和通るかな。。。w

## 最後に
サーバーレスre:Capでも話ありましたが、
VPC Lambdaはアンチパターンと言われ続けてきましたが、
今年のLambdaのアップデート(ENI作成タイミング変更)と、 この `Provisioned concurrency` と `Data Proxy` でもはやアンチパターンではなくなったのではないでしょうかとのこと。
RDSなどを使いたいはもちろんのこと、Lambdaから他のAPIを呼ぶ際に、セキュリティ要件でIP縛りをしたいという際にVPC Lambdaにして、外への通信をNAT Gateway経由にするという用途にも使いやすくなったようです。