ベンチマークRallyをEC2-VM上のelasticsearchで実行する

# TL;DR

* Elasticsearchのマイクロベンチマークフレームワーク[Rally](https://github.com/elastic/rally)を実行する方法を紹介します
    * EC2でVMを作り，DockerでElasticsearchを立ち上げた環境で実験します

# 実験条件

* EC2でt2.largeのVM (amazon linux2)を作成
    * vCPU 2コア，8GB RAM
    * SSHでログインして作業

# ベンチマークツールesrallyのインストール

* JAVA8，python3, gcc，gitが必要です
* pip3でesrallyをインストールできます

```bash
# 必要なツールをインストールする
sudo yum groupinstall -y 'Development Tools'
sudo yum install -y git
sudo yum install -y java-1.8.0-openjdk
# python3を使うためにepelを取り込み
sudo amazon-linux-extras install -y epel
sudo yum --enablerepo=epel install -y python36 python36-libs python36-devel python36-pip
sudo pip3 install --upgrade pip
# esrallyをインストールする
sudo /usr/local/bin/pip3 install esrally
```

# Dockerで対象となるElasticsearchを立ち上げる

* Dockerサーバをインストールします

```bash
sudo yum install -y docker
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -a -G docker ec2-user
```

* Elasticsearchを立ち上げます
    * 現在の最新バージョンは7.0.0ですが，実験の都合上，6.0.0を使用します

```bash
# /etc/sysctl.conf末尾に下記の数値を追加して，sudo sysctl -pする
# vm.max_map_count=262144
sudo sysctl -w vm.max_map_count=262144
mkdir -p es/{nodes,config}
sudo chown -R 1000:1000 es/{nodes,config}
# ulimit値を設定してElasticsearhの立ち上げ
docker run -d \
    --ulimit nofile=65536:65536\
    -p 127.0.0.1:9200:9200 \
    -p 9300 \
    -v ${PWD}/es:/usr/share/elasticsearch/data \
    docker.elastic.co/elasticsearch/elasticsearch-oss:6.0.0
```

# ベンチマークesrallyの実行

* ```JAVA_HOME```を設定して，```esrally```コマンドでベンチマークを実行します

```bash
# JAVA_HOMEを環境変数に設定．
# インストールしたJDKのバージョンによっては異なるパスになる場合があるので注意
export JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/"
# esrallyの実行
esrally --distribution-version=6.0.0
```

# 実行結果

* 実行結果は，Markdownの表のフォーマットで出力されるので，適当なエディタ(Qiitaでも)で確認できます
    * 実行後に出力される「Final Result」後の表です
    * ```~/.rally/logs/rally.log```にある実行ログにも実行結果が記載されています
    * 内容の分析は別記事で行います

```bash
    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/
[INFO] Preparing for race ...
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['defaults'] with version [6.0.0].
[INFO] Downloading data for track geonames (252.4 MB total size)                  [100.0%]
[INFO] Decompressing track data from [/home/ec2-user/.rally/benchmarks/data/geonames/documents-2.json.bz2] to [/home/ec2-user/.rally/benchmarks/data/geonames/documents-2.json] (resulting size: 3.30 GB) ... [OK]
[INFO] Preparing file offset table for [/home/ec2-user/.rally/benchmarks/data/geonames/documents-2.json] ... [OK]
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running index-stats                                                            [100% done]
Running node-stats                                                             [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running country_agg_uncached                                                   [100% done]
Running country_agg_cached                                                     [100% done]
Running scroll                                                                 [100% done]
Running expression                                                             [100% done]
Running painless_static                                                        [100% done]
Running painless_dynamic                                                       [100% done]
Running large_terms                                                            [100% done]
Running large_filtered_terms                                                   [100% done]
Running large_prohibited_terms                                                 [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
.....<snip>.....
----------------------------------
[INFO] SUCCESS (took 3894 seconds)
----------------------------------
```

|   Lap |                                                         Metric |                   Task |      Value |    Unit |
|------:|---------------------------------------------------------------:|-----------------------:|-----------:|--------:|
|   All |                     Cumulative indexing time of primary shards |                        |    25.5939 |     min |
|   All |             Min cumulative indexing time across primary shards |                        |     4.6259 |     min |
|   All |          Median cumulative indexing time across primary shards |                        |    5.19343 |     min |
|   All |             Max cumulative indexing time across primary shards |                        |    5.65612 |     min |
|   All |            Cumulative indexing throttle time of primary shards |                        |          0 |     min |
|   All |    Min cumulative indexing throttle time across primary shards |                        |          0 |     min |
|   All | Median cumulative indexing throttle time across primary shards |                        |          0 |     min |
|   All |    Max cumulative indexing throttle time across primary shards |                        |          0 |     min |
|   All |                        Cumulative merge time of primary shards |                        |    20.3031 |     min |
|   All |                       Cumulative merge count of primary shards |                        |        247 |         |
|   All |                Min cumulative merge time across primary shards |                        |    3.53572 |     min |
|   All |             Median cumulative merge time across primary shards |                        |    4.22415 |     min |
|   All |                Max cumulative merge time across primary shards |                        |    4.32668 |     min |
|   All |               Cumulative merge throttle time of primary shards |                        |    1.00265 |     min |
|   All |       Min cumulative merge throttle time across primary shards |                        |     0.1939 |     min |
|   All |    Median cumulative merge throttle time across primary shards |                        |     0.1977 |     min |
|   All |       Max cumulative merge throttle time across primary shards |                        |   0.207833 |     min |
|   All |                      Cumulative refresh time of primary shards |                        |    6.52667 |     min |
|   All |                     Cumulative refresh count of primary shards |                        |       2354 |         |
|   All |              Min cumulative refresh time across primary shards |                        |    1.25955 |     min |
|   All |           Median cumulative refresh time across primary shards |                        |    1.28618 |     min |
|   All |              Max cumulative refresh time across primary shards |                        |    1.37137 |     min |
|   All |                        Cumulative flush time of primary shards |                        |  0.0378833 |     min |
|   All |                       Cumulative flush count of primary shards |                        |         10 |         |
|   All |                Min cumulative flush time across primary shards |                        |     0.0066 |     min |
|   All |             Median cumulative flush time across primary shards |                        |     0.0075 |     min |
|   All |                Max cumulative flush time across primary shards |                        | 0.00943333 |     min |
|   All |                                               Median CPU usage |                        |      171.9 |       % |
|   All |                                             Total Young Gen GC |                        |     80.322 |       s |
|   All |                                               Total Old Gen GC |                        |      2.226 |       s |
|   All |                                                     Store size |                        |    3.02878 |      GB |
|   All |                                                  Translog size |                        |    2.62772 |      GB |
|   All |                                                     Index size |                        |    5.65651 |      GB |
|   All |                                                  Total written |                        |    19.6113 |      GB |
|   All |                                         Heap used for segments |                        |     18.432 |      MB |
|   All |                                       Heap used for doc values |                        |  0.0481262 |      MB |
|   All |                                            Heap used for terms |                        |    17.2579 |      MB |
|   All |                                            Heap used for norms |                        |  0.0744629 |      MB |
|   All |                                           Heap used for points |                        |   0.255194 |      MB |
|   All |                                    Heap used for stored fields |                        |    0.79628 |      MB |
|   All |                                                  Segment count |                        |         98 |         |
|   All |                                                 Min Throughput |           index-append |    12146.7 |  docs/s |
|   All |                                              Median Throughput |           index-append |    13113.7 |  docs/s |
|   All |                                                 Max Throughput |           index-append |    13460.6 |  docs/s |
|   All |                                        50th percentile latency |           index-append |    2415.24 |      ms |
|   All |                                        90th percentile latency |           index-append |    4133.85 |      ms |
|   All |                                        99th percentile latency |           index-append |    7828.23 |      ms |
|   All |                                      99.9th percentile latency |           index-append |    8268.85 |      ms |
|   All |                                       100th percentile latency |           index-append |     8417.6 |      ms |
|   All |                                   50th percentile service time |           index-append |    2415.24 |      ms |
|   All |                                   90th percentile service time |           index-append |    4133.85 |      ms |
|   All |                                   99th percentile service time |           index-append |    7828.23 |      ms |
|   All |                                 99.9th percentile service time |           index-append |    8268.85 |      ms |
|   All |                                  100th percentile service time |           index-append |     8417.6 |      ms |
|   All |                                                     error rate |           index-append |          0 |       % |
|   All |                                                 Min Throughput |            index-stats |     100.03 |   ops/s |
|   All |                                              Median Throughput |            index-stats |     100.07 |   ops/s |
|   All |                                                 Max Throughput |            index-stats |     100.13 |   ops/s |
|   All |                                        50th percentile latency |            index-stats |    1.91153 |      ms |
|   All |                                        90th percentile latency |            index-stats |    2.10202 |      ms |
|   All |                                        99th percentile latency |            index-stats |    3.05171 |      ms |
|   All |                                      99.9th percentile latency |            index-stats |    12.9248 |      ms |
|   All |                                       100th percentile latency |            index-stats |    21.1783 |      ms |
|   All |                                   50th percentile service time |            index-stats |    1.77476 |      ms |
|   All |                                   90th percentile service time |            index-stats |    1.94796 |      ms |
|   All |                                   99th percentile service time |            index-stats |    2.48402 |      ms |
|   All |                                 99.9th percentile service time |            index-stats |    10.4326 |      ms |
|   All |                                  100th percentile service time |            index-stats |     21.053 |      ms |
|   All |                                                     error rate |            index-stats |          0 |       % |
|   All |                                                 Min Throughput |             node-stats |     100.08 |   ops/s |
|   All |                                              Median Throughput |             node-stats |     100.15 |   ops/s |
|   All |                                                 Max Throughput |             node-stats |     100.79 |   ops/s |
|   All |                                        50th percentile latency |             node-stats |    1.98988 |      ms |
|   All |                                        90th percentile latency |             node-stats |    2.33142 |      ms |
|   All |                                        99th percentile latency |             node-stats |     3.1603 |      ms |
|   All |                                      99.9th percentile latency |             node-stats |    9.68748 |      ms |
|   All |                                       100th percentile latency |             node-stats |    13.9854 |      ms |
|   All |                                   50th percentile service time |             node-stats |    1.84291 |      ms |
|   All |                                   90th percentile service time |             node-stats |    2.17902 |      ms |
|   All |                                   99th percentile service time |             node-stats |    2.98435 |      ms |
|   All |                                 99.9th percentile service time |             node-stats |    9.56145 |      ms |
|   All |                                  100th percentile service time |             node-stats |    13.8329 |      ms |
|   All |                                                     error rate |             node-stats |          0 |       % |
|   All |                                                 Min Throughput |                default |      31.34 |   ops/s |
|   All |                                              Median Throughput |                default |      31.87 |   ops/s |
|   All |                                                 Max Throughput |                default |       32.8 |   ops/s |
|   All |                                        50th percentile latency |                default |    11396.2 |      ms |
|   All |                                        90th percentile latency |                default |    16651.7 |      ms |
|   All |                                        99th percentile latency |                default |    17773.7 |      ms |
|   All |                                      99.9th percentile latency |                default |    17879.2 |      ms |
|   All |                                       100th percentile latency |                default |    17890.9 |      ms |
|   All |                                   50th percentile service time |                default |    31.6201 |      ms |
|   All |                                   90th percentile service time |                default |      35.58 |      ms |
|   All |                                   99th percentile service time |                default |    52.2374 |      ms |
|   All |                                 99.9th percentile service time |                default |    57.0613 |      ms |
|   All |                                  100th percentile service time |                default |    57.0929 |      ms |
|   All |                                                     error rate |                default |          0 |       % |
|   All |                                                 Min Throughput |                   term |     200.11 |   ops/s |
|   All |                                              Median Throughput |                   term |     200.15 |   ops/s |
|   All |                                                 Max Throughput |                   term |     200.25 |   ops/s |
|   All |                                        50th percentile latency |                   term |    1.28876 |      ms |
|   All |                                        90th percentile latency |                   term |    1.42697 |      ms |
|   All |                                        99th percentile latency |                   term |    4.46916 |      ms |
|   All |                                      99.9th percentile latency |                   term |    7.76248 |      ms |
|   All |                                       100th percentile latency |                   term |    7.82244 |      ms |
|   All |                                   50th percentile service time |                   term |    1.14368 |      ms |
|   All |                                   90th percentile service time |                   term |    1.27048 |      ms |
|   All |                                   99th percentile service time |                   term |    4.34528 |      ms |
|   All |                                 99.9th percentile service time |                   term |    7.63329 |      ms |
|   All |                                  100th percentile service time |                   term |    7.67176 |      ms |
|   All |                                                     error rate |                   term |          0 |       % |
|   All |                                                 Min Throughput |                 phrase |      199.9 |   ops/s |
|   All |                                              Median Throughput |                 phrase |      200.1 |   ops/s |
|   All |                                                 Max Throughput |                 phrase |     200.14 |   ops/s |
|   All |                                        50th percentile latency |                 phrase |    1.96748 |      ms |
|   All |                                        90th percentile latency |                 phrase |    2.42205 |      ms |
|   All |                                        99th percentile latency |                 phrase |    6.54227 |      ms |
|   All |                                      99.9th percentile latency |                 phrase |    10.4821 |      ms |
|   All |                                       100th percentile latency |                 phrase |    11.7456 |      ms |
|   All |                                   50th percentile service time |                 phrase |    1.83008 |      ms |
|   All |                                   90th percentile service time |                 phrase |    2.22151 |      ms |
|   All |                                   99th percentile service time |                 phrase |    5.99949 |      ms |
|   All |                                 99.9th percentile service time |                 phrase |    10.3273 |      ms |
|   All |                                  100th percentile service time |                 phrase |    11.5994 |      ms |
|   All |                                                     error rate |                 phrase |          0 |       % |
|   All |                                                 Min Throughput |   country_agg_uncached |          2 |   ops/s |
|   All |                                              Median Throughput |   country_agg_uncached |       2.03 |   ops/s |
|   All |                                                 Max Throughput |   country_agg_uncached |       2.04 |   ops/s |
|   All |                                        50th percentile latency |   country_agg_uncached |      73651 |      ms |
|   All |                                        90th percentile latency |   country_agg_uncached |    84273.8 |      ms |
|   All |                                        99th percentile latency |   country_agg_uncached |      86666 |      ms |
|   All |                                       100th percentile latency |   country_agg_uncached |    86931.5 |      ms |
|   All |                                   50th percentile service time |   country_agg_uncached |    462.421 |      ms |
|   All |                                   90th percentile service time |   country_agg_uncached |    497.543 |      ms |
|   All |                                   99th percentile service time |   country_agg_uncached |    509.695 |      ms |
|   All |                                  100th percentile service time |   country_agg_uncached |     549.01 |      ms |
|   All |                                                     error rate |   country_agg_uncached |          0 |       % |
|   All |                                                 Min Throughput |     country_agg_cached |     100.06 |   ops/s |
|   All |                                              Median Throughput |     country_agg_cached |     100.09 |   ops/s |
|   All |                                                 Max Throughput |     country_agg_cached |     100.17 |   ops/s |
|   All |                                        50th percentile latency |     country_agg_cached |    1.40874 |      ms |
|   All |                                        90th percentile latency |     country_agg_cached |    1.59315 |      ms |
|   All |                                        99th percentile latency |     country_agg_cached |    5.60768 |      ms |
|   All |                                      99.9th percentile latency |     country_agg_cached |    7.79312 |      ms |
|   All |                                       100th percentile latency |     country_agg_cached |    8.86712 |      ms |
|   All |                                   50th percentile service time |     country_agg_cached |    1.26017 |      ms |
|   All |                                   90th percentile service time |     country_agg_cached |    1.45007 |      ms |
|   All |                                   99th percentile service time |     country_agg_cached |    5.28506 |      ms |
|   All |                                 99.9th percentile service time |     country_agg_cached |    7.63912 |      ms |
|   All |                                  100th percentile service time |     country_agg_cached |    8.72005 |      ms |
|   All |                                                     error rate |     country_agg_cached |          0 |       % |
|   All |                                                 Min Throughput |                 scroll |      25.04 | pages/s |
|   All |                                              Median Throughput |                 scroll |      25.05 | pages/s |
|   All |                                                 Max Throughput |                 scroll |      25.07 | pages/s |
|   All |                                        50th percentile latency |                 scroll |    461.769 |      ms |
|   All |                                        90th percentile latency |                 scroll |    477.373 |      ms |
|   All |                                        99th percentile latency |                 scroll |    494.322 |      ms |
|   All |                                       100th percentile latency |                 scroll |     513.31 |      ms |
|   All |                                   50th percentile service time |                 scroll |    461.115 |      ms |
|   All |                                   90th percentile service time |                 scroll |    476.709 |      ms |
|   All |                                   99th percentile service time |                 scroll |     493.68 |      ms |
|   All |                                  100th percentile service time |                 scroll |    512.683 |      ms |
|   All |                                                     error rate |                 scroll |          0 |       % |
|   All |                                                 Min Throughput |             expression |       0.99 |   ops/s |
|   All |                                              Median Throughput |             expression |          1 |   ops/s |
|   All |                                                 Max Throughput |             expression |          1 |   ops/s |
|   All |                                        50th percentile latency |             expression |     126797 |      ms |
|   All |                                        90th percentile latency |             expression |     146843 |      ms |
|   All |                                        99th percentile latency |             expression |     151294 |      ms |
|   All |                                       100th percentile latency |             expression |     151794 |      ms |
|   All |                                   50th percentile service time |             expression |    1004.36 |      ms |
|   All |                                   90th percentile service time |             expression |    1020.08 |      ms |
|   All |                                   99th percentile service time |             expression |    1027.16 |      ms |
|   All |                                  100th percentile service time |             expression |    1073.06 |      ms |
|   All |                                                     error rate |             expression |          0 |       % |
|   All |                                                 Min Throughput |        painless_static |        0.7 |   ops/s |
|   All |                                              Median Throughput |        painless_static |        0.7 |   ops/s |
|   All |                                                 Max Throughput |        painless_static |        0.7 |   ops/s |
|   All |                                        50th percentile latency |        painless_static |     193836 |      ms |
|   All |                                        90th percentile latency |        painless_static |     224122 |      ms |
|   All |                                        99th percentile latency |        painless_static |     230857 |      ms |
|   All |                                       100th percentile latency |        painless_static |     231625 |      ms |
|   All |                                   50th percentile service time |        painless_static |     1430.5 |      ms |
|   All |                                   90th percentile service time |        painless_static |    1465.87 |      ms |
|   All |                                   99th percentile service time |        painless_static |     1500.2 |      ms |
|   All |                                  100th percentile service time |        painless_static |    1500.55 |      ms |
|   All |                                                     error rate |        painless_static |          0 |       % |
|   All |                                                 Min Throughput |       painless_dynamic |       0.72 |   ops/s |
|   All |                                              Median Throughput |       painless_dynamic |       0.72 |   ops/s |
|   All |                                                 Max Throughput |       painless_dynamic |       0.72 |   ops/s |
|   All |                                        50th percentile latency |       painless_dynamic |     180020 |      ms |
|   All |                                        90th percentile latency |       painless_dynamic |     208289 |      ms |
|   All |                                        99th percentile latency |       painless_dynamic |     214501 |      ms |
|   All |                                       100th percentile latency |       painless_dynamic |     215270 |      ms |
|   All |                                   50th percentile service time |       painless_dynamic |    1378.16 |      ms |
|   All |                                   90th percentile service time |       painless_dynamic |    1412.12 |      ms |
|   All |                                   99th percentile service time |       painless_dynamic |    1428.57 |      ms |
|   All |                                  100th percentile service time |       painless_dynamic |    1444.22 |      ms |
|   All |                                                     error rate |       painless_dynamic |          0 |       % |
|   All |                                                 Min Throughput |            large_terms |        0.8 |   ops/s |
|   All |                                              Median Throughput |            large_terms |        0.8 |   ops/s |
|   All |                                                 Max Throughput |            large_terms |       0.81 |   ops/s |
|   All |                                        50th percentile latency |            large_terms |     186854 |      ms |
|   All |                                        90th percentile latency |            large_terms |     215773 |      ms |
|   All |                                        99th percentile latency |            large_terms |     222361 |      ms |
|   All |                                       100th percentile latency |            large_terms |     223148 |      ms |
|   All |                                   50th percentile service time |            large_terms |    1243.98 |      ms |
|   All |                                   90th percentile service time |            large_terms |    1271.08 |      ms |
|   All |                                   99th percentile service time |            large_terms |    1292.47 |      ms |
|   All |                                  100th percentile service time |            large_terms |    1293.73 |      ms |
|   All |                                                     error rate |            large_terms |          0 |       % |
|   All |                                                 Min Throughput |   large_filtered_terms |        0.8 |   ops/s |
|   All |                                              Median Throughput |   large_filtered_terms |       0.81 |   ops/s |
|   All |                                                 Max Throughput |   large_filtered_terms |       0.81 |   ops/s |
|   All |                                        50th percentile latency |   large_filtered_terms |     185424 |      ms |
|   All |                                        90th percentile latency |   large_filtered_terms |     214286 |      ms |
|   All |                                        99th percentile latency |   large_filtered_terms |     220652 |      ms |
|   All |                                       100th percentile latency |   large_filtered_terms |     221358 |      ms |
|   All |                                   50th percentile service time |   large_filtered_terms |    1225.57 |      ms |
|   All |                                   90th percentile service time |   large_filtered_terms |    1260.09 |      ms |
|   All |                                   99th percentile service time |   large_filtered_terms |    1296.66 |      ms |
|   All |                                  100th percentile service time |   large_filtered_terms |    1303.19 |      ms |
|   All |                                                     error rate |   large_filtered_terms |          0 |       % |
|   All |                                                 Min Throughput | large_prohibited_terms |       0.82 |   ops/s |
|   All |                                              Median Throughput | large_prohibited_terms |       0.83 |   ops/s |
|   All |                                                 Max Throughput | large_prohibited_terms |       0.83 |   ops/s |
|   All |                                        50th percentile latency | large_prohibited_terms |     178794 |      ms |
|   All |                                        90th percentile latency | large_prohibited_terms |     205239 |      ms |
|   All |                                        99th percentile latency | large_prohibited_terms |     211744 |      ms |
|   All |                                       100th percentile latency | large_prohibited_terms |     212399 |      ms |
|   All |                                   50th percentile service time | large_prohibited_terms |    1220.91 |      ms |
|   All |                                   90th percentile service time | large_prohibited_terms |    1261.77 |      ms |
|   All |                                   99th percentile service time | large_prohibited_terms |     1294.9 |      ms |
|   All |                                  100th percentile service time | large_prohibited_terms |    1376.22 |      ms |
|   All |                                                     error rate | large_prohibited_terms |          0 |       % |


# 参考

* [github.com/elastic/rally](https://github.com/elastic/rally)
* [Rally 1.0.4 Docs](https://esrally.readthedocs.io/en/stable/)