【Elasticsearch】ベンチマークツールRally実行環境構築

# TL;DR
- Elasticsearch公式のベンチマークRallyをAWS EC2上で実行できる環境を構築します。
    - https://www.elastic.co/jp/blog/announcing-rally-benchmarking-for-elasticsearch

# 環境

- AMI
    - Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type (ami-09d4b65ff082c3c6a)
- EC2
    - r6g.large

# Elasticsearch環境構築
- javaインストール

```bash
sudo yum install -y java-11-amazon-corretto-headless
```

- Elasticsearchインストールの準備
    - `/etc/yum.repos.d/`の下に以下のようなファイルを作成する。

```elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
```

- その後`yum`でインストール+サービス起動

```bash
# Elasticsearch インストール
sudo yum install -y --enablerepo=elasticsearch elasticsearch
# systemctl設定ファイル再読み込み
sudo systemctl daemon-reload
# Elasticsearchサービス有効化
sudo systemctl enable elasticsearch.service
# Elasticsearchサービス起動
sudo systemctl start elasticsearch.service
```

# Rally実行環境構築
- gitインストール

```bash
sudo yum install -y git
```

- gccインストール

```bash
sudo yum install -y gcc
```

- python関連

```bash
sudo yum install -y python3-devel
sudo amazon-linux-extras install -y epel
pip3 install --upgrade pip
```

- Rallyインストール

```bash
pip3 install esrally
esrally configure
esrally list tracks
```

# 実行方法

- ベンチマーク一覧表示

```bash
esrally list tracks
```

```shell:ベンチマーク一覧
    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

Available tracks:

Name           Description                                                              Documents    Compressed Size    Uncompressed Size    Default Challenge        All Challenges
-------------  -----------------------------------------------------------------------  -----------  -----------------  -------------------  -----------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
eql            EQL benchmarks based on endgame index of SIEM demo cluster               60,782,211   4.5 GB             109.2 GB             default                  default
geonames       POIs from Geonames                                                       11,396,503   252.9 MB           3.3 GB               append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts,significant-text
geopoint       Point coordinates from PlanetOSM                                         60,844,404   482.1 MB           2.3 GB               append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geopointshape  Point coordinates from PlanetOSM indexed as geoshapes                    60,844,404   470.8 MB           2.6 GB               append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,append-fast-with-conflicts
geoshape       Shapes from PlanetOSM                                                    60,523,283   13.4 GB            45.4 GB              append-no-conflicts      append-no-conflicts
http_logs      HTTP server log data                                                     247,249,096  1.2 GB             31.1 GB              append-no-conflicts      append-no-conflicts,runtime-fields,append-no-conflicts-index-only,append-sorted-no-conflicts,append-index-only-with-ingest-pipeline,update,append-no-conflicts-index-reindex-only
metricbeat     Metricbeat data                                                          1,079,600    87.7 MB            1.2 GB               append-no-conflicts      append-no-conflicts
nested         StackOverflow Q&A stored as nested docs                                  11,203,029   663.3 MB           3.4 GB               nested-search-challenge  nested-search-challenge,index-only
noaa           Global daily weather measurements from NOAA                              33,659,481   949.4 MB           9.0 GB               append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,top_metrics,aggs,sql
nyc_taxis      Taxi rides in New York in 2015                                           165,346,692  4.5 GB             74.3 GB              append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts-index-only,update,append-ml,date-histogram,indexing-querying
percolator     Percolator benchmark based on AOL queries                                2,000,000    121.1 kB           104.9 MB             append-no-conflicts      append-no-conflicts
pmc            Full text benchmark with academic papers from PMC                        574,199      5.5 GB             21.7 GB              append-no-conflicts      append-no-conflicts,append-no-conflicts-index-only,append-sorted-no-conflicts,append-fast-with-conflicts,indexing-querying
so             Indexing benchmark using up to questions and answers from StackOverflow  36,062,278   8.9 GB             33.1 GB              append-no-conflicts      append-no-conflicts

-------------------------------
[INFO] SUCCESS (took 2 seconds)
-------------------------------
```

- ベンチマークテスト実行

```
# ベンチマークpmcを実行
esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only
```

```shell:結果
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


************************************************************************
************** WARNING: A dark dungeon lies ahead of you  **************
************************************************************************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

************************************************************************
****** Use this pipeline only if you are aware of the tradeoffs.  ******
*************************** Watch your step! ***************************
************************************************************************

[INFO] Downloading data for track pmc (5.5 GB total size)                         [100.0%]
[INFO] Decompressing track data from [/home/ec2-user/.rally/benchmarks/data/pmc/documents.json.bz2] to [/home/ec2-user/.rally/benchmarks/data/pmc/documents.json] (resulting size: 21.66 GB) ... [OK]
[INFO] Preparing file offset table for [/home/ec2-user/.rally/benchmarks/data/pmc/documents.json] ... [OK]
[INFO] Racing on track [pmc], challenge [append-no-conflicts] and car ['external'] with version [7.15.2].

[WARNING] indexing_total_time is 1261 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 103 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 221 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running put-settings                                                           [100% done]
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running articles_monthly_agg_uncached                                          [100% done]
Running articles_monthly_agg_cached                                            [100% done]
Running scroll                                                                 [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                          Task |       Value |    Unit |
|---------------------------------------------------------------:|------------------------------:|------------:|--------:|
|                     Cumulative indexing time of primary shards |                               |     46.5094 |     min |
|             Min cumulative indexing time across primary shards |                               |   0.0210167 |     min |
|          Median cumulative indexing time across primary shards |                               |      8.9634 |     min |
|             Max cumulative indexing time across primary shards |                               |     10.1053 |     min |
|            Cumulative indexing throttle time of primary shards |                               |           0 |     min |
|    Min cumulative indexing throttle time across primary shards |                               |           0 |     min |
| Median cumulative indexing throttle time across primary shards |                               |           0 |     min |
|    Max cumulative indexing throttle time across primary shards |                               |           0 |     min |
|                        Cumulative merge time of primary shards |                               |     3.39033 |     min |
|                       Cumulative merge count of primary shards |                               |           1 |         |
|                Min cumulative merge time across primary shards |                               |           0 |     min |
|             Median cumulative merge time across primary shards |                               |           0 |     min |
|                Max cumulative merge time across primary shards |                               |     3.39033 |     min |
|               Cumulative merge throttle time of primary shards |                               |      1.7924 |     min |
|       Min cumulative merge throttle time across primary shards |                               |           0 |     min |
|    Median cumulative merge throttle time across primary shards |                               |           0 |     min |
|       Max cumulative merge throttle time across primary shards |                               |      1.7924 |     min |
|                      Cumulative refresh time of primary shards |                               |     7.68072 |     min |
|                     Cumulative refresh count of primary shards |                               |         121 |         |
|              Min cumulative refresh time across primary shards |                               |  0.00171667 |     min |
|           Median cumulative refresh time across primary shards |                               |    0.918117 |     min |
|              Max cumulative refresh time across primary shards |                               |     2.67948 |     min |
|                        Cumulative flush time of primary shards |                               |      12.617 |     min |
|                       Cumulative flush count of primary shards |                               |          49 |         |
|                Min cumulative flush time across primary shards |                               |  0.00368333 |     min |
|             Median cumulative flush time across primary shards |                               |     2.32497 |     min |
|                Max cumulative flush time across primary shards |                               |     3.49425 |     min |
|                                             Total Young Gen GC |                               |       3.805 |       s |
|                                               Total Old Gen GC |                               |           0 |       s |
|                                                     Store size |                               |     19.0909 |      GB |
|                                                  Translog size |                               | 3.07336e-07 |      GB |
|                                         Heap used for segments |                               |    0.386272 |      MB |
|                                       Heap used for doc values |                               |   0.0587959 |      MB |
|                                            Heap used for terms |                               |    0.224182 |      MB |
|                                            Heap used for norms |                               |    0.038147 |      MB |
|                                           Heap used for points |                               |           0 |      MB |
|                                    Heap used for stored fields |                               |   0.0651474 |      MB |
|                                                  Segment count |                               |         131 |         |
|                                                 Min Throughput |                  index-append |      357.63 |  docs/s |
|                                              Median Throughput |                  index-append |      377.31 |  docs/s |
|                                                 Max Throughput |                  index-append |      387.36 |  docs/s |
|                                        50th percentile latency |                  index-append |     9729.85 |      ms |
|                                        90th percentile latency |                  index-append |     12174.4 |      ms |
|                                        99th percentile latency |                  index-append |     14831.7 |      ms |
|                                       100th percentile latency |                  index-append |     21458.7 |      ms |
|                                   50th percentile service time |                  index-append |     9729.85 |      ms |
|                                   90th percentile service time |                  index-append |     12174.4 |      ms |
|                                   99th percentile service time |                  index-append |     14831.7 |      ms |
|                                  100th percentile service time |                  index-append |     21458.7 |      ms |
|                                                     error rate |                  index-append |           0 |       % |
|                                                 Min Throughput |                       default |       20.02 |   ops/s |
|                                              Median Throughput |                       default |       20.03 |   ops/s |
|                                                 Max Throughput |                       default |       20.03 |   ops/s |
|                                        50th percentile latency |                       default |     7.93665 |      ms |
|                                        90th percentile latency |                       default |     8.37838 |      ms |
|                                        99th percentile latency |                       default |     13.7058 |      ms |
|                                       100th percentile latency |                       default |     21.8118 |      ms |
|                                   50th percentile service time |                       default |     7.87168 |      ms |
|                                   90th percentile service time |                       default |     8.30782 |      ms |
|                                   99th percentile service time |                       default |     13.6445 |      ms |
|                                  100th percentile service time |                       default |     21.7493 |      ms |
|                                                     error rate |                       default |           0 |       % |
|                                                 Min Throughput |                          term |       20.02 |   ops/s |
|                                              Median Throughput |                          term |       20.03 |   ops/s |
|                                                 Max Throughput |                          term |       20.03 |   ops/s |
|                                        50th percentile latency |                          term |     8.38097 |      ms |
|                                        90th percentile latency |                          term |     9.26057 |      ms |
|                                        99th percentile latency |                          term |     13.0642 |      ms |
|                                       100th percentile latency |                          term |     14.2108 |      ms |
|                                   50th percentile service time |                          term |      8.3186 |      ms |
|                                   90th percentile service time |                          term |     9.19805 |      ms |
|                                   99th percentile service time |                          term |     13.0041 |      ms |
|                                  100th percentile service time |                          term |      14.148 |      ms |
|                                                     error rate |                          term |           0 |       % |
|                                                 Min Throughput |                        phrase |       20.02 |   ops/s |
|                                              Median Throughput |                        phrase |       20.03 |   ops/s |
|                                                 Max Throughput |                        phrase |       20.03 |   ops/s |
|                                        50th percentile latency |                        phrase |     8.09686 |      ms |
|                                        90th percentile latency |                        phrase |     9.12908 |      ms |
|                                        99th percentile latency |                        phrase |     11.3293 |      ms |
|                                       100th percentile latency |                        phrase |     15.1762 |      ms |
|                                   50th percentile service time |                        phrase |      8.0343 |      ms |
|                                   90th percentile service time |                        phrase |      9.0663 |      ms |
|                                   99th percentile service time |                        phrase |     11.2656 |      ms |
|                                  100th percentile service time |                        phrase |     15.1135 |      ms |
|                                                     error rate |                        phrase |           0 |       % |
|                                                 Min Throughput | articles_monthly_agg_uncached |       20.02 |   ops/s |
|                                              Median Throughput | articles_monthly_agg_uncached |       20.02 |   ops/s |
|                                                 Max Throughput | articles_monthly_agg_uncached |       20.02 |   ops/s |
|                                        50th percentile latency | articles_monthly_agg_uncached |     20.4212 |      ms |
|                                        90th percentile latency | articles_monthly_agg_uncached |     22.3701 |      ms |
|                                        99th percentile latency | articles_monthly_agg_uncached |     29.1576 |      ms |
|                                       100th percentile latency | articles_monthly_agg_uncached |     29.3201 |      ms |
|                                   50th percentile service time | articles_monthly_agg_uncached |     20.3609 |      ms |
|                                   90th percentile service time | articles_monthly_agg_uncached |     22.3085 |      ms |
|                                   99th percentile service time | articles_monthly_agg_uncached |     29.0933 |      ms |
|                                  100th percentile service time | articles_monthly_agg_uncached |     29.2551 |      ms |
|                                                     error rate | articles_monthly_agg_uncached |           0 |       % |
|                                                 Min Throughput |   articles_monthly_agg_cached |       20.03 |   ops/s |
|                                              Median Throughput |   articles_monthly_agg_cached |       20.03 |   ops/s |
|                                                 Max Throughput |   articles_monthly_agg_cached |       20.04 |   ops/s |
|                                        50th percentile latency |   articles_monthly_agg_cached |     2.88189 |      ms |
|                                        90th percentile latency |   articles_monthly_agg_cached |     3.02556 |      ms |
|                                        99th percentile latency |   articles_monthly_agg_cached |      5.2449 |      ms |
|                                       100th percentile latency |   articles_monthly_agg_cached |     6.40146 |      ms |
|                                   50th percentile service time |   articles_monthly_agg_cached |     2.82049 |      ms |
|                                   90th percentile service time |   articles_monthly_agg_cached |     2.96245 |      ms |
|                                   99th percentile service time |   articles_monthly_agg_cached |     5.18323 |      ms |
|                                  100th percentile service time |   articles_monthly_agg_cached |     6.33876 |      ms |
|                                                     error rate |   articles_monthly_agg_cached |           0 |       % |
|                                                 Min Throughput |                        scroll |       12.54 | pages/s |
|                                              Median Throughput |                        scroll |       12.56 | pages/s |
|                                                 Max Throughput |                        scroll |       12.61 | pages/s |
|                                        50th percentile latency |                        scroll |      1083.9 |      ms |
|                                        90th percentile latency |                        scroll |     1097.44 |      ms |
|                                        99th percentile latency |                        scroll |     1136.81 |      ms |
|                                       100th percentile latency |                        scroll |     1139.19 |      ms |
|                                   50th percentile service time |                        scroll |     1082.96 |      ms |
|                                   90th percentile service time |                        scroll |      1096.5 |      ms |
|                                   99th percentile service time |                        scroll |     1135.88 |      ms |
|                                  100th percentile service time |                        scroll |     1138.25 |      ms |
|                                                     error rate |                        scroll |           0 |       % |


----------------------------------
[INFO] SUCCESS (took 3413 seconds)
----------------------------------
```

# 参考資料
- Rally登場：Elasticsearchのベンチマークツール
    - [Announcing Rally: Our benchmarking tool for Elasticsearch \| Elastic Blog](https://www.elastic.co/jp/blog/announcing-rally-benchmarking-for-elasticsearch)
- Rally github
    - [elastic/rally: Macrobenchmarking framework for Elasticsearch](https://github.com/elastic/rally)
- Elasticsearch公式
    - [Starting Elasticsearch \| Elasticsearch Guide \[7\.15\] \| Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html)
    - [Install Elasticsearch with RPM \| Elasticsearch Guide \[7\.15\] \| Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html)