IBM Cloud の Looking Glass (ゾーン間レイテンシの表) を自作する


# Looking Glass とは

こちらで公開されているプライベートネットワークバックボーンを経由した各クラウドゾーン間レイテンシの表です。
この表の内容を実際に測定して確認します。

- [SoftLayer® Technologies, Inc. :: IP Backbone Looking Glass](http://lg.softlayer.com/)

![](http://lg.softlayer.com/images/latency_maxtrix.png)

# Perfkit Benchmarker を使う
各クラウドを横断して同一のベンチマークをおこなえるツール `Perfkit Benchmarker` が公開されています。
こちらを活用したいところですが、残念ながら IBM Cloud は公式対応していません。

>[PerfKitBenchmarker: PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see.](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker)
>PerfKit Benchmarkerは、一般的なベンチマークツールに関するラッパーとワークロード定義を提供します。 私たちは可能な限りすべてを使用し、自動化することを非常に簡単にしました。 選択したクラウドプロバイダーでVMをインスタンス化し、自動的にベンチマークをインストールし、ユーザーの介入なしにワークロードを実行します。

ただ、本来の `Perfkit` をベースとして IBM Cloud (旧 SoftLayer) 用にカスタマイズしている人を発見しました。
[jbd214/PerfKitBenchmarker](https://github.com/jbd214/PerfKitBenchmarker) を試しましたが、少し古く、いくつか不具合があったので、それらを修正して動かせるようにして公開しました。
今回はこのレポジトリを使って `ping` ベンチマークを実施し、Looking Glass を自作します。

>[khayama/PerfKitBenchmarker](https://github.com/khayama/PerfKitBenchmarker)

#環境準備
`Ubuntu 16.04.6 LTS` の仮想サーバーを IBM Cloud で準備します。

```
# cat /etc/os-release | grep PRETTY
PRETTY_NAME="Ubuntu 16.04.6 LTS"
```

古いですが、 `Python 2.7` をインストールします。

```
# apt update
# apt install python -y
# python -V
Python 2.7.12
```

続いて `pip` をインストール・アップグレードします。

```
# apt install python-pip -y
# pip install --upgrade pip
# python -m pip -V
pip 20.0.2 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
```

後でエラーが出ないようにロケールを設定しておきます。

```
# export LC_ALL="en_US.UTF-8"
# export LC_CTYPE="en_US.UTF-8"
# dpkg-reconfigure locales
```

カスタマイズした `Perfkit` レポジトリを `git clone` して、必要なモジュールをインストールします。

```
# git clone https://github.com/khayama/PerfKitBenchmarker.git
# cd PerfKitBenchmarker
# python -m pip install -r requirements.txt
# python -m pip install -r requirements-softlayer.txt
```

`SoftLayer CLI` がインストールされていることを確認します。



```
# slcli --version
slcli (SoftLayer Command-line), version 5.8.0
```

[IaaS API key の発行・取得](https://qiita.com/khayama/items/d7efe0412341164c17dc#iaas-api-key-%E3%81%AE%E7%99%BA%E8%A1%8C%E5%8F%96%E5%BE%97) をし、[「slcli」のセットアップ](https://qiita.com/khayama/items/d7efe0412341164c17dc#slcli%E3%81%AE%E3%82%BB%E3%83%83%E3%83%88%E3%82%A2%E3%83%83%E3%83%97) をおこないます。

```
# slcli setup
Username [xxxxx]: xxxxx@example.com
API Key or Password [xxxxx]: 
Endpoint (public|private|custom) [https://api.softlayer.com/xmlrpc/v3.1]: 
Timeout [0]: 
:..............:..................................................................:
:         name : value                                                            :
:..............:..................................................................:
:     Username : xxxxx@example.com                                       :
:      API Key : xxxxx :
: Endpoint URL : https://api.softlayer.com/xmlrpc/v3.1                            :
:      Timeout : not set                                                          :
:..............:..................................................................:
Are you sure you want to write settings to "/Users/khayama/.softlayer"? [Y/n]: y
Configuration Updated Successfully
```

# 1ゾーン内 `ping` ベンチマーク
1つのゾーン（データセンター）を指定して、2つの仮想サーバーをプロビジョニングして、その間のネットワークレイテンシを測定します。
以下のコマンドで実行でき、双方向の測定が10分弱で完了します。
実行状況をみると、`ping` はデフォルトでお互いに `10.x.x.x` のプライベートネットワークを使って100回ずつ送り合います。

```
# ログアウトしても継続したい場合は nohup ./pkb.py *** & のようにする
# 進行状況は tail -f nohup.out で確認できる

./pkb.py --owner=khayama --cloud=SoftLayer --benchmarks=ping --zones=tok04 --machine_type="{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }"

...

-------------------------PerfKitBenchmarker Results Summary-------------------------
PING:
  ip_type="internal" receiving_zone="tok04" sending_zone="tok04"
  Min Latency                           0.232000 ms                            
  Average Latency                       0.466000 ms                            
  Max Latency                          11.328000 ms                            
  Latency Std Dev                       1.134000 ms                            
  Min Latency                           0.235000 ms                            
  Average Latency                       0.791000 ms                            
  Max Latency                          17.826000 ms                            
  Latency Std Dev                       2.462000 ms                            
  End to End Runtime                  490.835629 seconds                       

-------------------------
```

結果はこのように `json` で出ます。
（さすがに同一ゾーン内のレイテンシは1ms以下ですね）

```
# cat /tmp/perfkitbenchmarker/runs/a30d3301/perfkitbenchmarker_results.json
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609679.887866, "metric": "Min Latency", "official": false, "value": 0.232, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "215bdc5b-4cd9-44f1-8ec2-a07ba11848a1", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609679.887904, "metric": "Average Latency", "official": false, "value": 0.466, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "a2498fb5-34ab-4a7a-91c0-e57a73c27692", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609679.887919, "metric": "Max Latency", "official": false, "value": 11.328, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "a22728ea-b080-42dd-8100-3b254d106cac", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609679.887925, "metric": "Latency Std Dev", "official": false, "value": 1.134, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "e395feb6-4e72-4550-afba-7c07cf931de4", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609782.456322, "metric": "Min Latency", "official": false, "value": 0.235, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "602bc550-f618-464c-a7a0-92efe7c75e28", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609782.456368, "metric": "Average Latency", "official": false, "value": 0.791, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "ce6dcde1-4be0-4795-84db-20c0f94e938d", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609782.45638, "metric": "Max Latency", "official": false, "value": 17.826, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "5d4ab50e-e06a-416f-9591-7a0d22fe2fc9", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|receiving_zone:tok04|,|ip_type:internal|,|vm_2_zone:tok04|,|vm_1_zone:tok04|,|sending_zone:tok04|", "timestamp": 1581609782.456386, "metric": "Latency Std Dev", "official": false, "value": 2.462, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "b8761e94-a20e-4fd4-9deb-cca32d91ad57", "product_name": "PerfKitBenchmarker", "unit": "ms"}
{"labels": "|vm_2_image:None|,|vm_1_cloud:SoftLayer|,|vm_2_cloud:SoftLayer|,|vm_1_vm_count:1|,|vm_2_vm_count:1|,|perfkitbenchmarker_version:v1.4.0-754-gfb815ba|,|vm_1_image:None|,|vm_1_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_machine_type:{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }|,|vm_2_zone:tok04|,|vm_1_zone:tok04|", "timestamp": 1581609876.880598, "metric": "End to End Runtime", "official": false, "value": 490.83562898635864, "owner": "khayama", "run_uri": "a30d3301-893ddf3a-04b6-4d65-926c-2646f6835ed4", "test": "ping", "sample_uri": "d2bf90ba-13e7-4c77-95d2-c38873621fd2", "product_name": "PerfKitBenchmarker", "unit": "seconds"}
```

# 2ゾーン間 `ping` ベンチマーク
2つのゾーン（データセンター）を指定して、それぞれに仮想サーバーをプロビジョニングして、その間のネットワークレイテンシを測定します。
以下のコマンドで実行でき、双方向の測定が10分弱で完了します。

```
./pkb.py --owner=khayama --cloud=SoftLayer --benchmarks=ping --zones=tok04,tok05 --machine_type="{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }"
```

# 3ゾーン間1リージョン内 `ping` ベンチマーク
3つのゾーン（データセンター）を指定して、それぞれに仮想サーバーをプロビジョニングして、その間のネットワークレイテンシを測定します。
`config_file` として、東京リージョンの場合は以下の `yaml` ファイルを使います。

```yaml:tok_region_latency.yaml
ping:
  flag_matrix: inter_zone
  flag_matrix_filters:
    inter_zone: "zones < extra_zones"
  flag_matrix_defs:
    inter_zone:    
      zones: [tok02,tok04,tok05]
      extra_zones: [tok02,tok04,tok05]
      
  flags:
    owner: khayama
    cloud: SoftLayer
    machine_type: '{"cpus": 2, "memory": 4096, "os": "UBUNTU_LATEST_64", "nic": 1000 }'
```

以下のコマンドで実行でき、双方向の測定が30分弱で完了します。
`--run_processes=<# of processes>` を使った並列実行はたびたびエラーになるので、時間はかかりますが、なるべく使わない方針です。

```
./pkb.py --benchmarks=ping --benchmark_config_file=tok_region_latency.yaml --run_processes=1
```

・tok02 <--> tok04
・tok02 <--> tok05
・tok04 <--> tok05
という形で進んでいきますが、
結果の表示は以下のコマンドで確認できます。
（同一リージョン内のレイテンシは2ms以下と思っていれば良さそうです。）

```
# python show_tok_table.py /tmp/perfkitbenchmarker/runs/b1be6e3f/perfkitbenchmarker_results.json
/tmp/perfkitbenchmarker/runs/b1be6e3f/perfkitbenchmarker_results.json
0 : tok02 --> tok04 : 1.482 ms
1 : tok02 --> tok05 : 1.582 ms
2 : tok04 --> tok05 : 2.005 ms
3 : tok04 --> tok02 : 1.59 ms
4 : tok05 --> tok02 : 1.773 ms
5 : tok05 --> tok04 : 1.671 ms
+-------+-------+-------+-------+
|  (ms) | tok02 | tok04 | tok05 |
+-------+-------+-------+-------+
| tok02 |   0   | 1.482 | 1.582 |
| tok04 |  1.59 |   0   | 2.005 |
| tok05 | 1.773 | 1.671 |   0   |
+-------+-------+-------+-------+
```

# Looking Glass を作る
グローバルの20ゾーン（データセンター）を指定して、それぞれに仮想サーバーをプロビジョニングして、その間のネットワークレイテンシを測定します。
`config_file` は以下の `yaml` ファイルを使います。

```yaml:all_region_latency.yaml 
ping:
  flag_matrix: inter_zone
  flag_matrix_filters:
    inter_zone: "zones < extra_zones"
  flag_matrix_defs:
    inter_zone:    
      zones: [dal13,hou02,mex01,mon01,sea01,sjc04,tor01,wdc07,ams03,fra05,lon02,mil01,par01,che01,hkg02,mel01,sng01,syd05,tok05,sao01]
      extra_zones: [dal13,hou02,mex01,mon01,sea01,sjc04,tor01,wdc07,ams03,fra05,lon02,mil01,par01,che01,hkg02,mel01,sng01,syd05,tok05,sao01]
      
  flags:
    owner: khayama
    cloud: SoftLayer
    machine_type: '{"cpus": 2, "memory": 4096, "os": "UBUNTU_LATEST_64", "nic": 1000 }'
```

以下のコマンドで実行でき、双方向の測定には、合計で31時間かかりました...！！！

```
./pkb.py --benchmarks=ping --benchmark_config_file=all_region_latency.yaml --run_processes=1
```

結果の表示は以下のコマンドで確認できます。
（ようやく Looking Glass と同じものが作れました。）

```
# python show_all_table.py /tmp/perfkitbenchmarker/runs/4bc6ff85/perfkitbenchmarker_results.json
/tmp/perfkitbenchmarker/runs/4bc6ff85/perfkitbenchmarker_results.json
0 : dal13 --> hou02 : 7.256 ms
1 : dal13 --> mex01 : 26.238 ms
2 : dal13 --> mon01 : 40.005 ms
3 : dal13 --> sea01 : 43.206 ms
4 : dal13 --> sjc04 : 38.092 ms
5 : dal13 --> tor01 : 35.594 ms
6 : dal13 --> wdc07 : 31.586 ms
7 : dal13 --> fra05 : 120.104 ms
8 : dal13 --> lon02 : 112.667 ms
9 : dal13 --> mil01 : 133.501 ms
10 : dal13 --> par01 : 121.019 ms
11 : dal13 --> hkg02 : 184.305 ms
12 : dal13 --> mel01 : 192.661 ms
13 : dal13 --> sng01 : 212.551 ms
14 : dal13 --> syd05 : 184.427 ms
15 : dal13 --> tok05 : 138.281 ms
16 : dal13 --> sao01 : 143.799 ms
17 : hou02 --> mex01 : 30.778 ms
18 : hou02 --> mon01 : 47.092 ms
19 : hou02 --> sea01 : 50.598 ms
20 : hou02 --> sjc04 : 43.652 ms
21 : hou02 --> tor01 : 40.731 ms
22 : hou02 --> wdc07 : 39.587 ms
23 : hou02 --> lon02 : 114.283 ms
24 : hou02 --> mil01 : 139.616 ms
25 : hou02 --> par01 : 121.255 ms
26 : hou02 --> mel01 : 196.531 ms
27 : hou02 --> sng01 : 218.734 ms
28 : hou02 --> syd05 : 181.815 ms
29 : hou02 --> tok05 : 142.271 ms
30 : hou02 --> sao01 : 138.242 ms
31 : mex01 --> mon01 : 66.553 ms
32 : mex01 --> sea01 : 66.677 ms
33 : mex01 --> sjc04 : 56.408 ms
34 : mex01 --> tor01 : 56.275 ms
35 : mex01 --> wdc07 : 55.046 ms
36 : mex01 --> mil01 : 158.434 ms
37 : mex01 --> par01 : 143.564 ms
38 : mex01 --> sng01 : 231.667 ms
39 : mex01 --> syd05 : 202.325 ms
40 : mex01 --> tok05 : 154.661 ms
41 : mex01 --> sao01 : 165.846 ms
42 : mon01 --> sea01 : 59.933 ms
43 : mon01 --> sjc04 : 68.353 ms
44 : mon01 --> tor01 : 9.497 ms
45 : mon01 --> wdc07 : 15.68 ms
46 : mon01 --> par01 : 85.905 ms
47 : mon01 --> sng01 : 252.976 ms
48 : mon01 --> syd05 : 211.901 ms
49 : mon01 --> tok05 : 141.489 ms
50 : mon01 --> sao01 : 118.688 ms
51 : sea01 --> sjc04 : 18.518 ms
52 : sea01 --> tor01 : 52.589 ms
53 : sea01 --> wdc07 : 54.738 ms
54 : sea01 --> sng01 : 161.584 ms
55 : sea01 --> syd05 : 173.491 ms
56 : sea01 --> tok05 : 82.771 ms
57 : sjc04 --> tor01 : 62.312 ms
58 : sjc04 --> wdc07 : 58.862 ms
59 : sjc04 --> sng01 : 179.015 ms
60 : sjc04 --> syd05 : 160.192 ms
61 : sjc04 --> tok05 : 99.626 ms
62 : tor01 --> wdc07 : 21.594 ms
63 : ams03 --> dal13 : 113.195 ms
64 : ams03 --> hou02 : 116.855 ms
65 : ams03 --> mex01 : 136.166 ms
66 : ams03 --> mon01 : 85.181 ms
67 : ams03 --> sea01 : 137.17 ms
68 : ams03 --> sjc04 : 147.164 ms
69 : ams03 --> tor01 : 90.811 ms
70 : ams03 --> wdc07 : 80.328 ms
71 : ams03 --> fra05 : 7.209 ms
72 : ams03 --> lon02 : 8.132 ms
73 : ams03 --> mil01 : 31.5 ms
74 : ams03 --> par01 : 11.914 ms
75 : ams03 --> che01 : 141.796 ms
76 : ams03 --> hkg02 : 200.439 ms
77 : ams03 --> mel01 : 259.803 ms
78 : ams03 --> sng01 : 170.338 ms
79 : ams03 --> syd05 : 261.095 ms
80 : ams03 --> tok05 : 250.645 ms
81 : ams03 --> sao01 : 184.37 ms
82 : fra05 --> hou02 : 127.918 ms
83 : fra05 --> mex01 : 145.33 ms
84 : fra05 --> mon01 : 87.989 ms
85 : fra05 --> sea01 : 139.923 ms
86 : fra05 --> sjc04 : 145.483 ms
87 : fra05 --> tor01 : 94.046 ms
88 : fra05 --> wdc07 : 83.923 ms
89 : fra05 --> lon02 : 11.906 ms
90 : fra05 --> mil01 : 15.119 ms
91 : fra05 --> par01 : 10.522 ms
92 : fra05 --> hkg02 : 179.752 ms
93 : fra05 --> mel01 : 239.877 ms
94 : fra05 --> sng01 : 150.373 ms
95 : fra05 --> syd05 : 243.415 ms
96 : fra05 --> tok05 : 232.297 ms
97 : fra05 --> sao01 : 188.08 ms
98 : lon02 --> mex01 : 135.834 ms
99 : lon02 --> mon01 : 78.705 ms
100 : lon02 --> sea01 : 130.216 ms
101 : lon02 --> sjc04 : 141.153 ms
102 : lon02 --> tor01 : 84.835 ms
103 : lon02 --> wdc07 : 74.385 ms
104 : lon02 --> mil01 : 25.081 ms
105 : lon02 --> par01 : 8.66 ms
106 : lon02 --> mel01 : 247.249 ms
107 : lon02 --> sng01 : 160.918 ms
108 : lon02 --> syd05 : 257.151 ms
109 : lon02 --> tok05 : 245.101 ms
110 : lon02 --> sao01 : 178.271 ms
111 : mil01 --> mon01 : 101.386 ms
112 : mil01 --> sea01 : 153.724 ms
113 : mil01 --> sjc04 : 160.178 ms
114 : mil01 --> tor01 : 106.34 ms
115 : mil01 --> wdc07 : 107.832 ms
116 : mil01 --> par01 : 19.597 ms
117 : mil01 --> sng01 : 145.923 ms
118 : mil01 --> syd05 : 235.503 ms
119 : mil01 --> tok05 : 226.058 ms
120 : mil01 --> sao01 : 202.542 ms
121 : par01 --> sea01 : 138.359 ms
122 : par01 --> sjc04 : 146.118 ms
123 : par01 --> tor01 : 91.271 ms
124 : par01 --> wdc07 : 91.098 ms
125 : par01 --> sng01 : 157.512 ms
126 : par01 --> syd05 : 248.321 ms
127 : par01 --> tok05 : 243.18 ms
128 : par01 --> sao01 : 185.954 ms
129 : che01 --> dal13 : 244.962 ms
130 : che01 --> hou02 : 250.765 ms
131 : che01 --> mex01 : 264.027 ms
132 : che01 --> mon01 : 209.603 ms
133 : che01 --> sea01 : 193.547 ms
134 : che01 --> sjc04 : 210.561 ms
135 : che01 --> tor01 : 219.908 ms
136 : che01 --> wdc07 : 218.152 ms
137 : che01 --> fra05 : 129.511 ms
138 : che01 --> lon02 : 133.673 ms
139 : che01 --> mil01 : 141.442 ms
140 : che01 --> par01 : 128.767 ms
141 : che01 --> hkg02 : 64.047 ms
142 : che01 --> mel01 : 119.488 ms
143 : che01 --> sng01 : 33.438 ms
144 : che01 --> syd05 : 123.781 ms
145 : che01 --> tok05 : 112.931 ms
146 : che01 --> sao01 : 313.851 ms
147 : hkg02 --> hou02 : 188.921 ms
148 : hkg02 --> mex01 : 201.986 ms
149 : hkg02 --> mon01 : 221.088 ms
150 : hkg02 --> sea01 : 130.83 ms
151 : hkg02 --> sjc04 : 147.253 ms
152 : hkg02 --> tor01 : 212.801 ms
153 : hkg02 --> wdc07 : 215.524 ms
154 : hkg02 --> lon02 : 195.249 ms
155 : hkg02 --> mil01 : 173.019 ms
156 : hkg02 --> par01 : 190.937 ms
157 : hkg02 --> mel01 : 117.286 ms
158 : hkg02 --> sng01 : 31.835 ms
159 : hkg02 --> syd05 : 115.035 ms
160 : hkg02 --> tok05 : 49.722 ms
161 : hkg02 --> sao01 : 325.825 ms
162 : mel01 --> mex01 : 210.198 ms
163 : mel01 --> mon01 : 222.079 ms
164 : mel01 --> sea01 : 183.411 ms
165 : mel01 --> sjc04 : 167.759 ms
166 : mel01 --> tor01 : 222.527 ms
167 : mel01 --> wdc07 : 221.849 ms
168 : mel01 --> mil01 : 236.014 ms
169 : mel01 --> par01 : 246.058 ms
170 : mel01 --> sng01 : 87.303 ms
171 : mel01 --> syd05 : 16.783 ms
172 : mel01 --> tok05 : 131.454 ms
173 : mel01 --> sao01 : 338.83 ms
174 : sng01 --> tor01 : 243.057 ms
175 : sng01 --> wdc07 : 244.193 ms
176 : sng01 --> syd05 : 91.73 ms
177 : sng01 --> tok05 : 80.319 ms
178 : syd05 --> tor01 : 210.123 ms
179 : syd05 --> wdc07 : 214.317 ms
180 : syd05 --> tok05 : 113.862 ms
181 : tok05 --> tor01 : 133.838 ms
182 : tok05 --> wdc07 : 136.072 ms
183 : sao01 --> sea01 : 170.654 ms
184 : sao01 --> sjc04 : 181.788 ms
185 : sao01 --> tor01 : 125.015 ms
186 : sao01 --> wdc07 : 115.295 ms
187 : sao01 --> sng01 : 354.324 ms
188 : sao01 --> syd05 : 329.638 ms
189 : sao01 --> tok05 : 279.892 ms
190 : hou02 --> dal13 : 7.582 ms
191 : mex01 --> dal13 : 25.747 ms
192 : mon01 --> dal13 : 39.897 ms
193 : sea01 --> dal13 : 43.543 ms
194 : sjc04 --> dal13 : 37.135 ms
195 : tor01 --> dal13 : 35.604 ms
196 : wdc07 --> dal13 : 31.537 ms
197 : fra05 --> dal13 : 119.863 ms
198 : lon02 --> dal13 : 112.408 ms
199 : mil01 --> dal13 : 133.883 ms
200 : par01 --> dal13 : 121.507 ms
201 : hkg02 --> dal13 : 184.514 ms
202 : mel01 --> dal13 : 194.484 ms
203 : sng01 --> dal13 : 213.348 ms
204 : syd05 --> dal13 : 184.378 ms
205 : tok05 --> dal13 : 138.176 ms
206 : sao01 --> dal13 : 143.805 ms
207 : mex01 --> hou02 : 30.752 ms
208 : mon01 --> hou02 : 46.794 ms
209 : sea01 --> hou02 : 50.761 ms
210 : sjc04 --> hou02 : 44.976 ms
211 : tor01 --> hou02 : 39.393 ms
212 : wdc07 --> hou02 : 39.2 ms
213 : lon02 --> hou02 : 114.28 ms
214 : mil01 --> hou02 : 141.214 ms
215 : par01 --> hou02 : 121.156 ms
216 : mel01 --> hou02 : 196.692 ms
217 : sng01 --> hou02 : 218.723 ms
218 : syd05 --> hou02 : 181.325 ms
219 : tok05 --> hou02 : 142.607 ms
220 : sao01 --> hou02 : 138.63 ms
221 : mon01 --> mex01 : 67.569 ms
222 : sea01 --> mex01 : 66.673 ms
223 : sjc04 --> mex01 : 56.036 ms
224 : tor01 --> mex01 : 56.755 ms
225 : wdc07 --> mex01 : 55.008 ms
226 : mil01 --> mex01 : 155.67 ms
227 : par01 --> mex01 : 143.445 ms
228 : sng01 --> mex01 : 231.598 ms
229 : syd05 --> mex01 : 195.162 ms
230 : tok05 --> mex01 : 154.654 ms
231 : sao01 --> mex01 : 165.27 ms
232 : sea01 --> mon01 : 60.261 ms
233 : sjc04 --> mon01 : 67.742 ms
234 : tor01 --> mon01 : 10.12 ms
235 : wdc07 --> mon01 : 15.725 ms
236 : par01 --> mon01 : 85.747 ms
237 : sng01 --> mon01 : 253.165 ms
238 : syd05 --> mon01 : 212.485 ms
239 : tok05 --> mon01 : 141.548 ms
240 : sao01 --> mon01 : 119.055 ms
241 : sjc04 --> sea01 : 18.646 ms
242 : tor01 --> sea01 : 52.388 ms
243 : wdc07 --> sea01 : 54.778 ms
244 : sng01 --> sea01 : 161.613 ms
245 : syd05 --> sea01 : 168.44 ms
246 : tok05 --> sea01 : 82.984 ms
247 : tor01 --> sjc04 : 62.691 ms
248 : wdc07 --> sjc04 : 58.79 ms
249 : sng01 --> sjc04 : 178.998 ms
250 : syd05 --> sjc04 : 166.063 ms
251 : tok05 --> sjc04 : 99.461 ms
252 : wdc07 --> tor01 : 21.776 ms
253 : dal13 --> ams03 : 109.936 ms
254 : hou02 --> ams03 : 116.328 ms
255 : mex01 --> ams03 : 136.264 ms
256 : mon01 --> ams03 : 85.33 ms
257 : sea01 --> ams03 : 137.656 ms
258 : sjc04 --> ams03 : 147.043 ms
259 : tor01 --> ams03 : 90.952 ms
260 : wdc07 --> ams03 : 80.417 ms
261 : fra05 --> ams03 : 7.293 ms
262 : lon02 --> ams03 : 7.617 ms
263 : mil01 --> ams03 : 32.741 ms
264 : par01 --> ams03 : 11.759 ms
265 : che01 --> ams03 : 140.786 ms
266 : hkg02 --> ams03 : 200.616 ms
267 : mel01 --> ams03 : 258.165 ms
268 : sng01 --> ams03 : 170.328 ms
269 : syd05 --> ams03 : 261.191 ms
270 : tok05 --> ams03 : 250.947 ms
271 : sao01 --> ams03 : 184.626 ms
272 : hou02 --> fra05 : 128.026 ms
273 : mex01 --> fra05 : 145.337 ms
274 : mon01 --> fra05 : 87.781 ms
275 : sea01 --> fra05 : 139.901 ms
276 : sjc04 --> fra05 : 145.367 ms
277 : tor01 --> fra05 : 94.418 ms
278 : wdc07 --> fra05 : 83.746 ms
279 : lon02 --> fra05 : 11.948 ms
280 : mil01 --> fra05 : 15.51 ms
281 : par01 --> fra05 : 10.708 ms
282 : hkg02 --> fra05 : 181.103 ms
283 : mel01 --> fra05 : 241.097 ms
284 : sng01 --> fra05 : 150.226 ms
285 : syd05 --> fra05 : 243.074 ms
286 : tok05 --> fra05 : 232.018 ms
287 : sao01 --> fra05 : 188.104 ms
288 : mex01 --> lon02 : 135.827 ms
289 : mon01 --> lon02 : 78.969 ms
290 : sea01 --> lon02 : 130.482 ms
291 : sjc04 --> lon02 : 140.083 ms
292 : tor01 --> lon02 : 84.457 ms
293 : wdc07 --> lon02 : 74.508 ms
294 : mil01 --> lon02 : 24.478 ms
295 : par01 --> lon02 : 8.732 ms
296 : mel01 --> lon02 : 247.035 ms
297 : sng01 --> lon02 : 160.738 ms
298 : syd05 --> lon02 : 256.692 ms
299 : tok05 --> lon02 : 245.076 ms
300 : sao01 --> lon02 : 178.235 ms
301 : mon01 --> mil01 : 101.163 ms
302 : sea01 --> mil01 : 151.781 ms
303 : sjc04 --> mil01 : 160.164 ms
304 : tor01 --> mil01 : 104.802 ms
305 : wdc07 --> mil01 : 107.764 ms
306 : par01 --> mil01 : 18.717 ms
307 : sng01 --> mil01 : 146.445 ms
308 : syd05 --> mil01 : 235.762 ms
309 : tok05 --> mil01 : 226.593 ms
310 : sao01 --> mil01 : 200.54 ms
311 : sea01 --> par01 : 138.504 ms
312 : sjc04 --> par01 : 146.082 ms
313 : tor01 --> par01 : 91.441 ms
314 : wdc07 --> par01 : 90.997 ms
315 : sng01 --> par01 : 157.344 ms
316 : syd05 --> par01 : 248.384 ms
317 : tok05 --> par01 : 243.227 ms
318 : sao01 --> par01 : 186.249 ms
319 : dal13 --> che01 : 244.802 ms
320 : hou02 --> che01 : 250.798 ms
321 : mex01 --> che01 : 264.097 ms
322 : mon01 --> che01 : 217.033 ms
323 : sea01 --> che01 : 193.851 ms
324 : sjc04 --> che01 : 211.018 ms
325 : tor01 --> che01 : 220.153 ms
326 : wdc07 --> che01 : 215.766 ms
327 : fra05 --> che01 : 129.318 ms
328 : lon02 --> che01 : 136.508 ms
329 : mil01 --> che01 : 145.576 ms
330 : par01 --> che01 : 126.139 ms
331 : hkg02 --> che01 : 64.007 ms
332 : mel01 --> che01 : 119.331 ms
333 : sng01 --> che01 : 33.49 ms
334 : syd05 --> che01 : 123.778 ms
335 : tok05 --> che01 : 112.277 ms
336 : sao01 --> che01 : 313.742 ms
337 : hou02 --> hkg02 : 189.014 ms
338 : mex01 --> hkg02 : 201.956 ms
339 : mon01 --> hkg02 : 221.394 ms
340 : sea01 --> hkg02 : 130.899 ms
341 : sjc04 --> hkg02 : 147.557 ms
342 : tor01 --> hkg02 : 212.701 ms
343 : wdc07 --> hkg02 : 215.142 ms
344 : lon02 --> hkg02 : 194.997 ms
345 : mil01 --> hkg02 : 173.89 ms
346 : par01 --> hkg02 : 190.894 ms
347 : mel01 --> hkg02 : 116.756 ms
348 : sng01 --> hkg02 : 32.037 ms
349 : syd05 --> hkg02 : 115.227 ms
350 : tok05 --> hkg02 : 49.959 ms
351 : sao01 --> hkg02 : 326.016 ms
352 : mex01 --> mel01 : 209.966 ms
353 : mon01 --> mel01 : 221.753 ms
354 : sea01 --> mel01 : 188.21 ms
355 : sjc04 --> mel01 : 169.247 ms
356 : tor01 --> mel01 : 213.681 ms
357 : wdc07 --> mel01 : 220.186 ms
358 : mil01 --> mel01 : 236.444 ms
359 : par01 --> mel01 : 247.394 ms
360 : sng01 --> mel01 : 86.007 ms
361 : syd05 --> mel01 : 16.736 ms
362 : tok05 --> mel01 : 129.25 ms
363 : sao01 --> mel01 : 336.299 ms
364 : tor01 --> sng01 : 242.918 ms
365 : wdc07 --> sng01 : 244.042 ms
366 : syd05 --> sng01 : 91.362 ms
367 : tok05 --> sng01 : 80.367 ms
368 : tor01 --> syd05 : 211.296 ms
369 : wdc07 --> syd05 : 214.035 ms
370 : tok05 --> syd05 : 113.929 ms
371 : tor01 --> tok05 : 133.838 ms
372 : wdc07 --> tok05 : 135.879 ms
373 : sea01 --> sao01 : 170.505 ms
374 : sjc04 --> sao01 : 181.346 ms
375 : tor01 --> sao01 : 124.855 ms
376 : wdc07 --> sao01 : 115.714 ms
377 : sng01 --> sao01 : 354.263 ms
378 : syd05 --> sao01 : 327.542 ms
379 : tok05 --> sao01 : 279.942 ms
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|  (ms) | dal13 | hou02 | mex01 | mon01 | sea01 | sjc04 | tor01 | wdc07 | ams03 | fra05 | lon02 | mil01 | par01 | che01 | hkg02 | mel01 | sng01 | syd05 | tok05 | sao01 |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| dal13 |   0   |   7   |   26  |   40  |   43  |   38  |   35  |   31  |  109  |  120  |  112  |  133  |  121  |  244  |  184  |  192  |  212  |  184  |  138  |  143  |
| hou02 |   7   |   0   |   30  |   47  |   50  |   43  |   40  |   39  |  116  |  128  |  114  |  139  |  121  |  250  |  189  |  196  |  218  |  181  |  142  |  138  |
| mex01 |   25  |   30  |   0   |   66  |   66  |   56  |   56  |   55  |  136  |  145  |  135  |  158  |  143  |  264  |  201  |  209  |  231  |  202  |  154  |  165  |
| mon01 |   39  |   46  |   67  |   0   |   59  |   68  |   9   |   15  |   85  |   87  |   78  |  101  |   85  |  217  |  221  |  221  |  252  |  211  |  141  |  118  |
| sea01 |   43  |   50  |   66  |   60  |   0   |   18  |   52  |   54  |  137  |  139  |  130  |  151  |  138  |  193  |  130  |  188  |  161  |  173  |   82  |  170  |
| sjc04 |   37  |   44  |   56  |   67  |   18  |   0   |   62  |   58  |  147  |  145  |  140  |  160  |  146  |  211  |  147  |  169  |  179  |  160  |   99  |  181  |
| tor01 |   35  |   39  |   56  |   10  |   52  |   62  |   0   |   21  |   90  |   94  |   84  |  104  |   91  |  220  |  212  |  213  |  242  |  211  |  133  |  124  |
| wdc07 |   31  |   39  |   55  |   15  |   54  |   58  |   21  |   0   |   80  |   83  |   74  |  107  |   90  |  215  |  215  |  220  |  244  |  214  |  135  |  115  |
| ams03 |  113  |  116  |  136  |   85  |  137  |  147  |   90  |   80  |   0   |   7   |   8   |   31  |   11  |  141  |  200  |  259  |  170  |  261  |  250  |  184  |
| fra05 |  119  |  127  |  145  |   87  |  139  |  145  |   94  |   83  |   7   |   0   |   11  |   15  |   10  |  129  |  179  |  239  |  150  |  243  |  232  |  188  |
| lon02 |  112  |  114  |  135  |   78  |  130  |  141  |   84  |   74  |   7   |   11  |   0   |   25  |   8   |  136  |  194  |  247  |  160  |  257  |  245  |  178  |
| mil01 |  133  |  141  |  155  |  101  |  153  |  160  |  106  |  107  |   32  |   15  |   24  |   0   |   19  |  145  |  173  |  236  |  145  |  235  |  226  |  202  |
| par01 |  121  |  121  |  143  |   85  |  138  |  146  |   91  |   91  |   11  |   10  |   8   |   18  |   0   |  126  |  190  |  247  |  157  |  248  |  243  |  185  |
| che01 |  244  |  250  |  264  |  209  |  193  |  210  |  219  |  218  |  140  |  129  |  133  |  141  |  128  |   0   |   64  |  119  |   33  |  123  |  112  |  313  |
| hkg02 |  184  |  188  |  201  |  221  |  130  |  147  |  212  |  215  |  200  |  181  |  195  |  173  |  190  |   64  |   0   |  117  |   31  |  115  |   49  |  325  |
| mel01 |  194  |  196  |  210  |  222  |  183  |  167  |  222  |  221  |  258  |  241  |  247  |  236  |  246  |  119  |  116  |   0   |   87  |   16  |  131  |  338  |
| sng01 |  213  |  218  |  231  |  253  |  161  |  178  |  243  |  244  |  170  |  150  |  160  |  146  |  157  |   33  |   32  |   86  |   0   |   91  |   80  |  354  |
| syd05 |  184  |  181  |  195  |  212  |  168  |  166  |  210  |  214  |  261  |  243  |  256  |  235  |  248  |  123  |  115  |   16  |   91  |   0   |  113  |  327  |
| tok05 |  138  |  142  |  154  |  141  |   82  |   99  |  133  |  136  |  250  |  232  |  245  |  226  |  243  |  112  |   49  |  129  |   80  |  113  |   0   |  279  |
| sao01 |  143  |  138  |  165  |  119  |  170  |  181  |  125  |  115  |  184  |  188  |  178  |  200  |  186  |  313  |  326  |  336  |  354  |  329  |  279  |   0   |
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
```

# さいごに
`Perfkit` ではマルチクラウドで共通のベンチマークをおこない、パフォーマンス指標を明確にし、コスト試算をよりわかりやすくするためのツールです。
`ping` ベンチマークのほかに、`iperf` や `fio` なども実施できるので、測定結果をもとにシステムリソースを考慮するとコスト試算も確実なものにできるでしょう。

# 参考
[Google PerfKitBenchmarker - wiki.mikejung.biz](https://wiki.mikejung.biz/Google_PerfKitBenchmarker)

## `pkb` option

```
perfkitbenchmarker.pkb:
  --archive_bucket: Archive results to the given S3/GCS bucket.
  --benchmarks: Benchmarks and/or benchmark sets that should be run. The default is the standard set. For more information about
    benchmarks and benchmark sets, see the README and benchmark_sets.py.
    (default: 'standard_set')
    (a comma separated list)
  --[no]boot_samples: Whether to publish boot time samples for all tests.
    (default: 'false')
  --data_disk_size: Size, in gb, for all data disks.
    (an integer)
  --data_disk_type: Type for all data disks. If a provider keeps the operating system and user data on separate disks, this only
    affects the user data disk(s).If the provider has OS and user data on the same disk, this flag affectsthat disk.
  --duration_in_seconds: duration of benchmarks. (only valid for mesh_benchmark)
    (an integer)
  --extra_zones: Zones that will be appended to the "zones" list. This is functionally the same, but allows flag matrices to have
    two zone axes.
    (default: '')
    (a comma separated list)
  --file_log_level: <debug|info>: Anything logged at this level or higher will be written to the log file.
    (default: 'debug')
  --ftp_proxy: Specify a proxy for FTP in the form [user:passwd@]proxy.server:port.
    (default: '')
  --helpmatch: Shows only flags defined in a module whose name matches the given regex.
    (default: '')
  --http_proxy: Specify a proxy for HTTP in the form [user:passwd@]proxy.server:port.
    (default: '')
  --https_proxy: Specify a proxy for HTTPS in the form [user:passwd@]proxy.server:port.
    (default: '')
  --[no]ignore_package_requirements: Disables Python package requirement runtime checks.
    (default: 'false')
  --image: Default image that will be linked to the VM
  --[no]install_packages: Override for determining whether packages should be installed. If this is false, no packages will be
    installed on any VMs. This option should probably only ever be used if you have already created an image with all relevant
    packages installed.
  --log_level: <debug|info>: The log level to run at.
    (default: 'info')
  --machine_type: Machine types that will be created for benchmarks that don't require a particular type.
  --num_striped_disks: The number of data disks to stripe together to form one "logical" data disk. This defaults to 1 (except
    with local disks), which means no striping. When using local disks, they default to striping all disks together. The striped
    disks will appear as one disk (data_disk_0) in the metadata.
    (a positive integer)
  --num_vms: For benchmarks which can make use of a variable number of machines, the number of VMs to use.
    (default: '1')
    (an integer)
  --owner: Owner name. Used to tag created resources and performance records.
    (default: 'root')
  --project: GCP project ID under which to create the virtual machines
  --[no]publish_after_run: If true, PKB will publish all samples available immediately after running each benchmark. This may be
    useful in scenarios where the PKB run time for all benchmarks is much greater than a single benchmark.
    (default: 'false')
  --run_processes: The number of parallel processes to use to run benchmarks.
    (default: '1')
    (a positive integer)
  --run_stage_retries: The number of allowable consecutive failures during the run stage. After this number of failures any
    exceptions will cause benchmark termination. If run_stage_time is exceeded, the run stage will not be retried even if the
    number of failures is less than the value of this flag.
    (default: '0')
    (an integer)
  --run_stage_time: PKB will run/re-run the run stage of each benchmark until it has spent at least this many seconds. It defaults
    to 0, so benchmarks will only be run once unless some other value is specified.
    (default: '0')
    (an integer)
  --run_uri: Name of the Run. If provided, this should be alphanumeric and less than or equal to 10 characters in length.
  --scratch_disk_iops: IOPS for Provisioned IOPS (SSD) volumes in AWS.
    (an integer)
  --scratch_disk_size: Size, in gb, for all scratch disks.
    (an integer)
  --scratch_disk_type: <standard|remote_ssd|piops|local>: Type for all scratch disks. The default is standard
  --spark_service_type: <pkb_managed|managed>: Type of spark service to use
  --ssh_options: Additional options to pass to ssh.
    (default: '')
    (a comma separated list)
  --static_vm_file: The file path for the Static Machine file. See static_virtual_machine.py for a description of this file.
  --[no]stop_after_benchmark_failure: Determines response when running multiple benchmarks serially and a benchmark run fails.
    When True, no further benchmarks are scheduled, and execution ends. When False, benchmarks continue to be scheduled. Does not
    apply to keyboard interrupts, which will always prevent further benchmarks from being scheduled.
    (default: 'false')
  --[no]version: Display the version and exit.
    (default: 'false')
  --zones: A list of zones within which to run PerfKitBenchmarker. This is specific to the cloud provider you are running on. If
    multiple zones are given, PerfKitBenchmarker will create 1 VM in zone, until enough VMs are created as specified in each
    benchmark. The order in which this flag is applied to VMs is undefined.
    (default: '')
    (a comma separated list)
```

## `benchmarks` option

```
# ./pkb.py --helpmatch=benchmarks | grep perfkitbenchmarker.linux_benchmarks.
perfkitbenchmarker.linux_benchmarks.aerospike_benchmark:
perfkitbenchmarker.linux_benchmarks.blazemark_benchmark:
perfkitbenchmarker.linux_benchmarks.block_storage_workloads_benchmark:
perfkitbenchmarker.linux_benchmarks.cassandra_stress_benchmark:
perfkitbenchmarker.linux_benchmarks.cloud_bigtable_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.cloud_datastore_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_data_caching_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_data_serving_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_graph_analytics_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_in_memory_analytics_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_web_search_benchmark:
perfkitbenchmarker.linux_benchmarks.cloudsuite_web_serving_benchmark:
perfkitbenchmarker.linux_benchmarks.copy_throughput_benchmark:
perfkitbenchmarker.linux_benchmarks.dpb_wordcount_benchmark:
perfkitbenchmarker.linux_benchmarks.fio_benchmark:
perfkitbenchmarker.linux_benchmarks.gpu_pcie_bandwidth_benchmark:
perfkitbenchmarker.linux_benchmarks.hadoop_terasort_benchmark:
perfkitbenchmarker.linux_benchmarks.hbase_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.hpcc_benchmark:
perfkitbenchmarker.linux_benchmarks.iperf_benchmark:
perfkitbenchmarker.linux_benchmarks.jdbc_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.mesh_network_benchmark:
perfkitbenchmarker.linux_benchmarks.mongodb_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.multichase_benchmark:
perfkitbenchmarker.linux_benchmarks.mysql_service_benchmark:
perfkitbenchmarker.linux_benchmarks.netperf_benchmark:
perfkitbenchmarker.linux_benchmarks.object_storage_service_benchmark:
perfkitbenchmarker.linux_benchmarks.oldisim_benchmark:
perfkitbenchmarker.linux_benchmarks.redis_benchmark:
perfkitbenchmarker.linux_benchmarks.redis_ycsb_benchmark:
perfkitbenchmarker.linux_benchmarks.silo_benchmark:
perfkitbenchmarker.linux_benchmarks.spark_benchmark:
perfkitbenchmarker.linux_benchmarks.speccpu2006_benchmark:
perfkitbenchmarker.linux_benchmarks.specsfs2014_benchmark:
perfkitbenchmarker.linux_benchmarks.tomcat_wrk_benchmark:
perfkitbenchmarker.linux_benchmarks.unixbench_benchmark:
```

## `fio` option
実行例

```
./pkb.py --owner=khayama --cloud=SoftLayer --benchmarks=fio --zones=tok04 --machine_type="{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }"
```

実行オプション

```
# ./pkb.py --helpmatch=benchmarks.fio

perfkitbenchmarker.linux_benchmarks.fio_benchmark:
  --[no]fio_bw_log: Whether to collect a bandwidth log of the fio jobs.
    (default: 'false')
  --fio_fill_size: The amount of device to fill in prepare stage. The valid value can
    either be an integer, which represents the number of bytes to fill or a
    percentage, which represents the percentage of the device. A filesystem will be
    unmounted before filling and remounted afterwards. Only valid when
    --fio_target_mode is against_device_with_fill or against_file_with_fill.
    (default: '100%')
  --fio_generate_scenarios: Generate a job file with the given scenarios. Special
    scenario 'all' generates all scenarios. Available scenarios are sequential_write,
    sequential_read, random_write, and random_read. Cannot use with --fio_jobfile.
    (default: '')
    (a comma separated list)
  --[no]fio_iops_log: Whether to collect an IOPS log of the fio jobs.
    (default: 'false')
  --fio_jobfile: Job file that fio will use. If not given, use a job file bundled with
    PKB. Cannot use with --fio_generate_scenarios.
  --[no]fio_lat_log: Whether to collect a latency log of the fio jobs.
    (default: 'false')
  --fio_log_avg_msec: By default, this will average each log entry in the fio latency,
    bandwidth, and iops logs over the specified period of time in milliseconds. If set
    to 0, fio will log an entry for every IO that completes, this can grow very
    quickly in size and can cause performance overhead.
    (default: '1000')
    (a non-negative integer)
  --fio_parameters: Parameters to apply to all PKB generated fio jobs. Each member of
    the list should be of the form "param=value".
    (default: '')
    (a comma separated list)
  --fio_runtime: The number of seconds to run each fio job for.
    (default: '600')
    (a positive integer)
  --fio_target_mode: <against_device_with_fill|against_device_without_fill|against_fil
    e_with_fill|against_file_without_fill>: Whether to run against a raw device or a
    file, and whether to prefill.
    (default: 'against_file_without_fill')
  --fio_working_set_size: The size of the working set, in GB. If not given, use the
    full size of the device. If using --fio_generate_scenarios and not running against
    a raw device, you must pass --fio_working_set_size.
    (a non-negative integer)
```

## `object_storage_service` option

aws cli install

```
apt install unzip -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
aws configure
cat ~/.aws/credentials 
aws --endpoint-url https://s3.private.jp-tok.cloud-object-storage.appdomain.cloud s3 ls
```

prepare `~/.boto` file

```
# Example ~/.boto file
[Credentials]
aws_access_key_id = foo
aws_secret_access_key = bar
```

実行例

```
./pkb.py \
--machine_type="{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }" \
--object_storage_scenario=cli \
--object_storage_storage_class=standard \
--cloud=SoftLayer \
--owner=khayama \
--object_storage_region=jp-tok \
--zones=tok04 \
--s3_custom_endpoint=s3.private.jp-tok.cloud-object-storage.appdomain.cloud \
--benchmarks=object_storage_service \
--storage=SoftLayer 
```

実行オプション

```
# ./pkb.py --helpmatch=benchmarks.object_storage_service

perfkitbenchmarker.linux_benchmarks.object_storage_service_benchmark:
  --cli_test_size: <normal|large>: size of the cli tests. Normal means a mixture of
    various
    object sizes up to 32MiB (see data/cloud-storage-workload.sh).
    Large means all objects are of at least 1GiB.
    (default: 'normal')
  --object_storage_gcs_multiregion: Storage multiregion for GCS in object storage
    benchmark.
  --object_storage_list_consistency_iterations: Number of iterations to perform for
    the api_namespace list consistency benchmark. This flag is mainly for regression
    testing in the benchmarks. Reduce the number to shorten the execution time of the
    api_namespace scenario. However, to get useful metrics from the api_namespace
    scenario, a high number of iterations should be used (>=200).
    (default: '200')
    (an integer)
  --object_storage_multistream_objects_per_stream: Number of objects to send and/or
    receive per stream. Only applies to the api_multistream scenario.
    (default: '1000')
    (a positive integer)
  --object_storage_object_naming_scheme:
    <sequential_by_stream|approximately_sequential>: How objects will be named. Only
    applies to the api_multistream benchmark. sequential_by_stream: object names from
    each stream will be sequential, but different streams will have different name
    prefixes. approximately_sequential: object names from all streams will roughly
    increase together.
    (default: 'sequential_by_stream')
  --object_storage_region: Storage region for object storage benchmark.
  --object_storage_scenario:
    <all|cli|api_data|api_namespace|api_multistream|api_multistream_writes>: select
    all, or one particular scenario to run:
    ALL: runs all scenarios. This is the default.
    cli: runs the command line only scenario.
    api_data: runs API based benchmarking for data paths.
    api_namespace: runs API based benchmarking for namespace operations.
    api_multistream: runs API-based benchmarking with multiple upload/download
    streams.
    api_multistream_writes: runs API-based benchmarking with multiple upload streams.
    (default: 'all')
  --object_storage_storage_class: Storage class to use in object storage benchmark.
  --object_storage_streams_per_vm: Number of independent streams per VM. Only applies
    to the api_multistream scenario.
    (default: '10')
    (a positive integer)
  --object_storage_worker_output: If set, the worker threads' output will be written
    to thepath provided.
  --storage: <GCP|AWS|Azure|OpenStack|SoftLayer>: storage provider
    (GCP/AZURE/AWS/OPENSTACK/SOFTLAYER) to use.
    (default: 'GCP')
```

### result

`jp-tok` regional

```
-------------------------PerfKitBenchmarker Results Summary-------------------------
OBJECT_STORAGE_SERVICE:
  boto_lib_version="Version: 2.49.0
" gcs_multiregion_location="default" regional_bucket_location="jp-tok" storage provider="SoftLayer"
  upload throughput via cli Mbps p0.1      484.183572 Mbps                          
  upload throughput via cli Mbps p1      484.183572 Mbps                          
  upload throughput via cli Mbps p5      484.183572 Mbps                          
  upload throughput via cli Mbps p10      484.183572 Mbps                          
  upload throughput via cli Mbps p50      649.040033 Mbps                          
  upload throughput via cli Mbps p90      655.375994 Mbps                          
  upload throughput via cli Mbps p95      655.375994 Mbps                          
  upload throughput via cli Mbps p99      655.375994 Mbps                          
  upload throughput via cli Mbps p99.9      655.375994 Mbps                          
  upload throughput via cli Mbps average      596.199866 Mbps                          
  upload throughput via cli Mbps stddev       97.060670 Mbps                          
  download throughput via cli Mbps p0.1      839.516416 Mbps                          
  download throughput via cli Mbps p1      839.516416 Mbps                          
  download throughput via cli Mbps p5      839.516416 Mbps                          
  download throughput via cli Mbps p10      839.516416 Mbps                          
  download throughput via cli Mbps p50      845.796263 Mbps                          
  download throughput via cli Mbps p90      853.186064 Mbps                          
  download throughput via cli Mbps p95      853.186064 Mbps                          
  download throughput via cli Mbps p99      853.186064 Mbps                          
  download throughput via cli Mbps p99.9      853.186064 Mbps                          
  download throughput via cli Mbps average      846.166248 Mbps                          
  download throughput via cli Mbps stddev        6.842330 Mbps                          
  End to End Runtime                  516.152406 seconds                       

-------------------------
```

`ap-geo` cross-region

```
./pkb.py \
--machine_type="{\"cpus\": 2, \"memory\": 4096, \"os\": \"UBUNTU_LATEST_64\", \"nic\": 1000 }" \
--object_storage_scenario=cli \
--object_storage_storage_class=standard \
--cloud=SoftLayer \
--owner=khayama \
--object_storage_region=ap-geo \
--zones=tok04 \
--s3_custom_endpoint=s3.private.ap.cloud-object-storage.appdomain.cloud \
--benchmarks=object_storage_service \
--storage=SoftLayer 

-------------------------PerfKitBenchmarker Results Summary-------------------------
OBJECT_STORAGE_SERVICE:
  boto_lib_version="Version: 2.49.0
" gcs_multiregion_location="default" regional_bucket_location="ap-geo" storage provider="SoftLayer"
  upload throughput via cli Mbps p0.1      128.742285 Mbps                          
  upload throughput via cli Mbps p1      128.742285 Mbps                          
  upload throughput via cli Mbps p5      128.742285 Mbps                          
  upload throughput via cli Mbps p10      128.742285 Mbps                          
  upload throughput via cli Mbps p50      140.459138 Mbps                          
  upload throughput via cli Mbps p90      151.386267 Mbps                          
  upload throughput via cli Mbps p95      151.386267 Mbps                          
  upload throughput via cli Mbps p99      151.386267 Mbps                          
  upload throughput via cli Mbps p99.9      151.386267 Mbps                          
  upload throughput via cli Mbps average      140.195896 Mbps                          
  upload throughput via cli Mbps stddev       11.324286 Mbps                          
  download throughput via cli Mbps p0.1      384.866089 Mbps                          
  download throughput via cli Mbps p1      384.866089 Mbps                          
  download throughput via cli Mbps p5      384.866089 Mbps                          
  download throughput via cli Mbps p10      384.866089 Mbps                          
  download throughput via cli Mbps p50      405.114559 Mbps                          
  download throughput via cli Mbps p90      431.997287 Mbps                          
  download throughput via cli Mbps p95      431.997287 Mbps                          
  download throughput via cli Mbps p99      431.997287 Mbps                          
  download throughput via cli Mbps p99.9      431.997287 Mbps                          
  download throughput via cli Mbps average      407.325979 Mbps                          
  download throughput via cli Mbps stddev       23.643292 Mbps                          
  End to End Runtime                  641.290729 seconds                       

-------------------------
```