Dev in Container on WSL2 でボリュームのパフォーマンスを比べてみる

※この記事は [WESEEK Tech Blog に 2020/06.15 に投稿された記事](https://tech.weseek.co.jp/entry/2020/06/15/100000) の転載です

-----

こんにちは。武井@[WESEEK, Inc.](http://weseek.co.jp) です。

2020年5月末に待望の Windows 10 May 2020 Update が公開され、WESEEK, Inc. で開発中の [GROWI](https://growi.org) でも WSL2 と Docker Desktop を利用して [Dev in Container を実現した新しい開発スタートアップ](https://docs.growi.org/ja/dev/startup-v2/dev-env.html) を採用しました。

本日のエントリーでは、その開発時に利用するストレージ(Docker volume)の種類によってパフォーマンスがどれくらい変わるのかを実験した際の結果を紹介します。


<!-- more -->


取り立てて高速化のための新しい手法を試しているというわけではありませんが、昨今の node プロジェクト開発における node_modules、rails プロジェクト開発における vendor/bundle 等、依存関係ディレクトリの取り扱いの参考になれば幸いです。


# 経緯

## 開発環境起動の手間は減った！

[旧開発スタートアップ](https://docs.growi.org/ja/dev/startup-v1/dev-env.html) では Vagrant/Virtualbox で Linux を立ち上げ、その中でミドルウェア群(MongoDB や Elasticsearch)を docker-compose で起動していましたが、この操作は個人的には苦痛でした。`vagrant up` で待たされ、その後 `vagrant ssh` して、`tmux` して、`docker-compose up` して…  
Windows Update で再起動を迫られて一番億劫になるのがその部分でした。

それが今や新しい開発スタートアップでは Visual Studio Code を立ち上げるだけでミドルウェア群含めコンテナが立ち上がった状態になるのです。なんて素晴らしいのでしょう。

## 懸念されるパフォーマンスダウン

しかし代わりにパフォーマンスダウンも受け入れなければなりません。  
これまではホストOSで起動していた webpack-dev-server 及び Express server が、新開発環境ではコンテナ内で立ち上がることになったため、node プロジェクトで参照する node_modules も、bind mount されているプロジェクトリポジトリの中に置かれ、理論上は大なり小なりアクセスが遅くなってしまうのです。[^1]

[^1]: bind mount しないでコンテナ内でリポジトリを clone すればいいじゃん、という意見もあるかもしれないが、WESEEK では Git の操作は SourceTree 推奨であるため、ホストOSからアクセスできる場所にリポジトリを置く必要がある

特に Docker Desktop for Mac ではストレージのボトルネックが顕著であるため、bind mount を放棄して [Mutagen-based caching](https://docs.docker.com/docker-for-mac/mutagen-caching/) を採用する事例も多くなってきています。

## WSL2 では…？
では WSL2 と Docker Desktop for Windows の組み合わせではどうかというと、docker container に bind mount されたディレクトリへの読み書きは体感では十分なスピードが出ており、そのままでも実用には耐えられる感じでした。しかしここでふと使い古された手法、**node_modules を bind mount volume ではなく local volume に置くとどうなるのか？** という疑問に行き当たりました。

意外にもベンチマークを取った記事は見つからなかったので、今回はそれを測定してみよう、というわけです。


# 実験環境

|key|value|
|:--|:--|
|ホストOS| Windows 10 Home 2004 build 19041.264 |
|CPU| Intel Core i5-6500 |
|SSD| Crucial MX200 CT250MX200SSD1 x 2 (Raid 0) |

**「会社の代表なのにこんな古いCPUのマシンを…」**

とか思われるのを想像するとこれを晒すのは恥ずかしかったのですが、ベンチマークを載せる上ではそこそこ重要な情報だと思うので仕方なし。。[^pc]

[^pc]: 因みに自宅のプライベートマシンです。会社ではもっといいやつを貸与してます。


一方、仮想化側はこんな感じの仕様です。

|key|value|
|:--|:--|
|WSL2 OS|Ubuntu 20.04 LTS|
|Docker Desktop| 2.3.0.3 (45519) |
|Docker|19.03.8|

コンテナの inspect は、

```json
{
  ...
  "Driver": "overlay2",
  ...
  "Mounts": [
    {
      "Type": "bind",
      "Source": "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/xxx",
      "Destination": "/workspace",
      "Mode": "cached",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Source": "/var/lib/docker/volumes/xxx/_data",
      "Destination": "/workspace/node_modules",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": ""
    }
   ],
```

`df` 結果は、

```bash
node@aa9926a4e3d0:/workspace/growi$ df -hT
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay  251G   14G  225G   6% /
...
/dev/sdd       ext4     251G  2.2G  237G   1% /workspace
...
```

`/workspace` は ext4、  
`/workspace/node_modules` は overlay2 ということになります。


# 実験
## bind mount volume (失敗編)

ストレージのベンチマークということで、[fio](https://packages.ubuntu.com/focal/fio) を使います。

ちらっと調べてみても膨大な量のオプションがあることにげんなり、、  
しかしながらパフォーマンスを見たいシチュエーションは今回は動画データのコピーみたいなのではなく node_modules などの小さなファイルを大量に置く場所の IO なので、ランダムアクセスでの評価であることだけははっきりしています。それを念頭に、`rw=randrw` が入ったコマンドをそこら辺の記事から適当にコマンドを拾ってきました。

まずは bind mount volume からやろうと、 `--directory=/workspace` だけ変更してこんな感じ。

```bash
fio \
  --ioengine=libaio \
  --iodepth=16 \
  --direct=1 \
  --size=100m \
  --numjobs=16 \
  --group_reporting \
  --directory=/workspace \
  --bs=4k \
  --output=/root/result-bind-mount.txt \
  --name=random-r70w30 \
  --rw=randrw \
  --rwmixread=70
```

その他のオプションは精査はせずとりあえず3回実行して結果を見てみます。  

### コマンドログ
```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)

random-r70w30: (groupid=0, jobs=16): err= 0: pid=2668: Fri Jun 12 17:49:14 2020
  read : io=1120.5MB, bw=19427KB/s, iops=4856, runt= 59059msec
    slat (usec): min=92, max=8839.5K, avg=2350.42, stdev=73105.34
    clat (usec): min=14, max=10072K, avg=34949.61, stdev=291368.98
     lat (usec): min=344, max=10075K, avg=37300.03, stdev=302701.82
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   20], 90.00th=[   26], 95.00th=[   37],
     | 99.00th=[  235], 99.50th=[  537], 99.90th=[ 4686], 99.95th=[ 8979],
     | 99.99th=[10028]
  write: io=491080KB, bw=8315.8KB/s, iops=2078, runt= 59059msec
    slat (usec): min=111, max=8838.9K, avg=2039.70, stdev=58524.66
    clat (usec): min=13, max=10072K, avg=33542.78, stdev=277647.18
     lat (usec): min=301, max=10075K, avg=35582.48, stdev=285130.72
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   20], 90.00th=[   25], 95.00th=[   37],
     | 99.00th=[  217], 99.50th=[  465], 99.90th=[ 4686], 99.95th=[ 8979],
     | 99.99th=[10028]
    lat (usec) : 20=0.01%, 50=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=8.05%, 20=73.47%, 50=15.22%
    lat (msec) : 100=1.55%, 250=0.74%, 500=0.43%, 750=0.12%, 1000=0.01%
    lat (msec) : 2000=0.09%, >=2000=0.29%
  cpu          : usr=0.52%, sys=4.01%, ctx=437248, majf=0, minf=144
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=19426KB/s, minb=19426KB/s, maxb=19426KB/s, mint=59059msec, maxt=59059msec
  WRITE: io=491080KB, aggrb=8315KB/s, minb=8315KB/s, maxb=8315KB/s, mint=59059msec, maxt=59059msec
```


```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes

random-r70w30: (groupid=0, jobs=16): err= 0: pid=2865: Fri Jun 12 17:53:12 2020
  read : io=1120.5MB, bw=26392KB/s, iops=6598, runt= 43472msec
    slat (usec): min=79, max=1762.1K, avg=1647.55, stdev=21882.17
    clat (usec): min=13, max=2085.2K, avg=25339.53, stdev=95393.85
     lat (usec): min=387, max=2097.6K, avg=26987.08, stdev=98741.20
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   10], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   21], 90.00th=[   29], 95.00th=[   43],
     | 99.00th=[  182], 99.50th=[  343], 99.90th=[ 1827], 99.95th=[ 1958],
     | 99.99th=[ 2057]
  write: io=491080KB, bw=11296KB/s, iops=2824, runt= 43472msec
    slat (usec): min=114, max=1761.8K, avg=1661.28, stdev=25378.13
    clat (usec): min=14, max=2075.3K, avg=25617.88, stdev=97624.18
     lat (usec): min=239, max=2090.2K, avg=27279.16, stdev=101576.37
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   21], 90.00th=[   29], 95.00th=[   44],
     | 99.00th=[  182], 99.50th=[  343], 99.90th=[ 1827], 99.95th=[ 2008],
     | 99.99th=[ 2040]
    lat (usec) : 20=0.01%, 50=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=10.10%, 20=67.71%, 50=18.03%
    lat (msec) : 100=2.18%, 250=1.21%, 500=0.28%, 750=0.12%, 1000=0.07%
    lat (msec) : 2000=0.23%, >=2000=0.05%
  cpu          : usr=0.76%, sys=5.20%, ctx=438716, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=26392KB/s, minb=26392KB/s, maxb=26392KB/s, mint=43472msec, maxt=43472msec
  WRITE: io=491080KB, aggrb=11296KB/s, minb=11296KB/s, maxb=11296KB/s, mint=43472msec, maxt=43472msec
```


```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes

random-r70w30: (groupid=0, jobs=16): err= 0: pid=3034: Fri Jun 12 17:55:36 2020
  read : io=1120.5MB, bw=43048KB/s, iops=10762, runt= 26652msec
    slat (usec): min=107, max=344821, avg=1033.91, stdev=2925.78
    clat (usec): min=14, max=363954, avg=15509.54, stdev=14831.15
     lat (usec): min=281, max=364755, avg=16543.45, stdev=15431.53
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   10], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   12], 40.00th=[   13], 50.00th=[   14], 60.00th=[   15],
     | 70.00th=[   16], 80.00th=[   18], 90.00th=[   21], 95.00th=[   27],
     | 99.00th=[   47], 99.50th=[   62], 99.90th=[  277], 99.95th=[  359],
     | 99.99th=[  363]
  write: io=491080KB, bw=18426KB/s, iops=4606, runt= 26652msec
    slat (usec): min=103, max=344710, avg=906.47, stdev=3383.50
    clat (usec): min=14, max=363714, avg=15560.08, stdev=14841.43
     lat (usec): min=184, max=364281, avg=16466.55, stdev=15570.76
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   10], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   12], 40.00th=[   13], 50.00th=[   14], 60.00th=[   15],
     | 70.00th=[   16], 80.00th=[   18], 90.00th=[   21], 95.00th=[   27],
     | 99.00th=[   47], 99.50th=[   61], 99.90th=[  277], 99.95th=[  359],
     | 99.99th=[  363]
    lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
    lat (usec) : 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=12.41%, 20=75.90%, 50=10.88%
    lat (msec) : 100=0.48%, 250=0.19%, 500=0.11%
  cpu          : usr=1.12%, sys=8.05%, ctx=439598, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=43048KB/s, minb=43048KB/s, maxb=43048KB/s, mint=26652msec, maxt=26652msec
  WRITE: io=491080KB, aggrb=18425KB/s, minb=18425KB/s, maxb=18425KB/s, mint=26652msec, maxt=26652msec
```

### IOPS サマリ

|#|Read IOPS|Write IOPS|
|--:|--:|--:|
|1回目|4856|2078|
|2回目|6598|2824|
|3回目|10762|4606|
|Avg|7405.33|3169.33|

なんかかなりバラツキが出ました。しかも数字はどんどん良くなっています。うーむ…


## local volume (失敗編)

bind mount は WSL とかホストOSの影響を強く受けるかもな～となんとなく思いながら、同条件で local volume もやってみます。

コマンドは `--directory=/workspace/node_module`

```bash
fio \
  --ioengine=libaio \
  --iodepth=16 \
  --direct=1 \
  --size=100m \
  --numjobs=16 \
  --group_reporting \
  --directory=/workspace/node_module \
  --bs=4k \
  --output=/root/result-local-volume.txt \
  --name=random-r70w30 \
  --rw=randrw \
  --rwmixread=70
```

その他のオプションは精査はせずとりあえず3回実行して結果を見てみます。  

### コマンドログ

```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)
random-r70w30: Laying out IO file(s) (1 file(s) / 100MB)

random-r70w30: (groupid=0, jobs=16): err= 0: pid=3095: Fri Jun 12 17:57:25 2020
  read : io=1120.5MB, bw=96779KB/s, iops=24194, runt= 11855msec
    slat (usec): min=10, max=729378, avg=162.49, stdev=3947.81
    clat (usec): min=154, max=753921, avg=7212.49, stdev=24010.85
     lat (usec): min=166, max=753934, avg=7374.97, stdev=24340.19
    clat percentiles (usec):
     |  1.00th=[  466],  5.00th=[  716], 10.00th=[ 1020], 20.00th=[ 1784],
     | 30.00th=[ 2736], 40.00th=[ 3536], 50.00th=[ 4384], 60.00th=[ 5472],
     | 70.00th=[ 7008], 80.00th=[ 9536], 90.00th=[13760], 95.00th=[18304],
     | 99.00th=[32128], 99.50th=[41216], 99.90th=[585728], 99.95th=[716800],
     | 99.99th=[733184]
  write: io=491080KB, bw=41424KB/s, iops=10355, runt= 11855msec
    slat (usec): min=11, max=587055, avg=149.15, stdev=2113.18
    clat (usec): min=152, max=753069, avg=6601.73, stdev=25086.49
     lat (usec): min=165, max=753081, avg=6750.88, stdev=25193.47
    clat percentiles (usec):
     |  1.00th=[  358],  5.00th=[  524], 10.00th=[  836], 20.00th=[ 1432],
     | 30.00th=[ 2320], 40.00th=[ 3056], 50.00th=[ 3888], 60.00th=[ 4832],
     | 70.00th=[ 6176], 80.00th=[ 8384], 90.00th=[12480], 95.00th=[16768],
     | 99.00th=[30080], 99.50th=[39680], 99.90th=[585728], 99.95th=[716800],
     | 99.99th=[733184]
    lat (usec) : 250=0.01%, 500=2.68%, 750=3.66%, 1000=4.46%
    lat (msec) : 2=12.54%, 4=23.78%, 10=35.32%, 20=13.83%, 50=3.36%
    lat (msec) : 100=0.10%, 250=0.14%, 750=0.12%, 1000=0.01%
  cpu          : usr=1.42%, sys=12.26%, ctx=49495, majf=0, minf=140
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=96779KB/s, minb=96779KB/s, maxb=96779KB/s, mint=11855msec, maxt=11855msec
  WRITE: io=491080KB, aggrb=41423KB/s, minb=41423KB/s, maxb=41423KB/s, mint=11855msec, maxt=11855msec

Disk stats (read/write):
  sdc: ios=285407/122151, merge=0/18, ticks=856301/346468, in_queue=1252850, util=99.28%
```

```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes

random-r70w30: (groupid=0, jobs=16): err= 0: pid=3205: Fri Jun 12 17:58:53 2020
  read : io=1120.5MB, bw=64222KB/s, iops=16055, runt= 17865msec
    slat (usec): min=10, max=4600.5K, avg=238.75, stdev=17231.41
    clat (usec): min=225, max=4619.3K, avg=10883.17, stdev=112540.56
     lat (usec): min=262, max=4619.7K, avg=11121.92, stdev=113853.30
    clat percentiles (usec):
     |  1.00th=[  548],  5.00th=[  852], 10.00th=[ 1288], 20.00th=[ 2736],
     | 30.00th=[ 3824], 40.00th=[ 4832], 50.00th=[ 5920], 60.00th=[ 7392],
     | 70.00th=[ 9408], 80.00th=[12608], 90.00th=[17280], 95.00th=[21632],
     | 99.00th=[35584], 99.50th=[43776], 99.90th=[91648], 99.95th=[4620288],
     | 99.99th=[4620288]
  write: io=491080KB, bw=27488KB/s, iops=6872, runt= 17865msec
    slat (usec): min=11, max=4598.5K, avg=210.05, stdev=13195.12
    clat (usec): min=207, max=4617.1K, avg=10448.85, stdev=117589.18
     lat (usec): min=219, max=4617.2K, avg=10658.90, stdev=118329.70
    clat percentiles (usec):
     |  1.00th=[  446],  5.00th=[  724], 10.00th=[ 1064], 20.00th=[ 2416],
     | 30.00th=[ 3440], 40.00th=[ 4384], 50.00th=[ 5408], 60.00th=[ 6688],
     | 70.00th=[ 8640], 80.00th=[11456], 90.00th=[16064], 95.00th=[20096],
     | 99.00th=[34048], 99.50th=[41216], 99.90th=[92672], 99.95th=[4620288],
     | 99.99th=[4620288]
    lat (usec) : 250=0.01%, 500=0.85%, 750=3.00%, 1000=4.25%
    lat (msec) : 2=6.87%, 4=17.98%, 10=40.22%, 20=20.86%, 50=5.64%
    lat (msec) : 100=0.22%, 250=0.03%, >=2000=0.06%
  cpu          : usr=0.87%, sys=8.85%, ctx=52051, majf=0, minf=160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=64221KB/s, minb=64221KB/s, maxb=64221KB/s, mint=17865msec, maxt=17865msec
  WRITE: io=491080KB, aggrb=27488KB/s, minb=27488KB/s, maxb=27488KB/s, mint=17865msec, maxt=17865msec

Disk stats (read/write):
  sdc: ios=281630/120592, merge=0/48, ticks=1514001/707364, in_queue=2310560, util=99.56%
```

```
random-r70w30: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
fio-2.16
Starting 16 processes

random-r70w30: (groupid=0, jobs=16): err= 0: pid=3267: Fri Jun 12 18:00:27 2020
  read : io=1120.5MB, bw=40033KB/s, iops=10008, runt= 28659msec
    slat (usec): min=10, max=5904.4K, avg=465.57, stdev=27430.86
    clat (usec): min=201, max=6207.4K, avg=17514.68, stdev=150701.41
     lat (usec): min=213, max=6207.4K, avg=17980.26, stdev=153266.04
    clat percentiles (usec):
     |  1.00th=[  524],  5.00th=[ 1048], 10.00th=[ 1880], 20.00th=[ 3472],
     | 30.00th=[ 4832], 40.00th=[ 6304], 50.00th=[ 8256], 60.00th=[10688],
     | 70.00th=[13760], 80.00th=[17280], 90.00th=[26496], 95.00th=[37120],
     | 99.00th=[84480], 99.50th=[181248], 99.90th=[856064], 99.95th=[5931008],
     | 99.99th=[6062080]
  write: io=491080KB, bw=17135KB/s, iops=4283, runt= 28659msec
    slat (usec): min=11, max=5902.7K, avg=440.59, stdev=24409.91
    clat (usec): min=173, max=6207.3K, avg=16460.93, stdev=149825.98
     lat (usec): min=185, max=6207.3K, avg=16901.52, stdev=151860.71
    clat percentiles (usec):
     |  1.00th=[  418],  5.00th=[  852], 10.00th=[ 1512], 20.00th=[ 3024],
     | 30.00th=[ 4320], 40.00th=[ 5728], 50.00th=[ 7456], 60.00th=[ 9664],
     | 70.00th=[12608], 80.00th=[16320], 90.00th=[24192], 95.00th=[34048],
     | 99.00th=[79360], 99.50th=[173056], 99.90th=[856064], 99.95th=[5931008],
     | 99.99th=[6062080]
    lat (usec) : 250=0.01%, 500=1.08%, 750=1.85%, 1000=2.21%
    lat (msec) : 2=6.16%, 4=13.84%, 10=33.62%, 20=25.92%, 50=12.87%
    lat (msec) : 100=1.56%, 250=0.54%, 500=0.15%, 750=0.03%, 1000=0.09%
    lat (msec) : >=2000=0.06%
  cpu          : usr=0.55%, sys=5.20%, ctx=61809, majf=0, minf=130
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=40033KB/s, minb=40033KB/s, maxb=40033KB/s, mint=28659msec, maxt=28659msec
  WRITE: io=491080KB, aggrb=17135KB/s, minb=17135KB/s, maxb=17135KB/s, mint=28659msec, maxt=28659msec

Disk stats (read/write):
  sdc: ios=285907/122380, merge=0/73, ticks=2830298/1216395, in_queue=4119800, util=99.74%
```

### IOPS サマリ

|#|Read IOPS|Write IOPS|
|--:|--:|--:|
|1回目|24194|10355|
|2回目|16055|6872|
|3回目|10008|4283|
|Avg| 16752.33 | 7170.00 |

なんということでしょう、どんどん悪くなってしまいました、、

Avg と Max を比べてみると local volume の方が優れていそうだけど、local volume の一番悪い時は、bind mount の一番良い時よりも遅い。

これでは「その時その時で変わる」みたいな結論になってしまいそうで、これは fio のオプションを疑ってみるべきと判断し改めて使い方を調べてみると、`invalidate` なるオプションを発見。true にセットすることでカーネルバッファとページキャッシュを無効化するらしいです。きっとこれだーー。



## bind mount volume (なんとなく成功編)

```bash
fio \
  --ioengine=libaio \
  --iodepth=16 \
  --direct=1 \
  --invalidate=1 \
  --size=100m \
  --numjobs=16 \
  --group_reporting \
  --directory=/workspace \
  --bs=4k \
  --output=/root/result-bind-mount.txt \
  --name=random-r70w30 \
  --rw=randrw \
  --rwmixread=70
```

### コマンドログ

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=4890: Fri Jun 12 18:19:09 2020
  read : io=1120.5MB, bw=37958KB/s, iops=9489, runt= 30226msec
    slat (usec): min=102, max=2696.5K, avg=1184.04, stdev=16793.00
    clat (usec): min=14, max=2786.8K, avg=17587.83, stdev=66460.90
     lat (usec): min=324, max=2808.6K, avg=18771.87, stdev=68844.54
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   16], 80.00th=[   19], 90.00th=[   26], 95.00th=[   36],
     | 99.00th=[   69], 99.50th=[  103], 99.90th=[  277], 99.95th=[ 2704],
     | 99.99th=[ 2769]
  write: io=491080KB, bw=16247KB/s, iops=4061, runt= 30226msec
    slat (usec): min=113, max=2696.4K, avg=1067.11, stdev=17278.98
    clat (usec): min=14, max=2802.8K, avg=17845.99, stdev=71096.28
     lat (usec): min=190, max=2805.8K, avg=18913.10, stdev=73434.98
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   13], 60.00th=[   14],
     | 70.00th=[   16], 80.00th=[   19], 90.00th=[   26], 95.00th=[   35],
     | 99.00th=[   70], 99.50th=[  106], 99.90th=[  281], 99.95th=[ 2737],
     | 99.99th=[ 2769]
    lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=22.70%, 20=61.26%, 50=13.89%
    lat (msec) : 100=1.62%, 250=0.39%, 500=0.08%, >=2000=0.06%
  cpu          : usr=0.99%, sys=6.32%, ctx=411889, majf=0, minf=110
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=37958KB/s, minb=37958KB/s, maxb=37958KB/s, mint=30226msec, maxt=30226msec
  WRITE: io=491080KB, aggrb=16246KB/s, minb=16246KB/s, maxb=16246KB/s, mint=30226msec, maxt=30226msec
```

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=4933: Fri Jun 12 18:20:51 2020
  read : io=1120.5MB, bw=57217KB/s, iops=14304, runt= 20052msec
    slat (usec): min=103, max=63202, avg=787.42, stdev=784.70
    clat (usec): min=13, max=73706, avg=11693.63, stdev=3764.64
     lat (usec): min=449, max=74260, avg=12481.05, stdev=3907.20
    clat percentiles (usec):
     |  1.00th=[ 7200],  5.00th=[ 8032], 10.00th=[ 8512], 20.00th=[ 9280],
     | 30.00th=[ 9792], 40.00th=[10432], 50.00th=[10944], 60.00th=[11584],
     | 70.00th=[12480], 80.00th=[13504], 90.00th=[15168], 95.00th=[17280],
     | 99.00th=[24704], 99.50th=[30080], 99.90th=[49920], 99.95th=[71168],
     | 99.99th=[73216]
  write: io=491080KB, bw=24490KB/s, iops=6122, runt= 20052msec
    slat (usec): min=106, max=61690, avg=667.05, stdev=770.73
    clat (usec): min=14, max=73636, avg=11725.88, stdev=3789.35
     lat (usec): min=204, max=74086, avg=12392.93, stdev=3927.37
    clat percentiles (usec):
     |  1.00th=[ 7200],  5.00th=[ 8032], 10.00th=[ 8512], 20.00th=[ 9280],
     | 30.00th=[ 9920], 40.00th=[10432], 50.00th=[10944], 60.00th=[11712],
     | 70.00th=[12480], 80.00th=[13504], 90.00th=[15296], 95.00th=[17280],
     | 99.00th=[24960], 99.50th=[30592], 99.90th=[49920], 99.95th=[71168],
     | 99.99th=[73216]
    lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.79%, 20=64.79%, 50=2.30%
    lat (msec) : 100=0.10%
  cpu          : usr=1.49%, sys=9.52%, ctx=412061, majf=0, minf=120
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=57217KB/s, minb=57217KB/s, maxb=57217KB/s, mint=20052msec, maxt=20052msec
  WRITE: io=491080KB, aggrb=24490KB/s, minb=24490KB/s, maxb=24490KB/s, mint=20052msec, maxt=20052msec
```

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=4995: Fri Jun 12 18:21:55 2020
  read : io=1120.5MB, bw=56440KB/s, iops=14110, runt= 20328msec
    slat (usec): min=87, max=90988, avg=794.97, stdev=1033.96
    clat (usec): min=14, max=114818, avg=11841.98, stdev=5138.10
     lat (usec): min=363, max=115464, avg=12636.94, stdev=5344.76
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   16], 95.00th=[   18],
     | 99.00th=[   24], 99.50th=[   43], 99.90th=[   85], 99.95th=[  105],
     | 99.99th=[  112]
  write: io=491080KB, bw=24158KB/s, iops=6039, runt= 20328msec
    slat (usec): min=100, max=91008, avg=679.51, stdev=1074.95
    clat (usec): min=13, max=114338, avg=11856.25, stdev=5114.70
     lat (usec): min=267, max=114962, avg=12535.76, stdev=5305.65
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   11], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   16], 95.00th=[   18],
     | 99.00th=[   24], 99.50th=[   43], 99.90th=[   86], 99.95th=[  106],
     | 99.99th=[  114]
    lat (usec) : 20=0.01%, 50=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=32.90%, 20=65.01%, 50=1.71%
    lat (msec) : 100=0.29%, 250=0.07%
  cpu          : usr=1.44%, sys=9.51%, ctx=412116, majf=0, minf=137
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=56440KB/s, minb=56440KB/s, maxb=56440KB/s, mint=20328msec, maxt=20328msec
  WRITE: io=491080KB, aggrb=24157KB/s, minb=24157KB/s, maxb=24157KB/s, mint=20328msec, maxt=20328msec
```

### IOPS サマリ

|#|Read IOPS|Write IOPS|
|--:|--:|--:|
|1回目|9489 |4061 |
|2回目|14304 |6122 |
|3回目|14110|6039|
|Avg| 12634.33 | 5407.33 |

それっぽい安定した値が取れました。4回目以降は記録していませんが、だいたい2,3回目と似たような感じ。  
Linux 側でキャッシュなしの設定を入れたにも関わらず1回目と2回目以降の結果にかなり差があるのは、やはり bind mount ということで WSL と Windows の何らかの力が働いてるんでしょうか。





## local volume (なんとなく成功編)

```bash
fio \
  --ioengine=libaio \
  --iodepth=16 \
  --direct=1 \
  --invalidate=1 \
  --size=100m \
  --numjobs=16 \
  --group_reporting \
  --directory=/workspace/node_module \
  --bs=4k \
  --output=/root/result-local-volume.txt \
  --name=random-r70w30 \
  --rw=randrw \
  --rwmixread=70
```

### コマンドログ

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=544: Fri Jun 12 18:29:47 2020
  read : io=1120.5MB, bw=114207KB/s, iops=28551, runt= 10046msec
    slat (usec): min=10, max=35562, avg=151.56, stdev=1019.33
    clat (usec): min=128, max=76939, avg=6168.49, stdev=5043.70
     lat (usec): min=144, max=76950, avg=6320.05, stdev=5146.71
    clat percentiles (usec):
     |  1.00th=[  478],  5.00th=[  892], 10.00th=[ 1208], 20.00th=[ 2288],
     | 30.00th=[ 3280], 40.00th=[ 4080], 50.00th=[ 4896], 60.00th=[ 5856],
     | 70.00th=[ 7200], 80.00th=[ 9152], 90.00th=[12608], 95.00th=[15936],
     | 99.00th=[23936], 99.50th=[28544], 99.90th=[39168], 99.95th=[41728],
     | 99.99th=[57600]
  write: io=491080KB, bw=48883KB/s, iops=12220, runt= 10046msec
    slat (usec): min=11, max=39358, avg=152.82, stdev=1020.66
    clat (usec): min=96, max=65071, avg=5429.24, stdev=4602.00
     lat (usec): min=116, max=71292, avg=5582.06, stdev=4718.41
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  676], 10.00th=[  988], 20.00th=[ 1896],
     | 30.00th=[ 2832], 40.00th=[ 3536], 50.00th=[ 4256], 60.00th=[ 5088],
     | 70.00th=[ 6240], 80.00th=[ 8032], 90.00th=[11200], 95.00th=[14272],
     | 99.00th=[21888], 99.50th=[25728], 99.90th=[36096], 99.95th=[39168],
     | 99.99th=[45312]
    lat (usec) : 100=0.01%, 250=0.01%, 500=2.05%, 750=2.33%, 1000=3.44%
    lat (msec) : 2=10.82%, 4=22.61%, 10=42.98%, 20=13.86%, 50=1.89%
    lat (msec) : 100=0.01%
  cpu          : usr=1.50%, sys=14.66%, ctx=45788, majf=0, minf=152
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=114206KB/s, minb=114206KB/s, maxb=114206KB/s, mint=10046msec, maxt=10046msec
  WRITE: io=491080KB, aggrb=48883KB/s, minb=48883KB/s, maxb=48883KB/s, mint=10046msec, maxt=10046msec
Disk stats (read/write):
  sdc: ios=285998/122419, merge=0/14, ticks=827030/320514, in_queue=1148660, util=99.09%
```

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=565: Fri Jun 12 18:30:45 2020
  read : io=1120.5MB, bw=118378KB/s, iops=29594, runt=  9692msec
    slat (usec): min=10, max=60895, avg=135.40, stdev=941.80
    clat (usec): min=217, max=73789, avg=5949.04, stdev=5072.25
     lat (usec): min=229, max=78761, avg=6084.44, stdev=5159.10
    clat percentiles (usec):
     |  1.00th=[  482],  5.00th=[  908], 10.00th=[ 1224], 20.00th=[ 2224],
     | 30.00th=[ 3120], 40.00th=[ 3856], 50.00th=[ 4576], 60.00th=[ 5472],
     | 70.00th=[ 6816], 80.00th=[ 8896], 90.00th=[12224], 95.00th=[15552],
     | 99.00th=[23680], 99.50th=[28032], 99.90th=[39168], 99.95th=[62208],
     | 99.99th=[70144]
  write: io=491080KB, bw=50669KB/s, iops=12667, runt=  9692msec
    slat (usec): min=11, max=61492, avg=144.49, stdev=979.97
    clat (usec): min=126, max=71350, avg=5292.25, stdev=4746.88
     lat (usec): min=139, max=71372, avg=5436.75, stdev=4848.67
    clat percentiles (usec):
     |  1.00th=[  370],  5.00th=[  668], 10.00th=[  980], 20.00th=[ 1864],
     | 30.00th=[ 2736], 40.00th=[ 3408], 50.00th=[ 4048], 60.00th=[ 4832],
     | 70.00th=[ 5984], 80.00th=[ 7840], 90.00th=[11072], 95.00th=[14144],
     | 99.00th=[22144], 99.50th=[26240], 99.90th=[39168], 99.95th=[61696],
     | 99.99th=[68096]
    lat (usec) : 250=0.01%, 500=1.95%, 750=2.34%, 1000=3.33%
    lat (msec) : 2=11.33%, 4=25.33%, 10=40.82%, 20=13.15%, 50=1.67%
    lat (msec) : 100=0.07%
  cpu          : usr=1.72%, sys=15.48%, ctx=43943, majf=0, minf=120
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=118378KB/s, minb=118378KB/s, maxb=118378KB/s, mint=9692msec, maxt=9692msec
  WRITE: io=491080KB, aggrb=50668KB/s, minb=50668KB/s, maxb=50668KB/s, mint=9692msec, maxt=9692msec
Disk stats (read/write):
  sdc: ios=286152/122458, merge=0/16, ticks=770130/301702, in_queue=1069440, util=99.08%
```

```
random-r70w30: (groupid=0, jobs=16): err= 0: pid=586: Fri Jun 12 18:31:36 2020
  read : io=1120.5MB, bw=118586KB/s, iops=29646, runt=  9675msec
    slat (usec): min=10, max=151811, avg=137.68, stdev=1105.94
    clat (usec): min=200, max=169899, avg=6034.38, stdev=6633.76
     lat (usec): min=211, max=169911, avg=6172.06, stdev=6724.33
    clat percentiles (usec):
     |  1.00th=[  474],  5.00th=[  860], 10.00th=[ 1160], 20.00th=[ 2096],
     | 30.00th=[ 3024], 40.00th=[ 3760], 50.00th=[ 4512], 60.00th=[ 5472],
     | 70.00th=[ 6880], 80.00th=[ 9024], 90.00th=[12352], 95.00th=[15680],
     | 99.00th=[23680], 99.50th=[27776], 99.90th=[99840], 99.95th=[152576],
     | 99.99th=[158720]
  write: io=491080KB, bw=50758KB/s, iops=12689, runt=  9675msec
    slat (usec): min=11, max=163218, avg=144.77, stdev=1214.29
    clat (usec): min=130, max=167947, avg=5343.10, stdev=6039.10
     lat (usec): min=155, max=169858, avg=5487.86, stdev=6156.90
    clat percentiles (usec):
     |  1.00th=[  366],  5.00th=[  636], 10.00th=[  940], 20.00th=[ 1736],
     | 30.00th=[ 2608], 40.00th=[ 3280], 50.00th=[ 3984], 60.00th=[ 4768],
     | 70.00th=[ 5984], 80.00th=[ 7904], 90.00th=[11200], 95.00th=[14272],
     | 99.00th=[21888], 99.50th=[25728], 99.90th=[96768], 99.95th=[116224],
     | 99.99th=[156672]
    lat (usec) : 250=0.01%, 500=2.16%, 750=2.53%, 1000=3.80%
    lat (msec) : 2=11.73%, 4=25.19%, 10=39.28%, 20=13.51%, 50=1.67%
    lat (msec) : 100=0.04%, 250=0.09%
  cpu          : usr=1.56%, sys=15.68%, ctx=42816, majf=0, minf=121
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=286830/w=122770/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
   READ: io=1120.5MB, aggrb=118586KB/s, minb=118586KB/s, maxb=118586KB/s, mint=9675msec, maxt=9675msec
  WRITE: io=491080KB, aggrb=50757KB/s, minb=50757KB/s, maxb=50757KB/s, mint=9675msec, maxt=9675msec
Disk stats (read/write):
  sdc: ios=286327/122573, merge=0/22, ticks=754016/294820, in_queue=1096580, util=99.05%
```

### IOPS サマリ

|#|Read IOPS|Write IOPS|
|--:|--:|--:|
|1回目|28551 |12220|
|2回目|29594 |12667|
|3回目|29646|12689|
|Avg| 29263.66 | 12525.33 |

こちらも1回目と2回目以降はやや差が出たものの、5%以内には収まっている感じです。  




# 結論


|#|Read IOPS Avg |Write IOPS Avg|
|--:|--:|--:|
|bind mount | 7405.33 | 3169.33 |
|local volume | 16752.33 | 7170.00 |
|bind mount (`invalidate=1`) | 12634.33 | 5407.33 |
|local volume (`invalidate=1`) | 29263.66 | 12525.33 |

local volume の方が Avg で100～130%くらい高速であると言えそうです。


Dev in Container with Docker on WSL2 では、  
依存関係のファイルは local volume に置こう。


## (参考資料) Visual Studio Code で開発する時の .devcontainer の設定

### `.devcontainer/docker-compose.yml`

```yaml
#-------------------------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See https://go.microsoft.com/fwlink/?linkid=2090316 for license information.
#-------------------------------------------------------------------------------------------------------------
services:
  node:
    # Uncomment the next line to use a non-root user for all processes. You can also
    # simply use the "remoteUser" property in devcontainer.json if you just want VS Code
    # and its sub-processes (terminals, tasks, debugging) to execute as the user. On Linux,
    # you may need to update USER_UID and USER_GID in .devcontainer/Dockerfile to match your
    # user if not 1000. See https://aka.ms/vscode-remote/containers/non-root for details.
    user: node

    build:
      context: .
      dockerfile: Dockerfile

    volumes:
      - ..:/workspace:cached
      - /workspace/node_modules

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
```

### `.devcontainer/Dockerfile`

permission 設定のため、自動生成されるファイルを一部改変してあります。

```yaml
#-------------------------------------------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See https://go.microsoft.com/fwlink/?linkid=2090316 for license information.
#-------------------------------------------------------------------------------------------------------------

FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-12

# The node image includes a non-root user with sudo access. Use the
# "remoteUser" property in devcontainer.json to use it. On Linux, update
# these values to ensure the container user's UID/GID matches your local values.
# See https://aka.ms/vscode-remote/containers/non-root-user for details.
ARG USERNAME=node
ARG USER_UID=1000
ARG USER_GID=$USER_UID

RUN mkdir -p /workspace/node_modules

# [Optional] Update UID/GID if needed
RUN if [ "$USER_GID" != "1000" ] || [ "$USER_UID" != "1000" ]; then \
        groupmod --gid $USER_GID $USERNAME \
        && usermod --uid $USER_UID --gid $USER_GID $USERNAME; \
    fi
RUN chown -R $USER_UID:$USER_GID /home/$USERNAME /workspace;
```