Mountpoint for Amazon S3を使ってS3をマウントしてみる。

# はじめに
以前から話題になっていた`Mountpoint for Amazon S3`が一般公開となったそうなので、使い勝手を確認してみようと思います。

- [Mountpoint for Amazon S3 の一般提供を開始](https://aws.amazon.com/jp/about-aws/whats-new/2023/08/mountpoint-amazon-s3-generally-available/)
- [Mountpoint for Amazon S3 – 一般公開されており、本番環境のワークロードにも対応可能](https://aws.amazon.com/jp/blogs/news/mountpoint-for-amazon-s3-generally-available-and-ready-for-production-workloads/)
- [GitHub - awslabs/mountpoint-s3](https://github.com/awslabs/mountpoint-s3)

# Mountpoint for Amazon S3とは
AWSのS3領域をEC2やECSにマウントして共有ディスクのように使えるアプリケーションです。

S3をマウントして使用できるアプリケーションは今までも`s3fs`や`goofys`などいくつかありましたが、`Mountpoint for Amazon S3`はAWSが公式で公開しているのが特徴です。

また、`Mountpoint for Amazon S3`はファイルシステムをマウントしたような感覚で使用することは出来ますが、`POSIX`規格には準拠していないため、`POSIX`規格に準拠したファイル操作と何が異なってくるのかを予め抑えておく必要があります。

# Mountpoint for Amazon S3の使用
`Mountpoint for Amazon S3`は簡単な操作で使用することが出来ます。

以下に実際にS3をマウントして使用できるようにするまでの手順をまとめます。

- [事前設定](#事前設定)
- [Mountpoint for Amazon S3のインストール](#mountpoint-for-amazon-s3のインストール)
- [Mountpoint for Amazon S3のマウント](#mountpoint-for-amazon-s3のマウント)
- [Mountpoint for Amazon S3のマウント解除](#mountpoint-for-amazon-s3のマウント解除)

## 事前設定
`Mountpoint for Amazon S3`は、要はクライアントアプリケーションを使ってEC2などのクライアントからS3に対してアクセスするものとなるので、当然クライアントからS3にアクセスするためのIAMユーザ・ロールの権限が必要となります。

以下は該当のS3バケットに対してフルアクセスするためのIAM権限の例です。

```json:IAM権限の例
{
   "Version": "2012-10-17",
   "Statement": [
        {
            "Sid": "MountpointFullBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET"
            ]
        },
        {
            "Sid": "MountpointFullObjectAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
            ]
        }
   ]
}
```

## Mountpoint for Amazon S3のインストール
今回は適当なEC2を用意してS3をマウントしたいと思います。

`Mountpoint for Amazon S3`自体のインストールは簡単で、RPMパッケージをダウンロードして`yum`でインストールするだけです。

[公式GitHub](https://github.com/awslabs/mountpoint-s3)にはOSごとに２種類のインストール方法が紹介されております。

`Graviton`インスタンスにインストールする場合は、`x86_64`の部分を`arm64`に変えると良いそうです。

```bash:Mountpoint for Amazon S3のインストール（x86_64版Amazon Linux）
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm
```

```bash:Mountpoint for Amazon S3のインストール（x86_64版Ubuntu）
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb
```

## Mountpoint for Amazon S3のマウント
`Mountpoint for Amazon S3`をインストールすることで、`mount-s3 [S3バケット名] [マウントポイント]`の書式でS3をマウントできるようになります。

以下は`mountpoint-for-s3-test`バケットを、新たに作成した`s3mount`というマウントポイントにマウントする場合の例。

```bash:mount-s3コマンドによるS3マウント
mkdir s3mount
mount-s3 mountpoint-for-s3-test s3mount
```

以下のようにメッセージが表示されれば正常にマウントできています。

```bash:マウント時のメッセージ例
bucket mountpoint-for-s3-test is mounted at s3mount
```

また、上記のデフォルトマウントコマンドではS3上のオブジェクトを削除できないように制限されているため、削除も行いたい場合は`--allow-delete`オプションを付与して以下のようにマウントします。

```bash:削除オプションを付与したS3マウント
mount-s3 --allow-delete mountpoint-for-s3-test s3mount
```

`--prefix [プレフィックス名]/`を指定することでS3バケット全体ではなく、特定のプレフィックス（フォルダ）部分のみマウントすることもできるので、特定の領域のみユーザに使わせるといったことも可能です。

なお、`--prefix`オプションを付与する際には最後に「**/**」を付与するようにしてください。

```bash:prefixオプションを付与したS3マウント
mount-s3 --allow-delete --prefix testdir/ mountpoint-for-s3-test s3mount
```

## Mountpoint for Amazon S3のマウント解除
マウント解除は通常の`umount`で解除します。

```bash:S3マウントの解除
umount /s3mount
```

# Mountpoint for Amazon S3の確認
上記手順でS3バケットをマウントすることが出来ましたが、一般的なファイルシステムをマウントした場合と比べてどのような違いがあるのかを、実際にコマンド実行しながら確認していきたいと思います。

- [マウント時のコマンド確認](#マウント時のコマンド確認)
- [ファイルリネーム](#ファイルリネーム)
- [ファイル削除](#ファイル削除)
- [ファイル追記](#ファイル追記)
- [ディレクトリの作成](#ディレクトリの作成)
- [リンク作成](#リンク作成)

## マウント時のコマンド確認
共有ディスクをマウントした場合等、`df`コマンドを実行するとマウント済みの領域が表示されますが、`Mountpoint for Amazon S3`でマウントした場合は`df`コマンドでは確認できません。

```bash:dfコマンドでの確認例
# df -k
Filesystem            1K-blocks    Used        Available Use% Mounted on
devtmpfs                   4096       0             4096   0% /dev
tmpfs                    486288       0           486288   0% /dev/shm
tmpfs                    194516    2904           191612   2% /run
/dev/xvda1              8310764 1640856          6669908  20% /
tmpfs                    486288       0           486288   0% /tmp
tmpfs                     97256       0            97256   0% /run/user/0
127.0.0.1:/    9007199254739968       0 9007199254739968   0% /efs
/dev/xvdf             104792064  763656        104028408   1% /io2
```

`mount`コマンドで確認することはできるので、マウントできているかを確認する場合は`mount`コマンドで確認するのが良いかと思います。

```bash:mountコマンドで確認した場合の例
# mount
（略）
mountpoint-s3 on /s3mount type fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions)
```

## ファイルリネーム
ファイルリネームには対応していないため実行すると以下のように表示されます。

```bash:ファイルリネーム例
# mv testfile testfile.bak
mv: cannot move 'testfile' to 'testfile.bak': Function not implemented
```

## ファイル削除
前述の通り、`--allow-delete`を付与しないと以下のように失敗します。

```bash:ファイルの削除例
# rm testfile
rm: remove regular empty file 'testfile'? y
rm: cannot remove 'testfile': Operation not permitted
```

また、`--allow-delete`を付与しても、そもそもIAMロールやS3バケットポリシーで削除が許可されていないとできないので注意してください。

## ファイル追記
ファイルの追記は以下のように失敗してしまいます。

```bash:ファイルの追記例
# echo test >> testfile
-bash: testfile: Operation not permitted
```

これは通常のS3操作を思い浮かべてもらえればイメージできるかと思いますが、そもそもS3に代表されるオブジェクトストレージは追記を想定したストレージではないので`Mountpoint for Amazon S3`でも追記することは出来ません。

一応`vi`での作成・編集は新規ファイルをコピーするような動作となるため出来なくはないようですが、編集する際に以下のようなメッセージが表示され、編集後、スワップファイルが残ってしまうので編集は出来ないものと考えておきましょう。

```bash:viでファイル編集した際のメッセージ
E325: ATTENTION
Found a swap file by the name ".testfile.swp"
          owned by: root   dated: Sat Aug 19 00:19:13 2023
         [cannot be read]
While opening file "testfile"
             dated: Sat Aug 19 00:19:27 2023
      NEWER than swap file!

(1) Another program may be editing the same file.  If this is the case,
    be careful not to end up with two different instances of the same
    file when making changes.  Quit, or continue with caution.
(2) An edit session for this file crashed.
    If this is the case, use ":recover" or "vim -r testfile"
    to recover the changes (see ":help recovery").
    If you did this already, delete the swap file ".testfile.swp"
    to avoid this message.

Swap file ".testfile.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort:
```

## ディレクトリの作成
ディレクトリの作成は通常のディレクトリ作成と同様`mkdir`コマンドで作成することができます。

但し、ディレクトリの中身に何もファイルがない状態の場合、ローカル上での操作となり、S3にアップロードされません。

作成したディレクトリにファイルが格納されることでS3側にアップロードされる動作となるのでご注意ください。

```bash:S3領域にディレクトリのみ作成した場合のS3バケットの状態
# mkdir -p /s3mount/testdir1/testdir2
# tree /s3mount
/s3mount
└── testdir1
    └── testdir2

2 directories, 0 files
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
#
```

```bash:作成したディレクトリ配下にファイルを作成した場合のS3バケットの状態
# touch /s3mount/testdir1/testdir2/testfile
# tree /s3mount
/s3mount
└── testdir1
    └── testdir2
        └── testfile

2 directories, 1 file
# aws s3 ls s3://mountpoint-for-s3-test/ --recursive
2023-08-19 01:47:52          0 testdir/testdir1/testdir2/testfile
```

## リンク作成
`ln`コマンドで作成できるハードリンク、シンボリックリンクはどちらもサポートされておりません。

```bash:シンボリックリンクの作成例
# ln -s /var/log testlink
ln: failed to create symbolic link 'testlink': Operation not permitted
```

# Mountpoint for Amazon S3のベンチマーク
`Mountpoint for Amazon S3`でマウントした領域はどの程度の性能が出るのか、ベンチマークをしてみようと思います。

## ベンチマーク準備
今回は`EBS`、`EFS`、`Mountpoint for Amazon S3`の３つを比較してみようと思います。

以下それぞれのタイプ等。

|ベンチマーク対象|設定・タイプ|マウントポイント|
|:--|:--|:--|
|EBS|ボリュームタイプ：プロビジョンドIOPS（io2）<br>サイズ：100GB<br>IOPS：3000|/io2|
|EFS|パフォーマンスモード：汎用<br>スループットモード：伸縮自在<br>EFSマウントヘルパー使用|/efs|
|Mountpoint for Amazon S3|ストレージクラス：スタンダード<br>`--allow-delete`オプション付与|/s3mount|

また、ベンチマークには`Amazon Linux 2023`の`t3.small`を使用します。

## fioの設定
ちょうど「[EBSボリュームのベンチマーク](https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/benchmark_procedures.html)」のドキュメントに`fio`を使用したベンチマーク方法が書かれていたので、こちらを参考に`CrystalDiskMark`風なパターンでベンチマークしてみようと思います。

ベンチマークで使用した`fio`の設定は以下となります。

```text:fio.config
[global]
ioengine=libaio
iodepth=1
size=1g
direct=1
runtime=60
directory=/s3mount
filename=.fio-benchmark
norandommap
group_reporting
stonewall

[Seq-Read-1M-Q8-T1]
rw=read
bs=1m
iodepth=8
numjobs=1

[Seq-Write-1M-Q8-T1]
rw=write
bs=1m
iodepth=8
numjobs=1

[Seq-Read-1M-Q1-T1]
rw=read
bs=1m
iodepth=1
numjobs=1

[Seq-Write-1M-Q1-T1]
rw=write
bs=1m
iodepth=1
numjobs=1

[Rand-Read-4K-Q32-T16]
rw=randread
bs=4k
iodepth=32
numjobs=16

[Rand-Write-4K-Q32-T16]
rw=randwrite
bs=4k
iodepth=32
numjobs=16

[Rand-Read-4K-Q1-T1]
rw=randread
bs=4k
iodepth=1
numjobs=1

[Rand-Write-4K-Q1-T1]
rw=randwrite
bs=4k
iodepth=1
numjobs=1
```

`directory`部分を`EBS`や`EFS`、`Mountpoint for Amazon S3`をマウントしたパスに書き換えて以下のように実行します。

```bash:fioの実行
fio fio.config
```

## ベンチマーク実行結果
ベンチマークを行った結果が以下となります。

`Mountpoint for Amazon S3`のWriteの性能を「ー」としておりますが、上述の通り追記等の処理が行えないため`fio`を実行してもエラーとなるためです。

EBSのランダムRead・WriteはIOPSの上限に引っかかっているかと思うので、実際はもっと性能が出るかと思いますが、今回は`Mountpoint for Amazon S3`の評価が本題なのでこのままとします。

|パターン|EBS転送速度|EBS IOPS|EFS転送速度|EFS IOPS|S3転送速度|S3 IOPS|
|:--|:--|:--|:--|:--|:--|:--|
|シーケンシャルRead<br>1M Q8 T1|344MB/s|327|213MB/s|203|260MB/s|247|
|シーケンシャルWrite<br>1M Q8 T1|280MB/s|267|144MB/s|137|ー|ー|
|シーケンシャルRead<br>1M Q1 T1|286MB/s|272|47.2MB/s|45|296MB/s|282|
|シーケンシャルWrite<br>1M Q1 T1|285MB/s|272|34.9MB/s|33|ー|ー|
|ランダムRead<br>4K Q32 T16|12.5MB/s|3049|43.7MB/s|10.7k|382kB/s|93|
|ランダムWrite<br>4K Q32 T16|12.3MB/s|3015|19.3MB/s|4702|ー|ー|
|ランダムRead<br>4K Q1 T1|12.4MB/s|3019|1241kB/s|302|51.3kB/s|12|
|ランダムWrite<br>4K Q1 T1|12.3MB/s|3014|442kB/s|107|ー|ー|

各パターンの詳細を見たい方は以下展開して確認してください。

<details><summary>EBS結果（展開してください）</summary>

```bash:EBSのfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Jobs: 1 (f=0): [_(37),f(1)][100.0%][w=11.0MiB/s][w=2804 IOPS][eta 00m:00s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=156247: Sun Aug 20 02:14:52 2023
  read: IOPS=327, BW=328MiB/s (344MB/s)(1024MiB/3122msec)
    slat (usec): min=23, max=6936, avg=2013.71, stdev=1536.82
    clat (usec): min=2149, max=53106, avg=22330.75, stdev=11665.09
     lat (usec): min=2180, max=56222, avg=24344.46, stdev=12859.52
    clat percentiles (usec):
     |  1.00th=[ 2573],  5.00th=[ 3032], 10.00th=[ 3458], 20.00th=[ 3949],
     | 30.00th=[28443], 40.00th=[28705], 50.00th=[28967], 60.00th=[28967],
     | 70.00th=[29230], 80.00th=[29230], 90.00th=[30016], 95.00th=[32113],
     | 99.00th=[33162], 99.50th=[36963], 99.90th=[49021], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=253952, max=647739, per=99.21%, avg=333220.60, stdev=175823.13, samples=5
   iops        : min=  248, max=  632, avg=325.20, stdev=171.51, samples=5
  lat (msec)   : 4=20.80%, 10=6.74%, 20=0.29%, 50=72.07%, 100=0.10%
  cpu          : usr=0.54%, sys=1.86%, ctx=1651, majf=0, minf=2059
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=156251: Sun Aug 20 02:14:52 2023
  write: IOPS=267, BW=267MiB/s (280MB/s)(1024MiB/3830msec); 0 zone resets
    slat (usec): min=44, max=16322, avg=2231.71, stdev=1718.08
    clat (usec): min=1655, max=53202, avg=27652.55, stdev=8100.48
     lat (usec): min=1732, max=56471, avg=29884.27, stdev=8665.53
    clat percentiles (usec):
     |  1.00th=[ 3163],  5.00th=[ 3458], 10.00th=[21890], 20.00th=[28705],
     | 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[28967],
     | 70.00th=[28967], 80.00th=[31589], 90.00th=[32113], 95.00th=[34866],
     | 99.00th=[46400], 99.50th=[48497], 99.90th=[52691], 99.95th=[53216],
     | 99.99th=[53216]
   bw (  KiB/s): min=253952, max=354837, per=98.29%, avg=269094.43, stdev=37818.95, samples=7
   iops        : min=  248, max=  346, avg=262.57, stdev=36.80, samples=7
  lat (msec)   : 2=0.10%, 4=7.81%, 10=0.39%, 20=1.46%, 50=89.94%
  lat (msec)   : 100=0.29%
  cpu          : usr=1.88%, sys=1.41%, ctx=1678, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=156257: Sun Aug 20 02:14:52 2023
  read: IOPS=272, BW=273MiB/s (286MB/s)(1024MiB/3751msec)
    slat (usec): min=20, max=4781, avg=47.30, stdev=152.79
    clat (usec): min=684, max=7294, avg=3609.84, stdev=1057.06
     lat (usec): min=746, max=7328, avg=3657.14, stdev=1058.08
    clat percentiles (usec):
     |  1.00th=[  750],  5.00th=[  791], 10.00th=[  865], 20.00th=[ 3916],
     | 30.00th=[ 3949], 40.00th=[ 3982], 50.00th=[ 3982], 60.00th=[ 3982],
     | 70.00th=[ 3982], 80.00th=[ 4015], 90.00th=[ 4015], 95.00th=[ 4047],
     | 99.00th=[ 5014], 99.50th=[ 5669], 99.90th=[ 7242], 99.95th=[ 7308],
     | 99.99th=[ 7308]
   bw (  KiB/s): min=253952, max=428740, per=100.00%, avg=279652.00, stdev=65747.39, samples=7
   iops        : min=  248, max=  418, avg=272.86, stdev=64.01, samples=7
  lat (usec)   : 750=0.68%, 1000=9.96%
  lat (msec)   : 2=1.27%, 4=68.75%, 10=19.34%
  cpu          : usr=0.88%, sys=0.91%, ctx=1025, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=156262: Sun Aug 20 02:14:52 2023
  write: IOPS=272, BW=272MiB/s (285MB/s)(1024MiB/3764msec); 0 zone resets
    slat (usec): min=42, max=1731, avg=105.84, stdev=76.87
    clat (usec): min=1130, max=8655, avg=3564.31, stdev=965.57
     lat (usec): min=1172, max=8732, avg=3670.15, stdev=972.52
    clat percentiles (usec):
     |  1.00th=[ 1156],  5.00th=[ 1205], 10.00th=[ 1270], 20.00th=[ 3818],
     | 30.00th=[ 3884], 40.00th=[ 3884], 50.00th=[ 3916], 60.00th=[ 3916],
     | 70.00th=[ 3949], 80.00th=[ 3949], 90.00th=[ 3982], 95.00th=[ 4015],
     | 99.00th=[ 4883], 99.50th=[ 5800], 99.90th=[ 6849], 99.95th=[ 8717],
     | 99.99th=[ 8717]
   bw (  KiB/s): min=253952, max=387786, per=98.28%, avg=273801.43, stdev=50270.01, samples=7
   iops        : min=  248, max=  378, avg=267.14, stdev=48.89, samples=7
  lat (msec)   : 2=13.38%, 4=80.18%, 10=6.45%
  cpu          : usr=2.02%, sys=0.93%, ctx=1025, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=156267: Sun Aug 20 02:14:52 2023
  read: IOPS=3049, BW=11.9MiB/s (12.5MB/s)(715MiB/60027msec)
    slat (nsec): min=1924, max=128039k, avg=5238465.70, stdev=9531041.06
    clat (usec): min=181, max=474989, avg=162614.04, stdev=51900.61
     lat (usec): min=189, max=475000, avg=167852.51, stdev=52879.82
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   80], 10.00th=[  104], 20.00th=[  126],
     | 30.00th=[  140], 40.00th=[  153], 50.00th=[  163], 60.00th=[  176],
     | 70.00th=[  188], 80.00th=[  203], 90.00th=[  226], 95.00th=[  247],
     | 99.00th=[  284], 99.50th=[  300], 99.90th=[  342], 99.95th=[  363],
     | 99.99th=[  426]
   bw (  KiB/s): min= 7305, max=31279, per=99.67%, avg=12159.87, stdev=168.93, samples=1904
   iops        : min= 1826, max= 7814, avg=3037.86, stdev=42.21, samples=1904
  lat (usec)   : 250=0.04%, 500=0.10%, 750=0.03%, 1000=0.03%
  lat (msec)   : 2=0.20%, 4=0.62%, 10=0.45%, 20=0.27%, 50=0.92%
  lat (msec)   : 100=6.51%, 250=86.52%, 500=4.31%
  cpu          : usr=0.12%, sys=0.20%, ctx=135988, majf=0, minf=677
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=183078,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=156404: Sun Aug 20 02:14:52 2023
  write: IOPS=3015, BW=11.8MiB/s (12.3MB/s)(707MiB/60027msec); 0 zone resets
    slat (usec): min=2, max=296642, avg=5298.40, stdev=15383.26
    clat (usec): min=138, max=590686, avg=164448.26, stdev=71623.19
     lat (usec): min=242, max=590689, avg=169746.66, stdev=72696.41
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   73], 10.00th=[   86], 20.00th=[  104],
     | 30.00th=[  120], 40.00th=[  136], 50.00th=[  153], 60.00th=[  171],
     | 70.00th=[  194], 80.00th=[  220], 90.00th=[  259], 95.00th=[  296],
     | 99.00th=[  380], 99.50th=[  409], 99.90th=[  481], 99.95th=[  493],
     | 99.99th=[  542]
   bw (  KiB/s): min= 5264, max=20646, per=99.82%, avg=12038.22, stdev=172.27, samples=1897
   iops        : min= 1316, max= 5156, avg=3007.51, stdev=43.04, samples=1897
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.09%
  lat (msec)   : 2=0.07%, 4=0.02%, 10=0.08%, 20=0.05%, 50=1.03%
  lat (msec)   : 100=16.45%, 250=70.06%, 500=12.04%, 750=0.03%
  cpu          : usr=0.10%, sys=0.18%, ctx=162798, majf=0, minf=159
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,180986,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=156539: Sun Aug 20 02:14:52 2023
  read: IOPS=3019, BW=11.8MiB/s (12.4MB/s)(708MiB/60001msec)
    slat (usec): min=2, max=12040, avg= 6.52, stdev=46.70
    clat (nsec): min=1999, max=15429k, avg=322639.92, stdev=216497.01
     lat (usec): min=89, max=15432, avg=329.16, stdev=220.85
    clat percentiles (usec):
     |  1.00th=[  102],  5.00th=[  113], 10.00th=[  123], 20.00th=[  161],
     | 30.00th=[  297], 40.00th=[  314], 50.00th=[  322], 60.00th=[  330],
     | 70.00th=[  338], 80.00th=[  355], 90.00th=[  506], 95.00th=[  537],
     | 99.00th=[  889], 99.50th=[ 1221], 99.90th=[ 2835], 99.95th=[ 3949],
     | 99.99th=[ 6325]
   bw (  KiB/s): min=11823, max=19900, per=99.97%, avg=12075.57, stdev=727.08, samples=118
   iops        : min= 2955, max= 4975, avg=3018.85, stdev=181.78, samples=118
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.64%, 250=23.47%, 500=64.79%, 750=9.51%, 1000=0.79%
  lat (msec)   : 2=0.58%, 4=0.16%, 10=0.04%, 20=0.01%
  cpu          : usr=1.45%, sys=2.80%, ctx=181235, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=181188,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=156603: Sun Aug 20 02:14:52 2023
  write: IOPS=3014, BW=11.8MiB/s (12.3MB/s)(707MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=7151, avg= 7.62, stdev=40.18
    clat (usec): min=2, max=9981, avg=321.91, stdev=193.70
     lat (usec): min=177, max=9987, avg=329.54, stdev=197.56
    clat percentiles (usec):
     |  1.00th=[  190],  5.00th=[  200], 10.00th=[  210], 20.00th=[  258],
     | 30.00th=[  302], 40.00th=[  314], 50.00th=[  322], 60.00th=[  326],
     | 70.00th=[  334], 80.00th=[  343], 90.00th=[  355], 95.00th=[  392],
     | 99.00th=[  766], 99.50th=[ 1254], 99.90th=[ 3359], 99.95th=[ 4113],
     | 99.99th=[ 5997]
   bw (  KiB/s): min=11512, max=17744, per=100.00%, avg=12071.06, stdev=547.70, samples=119
   iops        : min= 2878, max= 4436, avg=3017.76, stdev=136.93, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=19.19%, 500=78.69%, 750=1.07%, 1000=0.35%
  lat (msec)   : 2=0.42%, 4=0.20%, 10=0.06%
  cpu          : usr=1.38%, sys=2.23%, ctx=181109, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,180898,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=1024MiB (1074MB), run=3122-3122msec

Run status group 1 (all jobs):
  WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=1024MiB (1074MB), run=3830-3830msec

Run status group 2 (all jobs):
   READ: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=1024MiB (1074MB), run=3751-3751msec

Run status group 3 (all jobs):
  WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=1024MiB (1074MB), run=3764-3764msec

Run status group 4 (all jobs):
   READ: bw=11.9MiB/s (12.5MB/s), 11.9MiB/s-11.9MiB/s (12.5MB/s-12.5MB/s), io=715MiB (750MB), run=60027-60027msec

Run status group 5 (all jobs):
  WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60027-60027msec

Run status group 6 (all jobs):
   READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=708MiB (742MB), run=60001-60001msec

Run status group 7 (all jobs):
  WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=372458/369537, merge=0/0, ticks=3817558/3740979, in_queue=7558537, util=99.15%
```
</details>

<details><summary>EFS結果（展開してください）</summary>

```bash:EFSのfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
Seq-Read-1M-Q8-T1: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [_(37),w(1)][13.1%][w=440KiB/s][w=110 IOPS][eta 34m:04s]
Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=165964: Sun Aug 20 03:56:48 2023
  read: IOPS=203, BW=203MiB/s (213MB/s)(1024MiB/5037msec)
    slat (usec): min=49, max=2487, avg=139.14, stdev=149.19
    clat (msec): min=16, max=236, avg=39.04, stdev=24.60
     lat (msec): min=16, max=236, avg=39.18, stdev=24.60
    clat percentiles (msec):
     |  1.00th=[   20],  5.00th=[   22], 10.00th=[   23], 20.00th=[   26],
     | 30.00th=[   29], 40.00th=[   31], 50.00th=[   33], 60.00th=[   35],
     | 70.00th=[   39], 80.00th=[   44], 90.00th=[   56], 95.00th=[   91],
     | 99.00th=[  148], 99.50th=[  163], 99.90th=[  228], 99.95th=[  236],
     | 99.99th=[  236]
   bw (  KiB/s): min=175028, max=235991, per=100.00%, avg=209907.00, stdev=18602.14, samples=9
   iops        : min=  170, max=  230, avg=204.67, stdev=18.25, samples=9
  lat (msec)   : 20=2.64%, 50=84.86%, 100=8.11%, 250=4.39%
  cpu          : usr=0.10%, sys=1.89%, ctx=1552, majf=0, minf=2058
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 0: pid=165970: Sun Aug 20 03:56:48 2023
  write: IOPS=137, BW=137MiB/s (144MB/s)(1024MiB/7450msec); 0 zone resets
    slat (usec): min=91, max=3787, avg=186.43, stdev=202.13
    clat (msec): min=20, max=302, avg=57.91, stdev=29.14
     lat (msec): min=21, max=302, avg=58.09, stdev=29.14
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   32], 20.00th=[   37],
     | 30.00th=[   45], 40.00th=[   51], 50.00th=[   56], 60.00th=[   61],
     | 70.00th=[   66], 80.00th=[   71], 90.00th=[   79], 95.00th=[   87],
     | 99.00th=[  186], 99.50th=[  241], 99.90th=[  288], 99.95th=[  305],
     | 99.99th=[  305]
   bw (  KiB/s): min=126470, max=151552, per=99.97%, avg=140701.07, stdev=8646.52, samples=14
   iops        : min=  123, max=  148, avg=137.29, stdev= 8.48, samples=14
  lat (msec)   : 50=39.36%, 100=57.42%, 250=2.73%, 500=0.49%
  cpu          : usr=0.78%, sys=1.53%, ctx=1046, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=165979: Sun Aug 20 03:56:48 2023
  read: IOPS=45, BW=45.0MiB/s (47.2MB/s)(1024MiB/22740msec)
    slat (usec): min=61, max=7907, avg=100.47, stdev=300.69
    clat (usec): min=13306, max=97495, avg=22095.31, stdev=7013.49
     lat (usec): min=13782, max=97571, avg=22195.78, stdev=7025.72
    clat percentiles (usec):
     |  1.00th=[16581],  5.00th=[17433], 10.00th=[17957], 20.00th=[18482],
     | 30.00th=[19006], 40.00th=[19530], 50.00th=[20055], 60.00th=[20841],
     | 70.00th=[21890], 80.00th=[23987], 90.00th=[27657], 95.00th=[31589],
     | 99.00th=[53740], 99.50th=[61080], 99.90th=[95945], 99.95th=[96994],
     | 99.99th=[96994]
   bw (  KiB/s): min=28500, max=53354, per=99.86%, avg=46045.75, stdev=5430.42, samples=44
   iops        : min=   27, max=   52, avg=44.84, stdev= 5.40, samples=44
  lat (msec)   : 20=47.36%, 50=51.07%, 100=1.56%
  cpu          : usr=0.05%, sys=0.38%, ctx=1044, majf=0, minf=265
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 0: pid=166006: Sun Aug 20 03:56:48 2023
  write: IOPS=33, BW=33.3MiB/s (34.9MB/s)(1024MiB/30761msec); 0 zone resets
    slat (usec): min=102, max=3276, avg=179.53, stdev=153.81
    clat (msec): min=20, max=422, avg=29.85, stdev=18.51
     lat (msec): min=21, max=423, avg=30.03, stdev=18.51
    clat percentiles (msec):
     |  1.00th=[   24],  5.00th=[   25], 10.00th=[   26], 20.00th=[   26],
     | 30.00th=[   27], 40.00th=[   28], 50.00th=[   28], 60.00th=[   29],
     | 70.00th=[   30], 80.00th=[   31], 90.00th=[   34], 95.00th=[   36],
     | 99.00th=[   48], 99.50th=[  113], 99.90th=[  359], 99.95th=[  422],
     | 99.99th=[  422]
   bw (  KiB/s): min= 6144, max=38989, per=99.77%, avg=34011.23, stdev=5932.22, samples=60
   iops        : min=    6, max=   38, avg=33.12, stdev= 5.81, samples=60
  lat (msec)   : 50=99.02%, 100=0.39%, 250=0.39%, 500=0.20%
  cpu          : usr=0.29%, sys=0.29%, ctx=1033, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=166042: Sun Aug 20 03:56:48 2023
  read: IOPS=10.7k, BW=41.7MiB/s (43.7MB/s)(2504MiB/60037msec)
    slat (nsec): min=1881, max=9893.2k, avg=23118.50, stdev=73466.89
    clat (msec): min=4, max=178, avg=47.92, stdev= 9.80
     lat (msec): min=4, max=178, avg=47.94, stdev= 9.80
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   37], 10.00th=[   39], 20.00th=[   41],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   47], 60.00th=[   48],
     | 70.00th=[   51], 80.00th=[   54], 90.00th=[   60], 95.00th=[   65],
     | 99.00th=[   79], 99.50th=[   87], 99.90th=[  146], 99.95th=[  157],
     | 99.99th=[  167]
   bw (  KiB/s): min=24408, max=55360, per=100.00%, avg=42738.87, stdev=325.55, samples=1904
   iops        : min= 6090, max=13840, avg=10682.66, stdev=81.36, samples=1904
  lat (msec)   : 10=0.01%, 20=0.02%, 50=67.94%, 100=31.83%, 250=0.21%
  cpu          : usr=0.38%, sys=0.94%, ctx=1035226, majf=0, minf=652
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=641041,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 0: pid=166176: Sun Aug 20 03:56:48 2023
  write: IOPS=4702, BW=18.4MiB/s (19.3MB/s)(1104MiB/60099msec); 0 zone resets
    slat (usec): min=2, max=8942, avg=32.17, stdev=100.44
    clat (msec): min=10, max=1307, avg=108.81, stdev=52.17
     lat (msec): min=10, max=1307, avg=108.84, stdev=52.17
    clat percentiles (msec):
     |  1.00th=[   94],  5.00th=[   96], 10.00th=[   97], 20.00th=[  100],
     | 30.00th=[  101], 40.00th=[  103], 50.00th=[  104], 60.00th=[  105],
     | 70.00th=[  107], 80.00th=[  110], 90.00th=[  115], 95.00th=[  121],
     | 99.00th=[  161], 99.50th=[  634], 99.90th=[  953], 99.95th=[  961],
     | 99.99th=[  961]
   bw (  KiB/s): min= 6752, max=20938, per=100.00%, avg=19142.83, stdev=125.47, samples=1856
   iops        : min= 1688, max= 5234, avg=4782.65, stdev=31.37, samples=1856
  lat (msec)   : 20=0.01%, 50=0.05%, 100=26.98%, 250=72.25%, 500=0.16%
  lat (msec)   : 750=0.22%, 1000=0.33%, 2000=0.01%
  cpu          : usr=0.20%, sys=0.39%, ctx=523481, majf=0, minf=151
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,282635,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=166311: Sun Aug 20 03:56:48 2023
  read: IOPS=302, BW=1212KiB/s (1241kB/s)(71.0MiB/60002msec)
    slat (usec): min=8, max=3208, avg=17.99, stdev=56.87
    clat (usec): min=594, max=29700, avg=3274.09, stdev=1037.00
     lat (usec): min=606, max=29721, avg=3292.08, stdev=1040.45
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[ 2507], 10.00th=[ 2638], 20.00th=[ 2769],
     | 30.00th=[ 2868], 40.00th=[ 2966], 50.00th=[ 3064], 60.00th=[ 3261],
     | 70.00th=[ 3458], 80.00th=[ 3687], 90.00th=[ 4228], 95.00th=[ 5145],
     | 99.00th=[ 6915], 99.50th=[ 8094], 99.90th=[10683], 99.95th=[11731],
     | 99.99th=[16909]
   bw (  KiB/s): min= 1029, max= 1376, per=100.00%, avg=1212.30, stdev=63.28, samples=119
   iops        : min=  257, max=  344, avg=302.95, stdev=15.85, samples=119
  lat (usec)   : 750=2.94%, 1000=0.69%
  lat (msec)   : 2=0.16%, 4=83.86%, 10=12.19%, 20=0.16%, 50=0.01%
  cpu          : usr=0.35%, sys=0.53%, ctx=18381, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=18180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 0: pid=166374: Sun Aug 20 03:56:48 2023
  write: IOPS=107, BW=432KiB/s (442kB/s)(25.3MiB/60008msec); 0 zone resets
    slat (usec): min=13, max=1871, avg=23.89, stdev=38.04
    clat (usec): min=6872, max=74459, avg=9229.58, stdev=1743.69
     lat (usec): min=6941, max=74477, avg=9253.46, stdev=1744.96
    clat percentiles (usec):
     |  1.00th=[ 7373],  5.00th=[ 7635], 10.00th=[ 7832], 20.00th=[ 8094],
     | 30.00th=[ 8291], 40.00th=[ 8586], 50.00th=[ 8848], 60.00th=[ 9241],
     | 70.00th=[ 9634], 80.00th=[10028], 90.00th=[10945], 95.00th=[12387],
     | 99.00th=[14484], 99.50th=[15795], 99.90th=[20579], 99.95th=[24773],
     | 99.99th=[74974]
   bw (  KiB/s): min=  368, max=  456, per=100.00%, avg=432.16, stdev=13.20, samples=119
   iops        : min=   92, max=  114, avg=108.02, stdev= 3.30, samples=119
  lat (msec)   : 10=79.13%, 20=20.72%, 50=0.14%, 100=0.02%
  cpu          : usr=0.24%, sys=0.22%, ctx=6553, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,6478,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1024MiB (1074MB), run=5037-5037msec

Run status group 1 (all jobs):
  WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=1024MiB (1074MB), run=7450-7450msec

Run status group 2 (all jobs):
   READ: bw=45.0MiB/s (47.2MB/s), 45.0MiB/s-45.0MiB/s (47.2MB/s-47.2MB/s), io=1024MiB (1074MB), run=22740-22740msec

Run status group 3 (all jobs):
  WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1024MiB (1074MB), run=30761-30761msec

Run status group 4 (all jobs):
   READ: bw=41.7MiB/s (43.7MB/s), 41.7MiB/s-41.7MiB/s (43.7MB/s-43.7MB/s), io=2504MiB (2626MB), run=60037-60037msec

Run status group 5 (all jobs):
  WRITE: bw=18.4MiB/s (19.3MB/s), 18.4MiB/s-18.4MiB/s (19.3MB/s-19.3MB/s), io=1104MiB (1158MB), run=60099-60099msec

Run status group 6 (all jobs):
   READ: bw=1212KiB/s (1241kB/s), 1212KiB/s-1212KiB/s (1241kB/s-1241kB/s), io=71.0MiB (74.5MB), run=60002-60002msec

Run status group 7 (all jobs):
  WRITE: bw=432KiB/s (442kB/s), 432KiB/s-432KiB/s (442kB/s-442kB/s), io=25.3MiB (26.5MB), run=60008-60008msec
```
</details>

<details><summary>S3結果（展開してください）</summary>

```bash:S3のfio結果
Seq-Read-1M-Q8-T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Write-1M-Q8-T1: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Seq-Read-1M-Q1-T1: (g=2): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Seq-Write-1M-Q1-T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Rand-Read-4K-Q32-T16: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Write-4K-Q32-T16: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
...
Rand-Read-4K-Q1-T1: (g=6): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Rand-Write-4K-Q1-T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32
Starting 38 processes
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1048576, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=2097152, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=3145728, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4194304, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=5242880, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=6291456, buflen=1048576
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=7340032, buflen=1048576
fio: pid=160247, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=0, buflen=1048576
fio: pid=160253, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=935215104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=857464832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468156416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=985276416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8654848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1052733440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=18911232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119975936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=176386048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547172352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=81301504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=459362304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=710455296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=295763968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1033744384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117538816, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495435776, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669491200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=294543360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384147456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=914653184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=477323264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1014996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=792219648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912949248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542818304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746360832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=936366080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251334656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784572416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=70127616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518848512, buflen=4096
fio: pid=160333, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=263307264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615055360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=532480000, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=384966656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=421335040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61181952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=708857856, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=121028608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313151488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=969424896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824680448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780730368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976297984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478814208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816156672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89661440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14708736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=16232448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305840128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=32333824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642404352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538820608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=858578944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=930562048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578523136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=753684480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038864384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=864620544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=504500224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=402882560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=838029312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954761216, buflen=4096
fio: pid=160338, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=523722752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=542588928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=308756480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=268566528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=293478400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=146636800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=371425280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=955932672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=382050304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=310792192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775139328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=548966400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=71241728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941813760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=483074048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=798691328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669417472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=886464512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12976128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=669818880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=694996992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=276897792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=837181440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=76701696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840101888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=243916800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=145715200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674836480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=743108608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=661725184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=438169600, buflen=4096
fio: pid=160335, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934752256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99217408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=425132032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=806121472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=990658560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=346288128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=953012224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28512256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=499052544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478146560, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823971840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=776744960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=754278400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=19656704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=429019136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959897600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12582912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=697966592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12124160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813838336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506232832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027772416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=543240192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=220508160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=342061056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=364220416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=304779264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954494976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=445227008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=58896384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=610508800, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=981893120, buflen=4096
fio: pid=160348, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=8601600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=878317568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=252477440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=479391744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=311762944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=968626176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=132915200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=731619328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813502464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=238047232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=303321088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=870821888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412164096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488108032, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219828224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=409120768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=645390336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=716910592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=960286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401412096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=823185408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=435511296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1062674432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=922300416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=36704256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=166371328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1007411200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102277120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=385798144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764690432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=281698304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055588352, buflen=4096
fio: pid=160346, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941694976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=658329600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=696217600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=885084160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=448086016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=755429376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52625408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1004957696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=514899968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=583254016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663240704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=23285760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=765550592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912629760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=719757312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=533606400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=888299520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831922176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581382144, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=699023360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1036546048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37203968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=428335104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1067794432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=894111744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=179408896, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1041235968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=395902976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1019281408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=923275264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=131219456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=359477248, buflen=4096
fio: pid=160342, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=239673344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440659968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=485097472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=549363712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230699008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588984320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=921407488, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=4083712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=535695360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=778104832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=181014528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=579887104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976420864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=194121728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1031229440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=739954688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=357748736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=407511040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=61419520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=941998080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=712503296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=996265984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=494002176, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562319360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=600989696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=924049408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=12562432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=700940288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=345415680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171368448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=916762624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=29659136, buflen=4096
fio: pid=160340, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=977674240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=284614656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=840081408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615800832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=946143232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=839675904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=585740288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488005632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=976441344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=573325312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002610688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=660320256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=390197248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=315158528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=652472320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=578666496, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=108425216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=151556096, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=785080320, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=466202624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=653971456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=129118208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771239936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678895616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581734400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=824451072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=615186432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030008832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=324722688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=232640512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=775360512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=440455168, buflen=4096
fio: pid=160339, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=468844544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=816947200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=331497472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030017024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=289689600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=223203328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=801005568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=518500352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=688750592, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=674770944, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=907816960, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1002545152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=44814336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=502349824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=216772608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=322326528, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=959201280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=774053888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=942166016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149180416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=766337024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=159719424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=511873024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=492613632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=124293120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=967725056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=829657088, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030090752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=934739968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=275898368, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248991744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=373313536, buflen=4096
fio: pid=160343, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=249245696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=618217472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426070016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=714809344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770711552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=135536640, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=119123968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=611643392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596529152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=678940672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642818048, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=351408128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=625250304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970063872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=596107264, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=657723392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=168054784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=89559040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=149348352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411435008, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=879968256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1021427712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=288948224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=204644352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=219381760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814739456, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=11239424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=979767296, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127823872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=99774464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=260157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=109899776, buflen=4096
fio: pid=160334, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=628183040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213262336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=919969792, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970616832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=883093504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1224704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=534097920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=638631936, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=908935168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=413306880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=901947392, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=478990336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=831348736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=488243200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055608832, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1030279168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1061765120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853839872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=117583872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1066917888, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=160763904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=939462656, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=317804544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=954675200, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=695492608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=305479680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670142464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=855990272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=376033280, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=180854784, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=764383232, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=213557248, buflen=4096
fio: pid=160344, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=171302912, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426074112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=631111680, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1049264128, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=768012288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=506085376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=127107072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=943030272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=178184192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=188891136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=14569472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=491732992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=330113024, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=398356480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=35921920, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=266612736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=580145152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=316366848, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=762724352, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827146240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=547999744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=52604928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=744288256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=389206016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=780058624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=138600448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568922112, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=947412992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=863789056, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=861249536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=100061184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=852398080, buflen=4096
fio: pid=160337, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=418398208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=45109248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=866889728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=860798976, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=206725120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=248606720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1016901632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=734695424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=401780736, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=597393408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=365465600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=230748160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=632897536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=481193984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=670666752, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=588877824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=727089152, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=951537664, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=538853376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=309227520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=531943424, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=871272448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=495022080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=853266432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=253792256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=360841216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251990016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=970469376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=50356224, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=813268992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=242343936, buflen=4096
fio: pid=160336, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=693460992, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=430518272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=229711872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=563814400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=193257472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=79380480, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=333225984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=412213248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=33120256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=297811968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=733782016, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=771977216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1027985408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=68124672, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=781545472, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=998039552, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1013321728, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1022607360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=184078336, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=568000512, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=452157440, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=175161344, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=642146304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=682221568, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=571371520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=67141632, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=37384192, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=789626880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=604008448, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=536186880, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1025413120, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=72269824, buflen=4096
fio: pid=160345, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=190599168, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=663441408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=490049536, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=794877952, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=233885696, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1005789184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011388416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909500416, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1038233600, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=777646080, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=626528256, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=735145984, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=676507648, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982581248, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1060880384, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=388255744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=814243840, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=420757504, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=627343360, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=442470400, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=361467904, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=964497408, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=576286720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=581394432, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=28442624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=78434304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=784994304, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1011179520, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=102154240, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189636608, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=48988160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=521666560, buflen=4096
fio: pid=160347, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=455966720, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=982298624, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=189042688, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=827351040, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=582787072, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=80031744, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=105869312, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=555749376, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=155660288, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1028173824, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=651608064, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=94019584, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=562163712, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=1055100928, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=803667968, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=906334208, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=426459136, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=154656768, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=415268864, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=503599104, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=142069760, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=313790464, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=770019328, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=208777216, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=874934272, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=432349184, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=153452544, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=909135872, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=746696704, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=912879616, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=123228160, buflen=4096
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=411148288, buflen=4096
fio: pid=160341, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor
fio: io_u error on file /s3mount/.fio-diskmark: Bad file descriptor: write offset=251985920, buflen=4096
fio: pid=160467, err=9/file:io_u.c:1846, func=io_u error, error=Bad file descriptor

Seq-Read-1M-Q8-T1: (groupid=0, jobs=1): err= 0: pid=160241: Sun Aug 20 02:54:25 2023
  read: IOPS=247, BW=248MiB/s (260MB/s)(1024MiB/4135msec)
    slat (usec): min=426, max=328857, avg=3972.90, stdev=22596.07
    clat (usec): min=4, max=335599, avg=25631.80, stdev=53986.45
     lat (usec): min=453, max=517739, avg=29604.70, stdev=59102.60
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[   13], 80.00th=[   28], 90.00th=[   59], 95.00th=[   90],
     | 99.00th=[  330], 99.50th=[  334], 99.90th=[  334], 99.95th=[  338],
     | 99.99th=[  338]
   bw (  KiB/s): min= 1793, max=372736, per=96.84%, avg=245570.62, stdev=129791.86, samples=8
   iops        : min=    1, max=  364, avg=239.62, stdev=126.95, samples=8
  lat (usec)   : 10=0.10%, 500=0.10%
  lat (msec)   : 2=0.20%, 4=1.27%, 10=62.60%, 20=13.77%, 50=7.13%
  lat (msec)   : 100=10.64%, 250=2.15%, 500=2.05%
  cpu          : usr=0.60%, sys=1.33%, ctx=1029, majf=0, minf=2059
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Write-1M-Q8-T1: (groupid=1, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160247: Sun Aug 20 02:54:25 2023
  cpu          : usr=4.85%, sys=0.00%, ctx=18, majf=0, minf=16
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=66.7%, 8=33.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
Seq-Read-1M-Q1-T1: (groupid=2, jobs=1): err= 0: pid=160248: Sun Aug 20 02:54:25 2023
  read: IOPS=282, BW=282MiB/s (296MB/s)(1024MiB/3626msec)
    slat (usec): min=316, max=404103, avg=3474.72, stdev=19094.04
    clat (nsec): min=1948, max=420109, avg=5823.21, stdev=18705.45
     lat (usec): min=318, max=404111, avg=3480.54, stdev=19094.42
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    5], 90.00th=[    7], 95.00th=[    9],
     | 99.00th=[   73], 99.50th=[   93], 99.90th=[  302], 99.95th=[  420],
     | 99.99th=[  420]
   bw (  KiB/s): min=46369, max=376832, per=93.02%, avg=268988.43, stdev=141001.68, samples=7
   iops        : min=   45, max=  368, avg=262.43, stdev=137.69, samples=7
  lat (usec)   : 2=0.10%, 4=73.93%, 10=22.46%, 20=0.88%, 50=1.07%
  lat (usec)   : 100=1.07%, 250=0.29%, 500=0.20%
  cpu          : usr=0.47%, sys=1.30%, ctx=1030, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Seq-Write-1M-Q1-T1: (groupid=3, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160253: Sun Aug 20 02:54:25 2023
  cpu          : usr=2.08%, sys=0.00%, ctx=3, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Read-4K-Q32-T16: (groupid=4, jobs=16): err= 0: pid=160255: Sun Aug 20 02:54:25 2023
  read: IOPS=93, BW=373KiB/s (382kB/s)(21.9MiB/60052msec)
    slat (msec): min=51, max=482, avg=170.26, stdev=38.02
    clat (usec): min=7, max=6693.0k, avg=5060592.48, stdev=983003.30
     lat (msec): min=56, max=6868, avg=5230.85, stdev=985.62
    clat percentiles (msec):
     |  1.00th=[  493],  5.00th=[ 3138], 10.00th=[ 4597], 20.00th=[ 4866],
     | 30.00th=[ 5000], 40.00th=[ 5067], 50.00th=[ 5201], 60.00th=[ 5336],
     | 70.00th=[ 5470], 80.00th=[ 5604], 90.00th=[ 5805], 95.00th=[ 6141],
     | 99.00th=[ 6477], 99.50th=[ 6544], 99.90th=[ 6611], 99.95th=[ 6678],
     | 99.99th=[ 6678]
   bw (  KiB/s): min=  135, max=  520, per=98.60%, avg=368.58, stdev= 4.79, samples=1753
   iops        : min=   33, max=  130, avg=91.44, stdev= 1.22, samples=1753
  lat (usec)   : 10=0.14%, 20=0.14%
  lat (msec)   : 100=0.14%, 250=0.18%, 500=0.46%, 750=0.43%, 1000=0.37%
  lat (msec)   : 2000=1.62%, >=2000=96.50%
  cpu          : usr=0.01%, sys=0.02%, ctx=5720, majf=1, minf=674
  IO depths    : 1=0.3%, 2=0.6%, 4=1.1%, 8=2.3%, 16=4.6%, 32=91.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=5603,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Write-4K-Q32-T16: (groupid=5, jobs=16): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160333: Sun Aug 20 02:54:25 2023
  cpu          : usr=0.14%, sys=0.04%, ctx=1030, majf=0, minf=228
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=66.7%, 8=0.0%, 16=0.0%, 32=33.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
Rand-Read-4K-Q1-T1: (groupid=6, jobs=1): err= 0: pid=160350: Sun Aug 20 02:54:25 2023
  read: IOPS=12, BW=50.1KiB/s (51.3kB/s)(3008KiB/60027msec)
    slat (msec): min=34, max=132, avg=79.72, stdev=16.86
    clat (usec): min=6, max=275, avg=10.55, stdev=18.51
     lat (msec): min=34, max=132, avg=79.73, stdev=16.86
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[  118], 99.50th=[  174], 99.90th=[  277], 99.95th=[  277],
     | 99.99th=[  277]
   bw (  KiB/s): min=   39, max=   64, per=99.78%, avg=50.03, stdev= 5.85, samples=119
   iops        : min=    9, max=   16, avg=12.47, stdev= 1.48, samples=119
  lat (usec)   : 10=96.54%, 20=0.40%, 50=1.46%, 100=0.40%, 250=1.06%
  lat (usec)   : 500=0.13%
  cpu          : usr=0.03%, sys=0.03%, ctx=759, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=752,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Rand-Write-4K-Q1-T1: (groupid=7, jobs=1): err= 9 (file:io_u.c:1846, func=io_u error, error=Bad file descriptor): pid=160467: Sun Aug 20 02:54:25 2023
  cpu          : usr=0.00%, sys=0.00%, ctx=3, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=1024MiB (1074MB), run=4135-4135msec

Run status group 1 (all jobs):

Run status group 2 (all jobs):
   READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=1024MiB (1074MB), run=3626-3626msec

Run status group 3 (all jobs):

Run status group 4 (all jobs):
   READ: bw=373KiB/s (382kB/s), 373KiB/s-373KiB/s (382kB/s-382kB/s), io=21.9MiB (22.9MB), run=60052-60052msec

Run status group 5 (all jobs):

Run status group 6 (all jobs):
   READ: bw=50.1KiB/s (51.3kB/s), 50.1KiB/s-50.1KiB/s (51.3kB/s-51.3kB/s), io=3008KiB (3080kB), run=60027-60027msec

Run status group 7 (all jobs):
```
</details>


また、`fio`だとS3のWrite性能が分からなかったので、`dd`コマンドでWrite性能を確認してみました。

以下コマンドをそれぞれのマウントポイントごとに10回実施した平均が表の結果となります。

```bash:ddコマンドによるWrite性能確認
dd if=/dev/zero of=/[EBS、EFS、S3マウントポイント]/.dd-benchmark bs=1M count=1000
```

|EBS dd結果|EFS dd結果|S3 dd結果|
|:--|:--|:--|
|415MB/s|316MB/s|303MB/s|

# ログアウトする際にコンソールが固まる
2023年8月現在、私がテストしたEC2環境では`Systems Manager`の`Session Manager`でEC2に接続してS3をマウント→`exit`でEC2から抜けようとするとコンソールが固まってしまいました。

他のウィンドウから再度EC2にログインして`umount`でマウント解除すると固まっていたコンソールも閉じられたので、`Mountpoint for Amazon S3`が悪さをしているようです。

マウントを実施したコンソール以外のウィンドウは特に`exit`で抜けても固まることはありませんでした。

また、`Session Manager`ではなく、SSHで接続してS3をマウント→`exit`であればコンソールが固まることはなかったので、`Session Manager`＋`Mountpoint for Amazon S3`の組み合わせが問題を引き起こすようです。

もし`Session Manager`で操作を行う場合は上記の問題が発生するかと思われるので気をつけましょう。

# おわりに
`Mountpoint for Amazon S3`はそもそもS3なので色々と制約はありますが、シーケンシャルReadに関してはEFSより良い性能が出たのは予想外でした。

また、書き込みについても300MB/s程度出たので、`Mountpoint for Amazon S3`の特性を理解して使うのであれば、`EFS`から置き換えることもできるのではないかと思います。

通常のファイルシステムとは勝手が違うので、導入は慎重に行う必要がありますが、うまく要件が合致するようであればS3の利点を活かしつつ、高いパフォーマンスも出すことができる保存領域としてシステムに導入できそうだと感じました。

今回紹介した内容以外の詳細は`Mountpoint for Amazon S3`のGitHubの`doc`に色々と書かれているので、詳しい情報を知りたい場合は、公式のGitHubを確認してみてください。

- [GitHub - awslabs/mountpoint-s3/doc](https://github.com/awslabs/mountpoint-s3/tree/main/doc)