Repository: skypilot-org/skypilot
Pull request: Sky storage microbenchmark YAML
Closes #965. Refer to `examples/benchmarks/results.md` for details.

Example output (`sky launch storage_rawperf.yaml`):
```
(storage-demo pid=29477) ===== Benchmark Results =====
(storage-demo pid=29477) All results are reported as (bandwidth, IOPS)
(storage-demo pid=29477)
(storage-demo pid=29477) ##### Sequential Read Results #####
(storage-demo pid=29477) EBS:   135.35 MB/s     2065.23 IOPS
(storage-demo pid=29477) S3:    535.00 MB/s     8163.43 IOPS
(storage-demo pid=29477)
(storage-demo pid=29477) ##### Sequential Write Results #####
(storage-demo pid=29477) EBS:   135.26 MB/s     2063.93 IOPS
(storage-demo pid=29477) S3:    1797.06 MB/s    27420.92 IOPS
(storage-demo pid=29477)
(storage-demo pid=29477) ##### Sky Storage COPY mode setup bandwidth #####
(storage-demo pid=29477) aws s3 cp reported bandwidth: (342.0 MiB/s)
(storage-demo pid=29477) Actual aws s3 cp -> EBS bandwidth (MB/s): 92
SKY INFO: Job finished (status: SUCCEEDED).
```


Problem statement:
YAMLs for profiling storage performance
We should add YAMLs which use fio/other benchmarking tools to measure Sky Storage (COPY and MOUNT) mode performance against EBS. And we should preferably do this for different workload types (sequential rw, random rw, real application performance (e.g., training))

Hints from issue discussion:

Commit message: update results

Patch to review:
diff --git a/examples/benchmarks/results.md b/examples/benchmarks/results.md
new file mode 100644
index 00000000000..2fb234916ef
--- /dev/null
+++ b/examples/benchmarks/results.md
@@ -0,0 +1,48 @@
+# Sky Benchmarks
+
+This directory is a collection of YAMLs used to benchmark Sky's performance. 
+
+## Sky Storage Benchmarks
+We benchmark the performance of Sky Storage using fio, a popular storage
+benchmarking utility. For fio, we use recommended parameters from [Azure's 
+disk benchmarking guide](https://docs.microsoft.com/en-us/azure/virtual-machines/disks-benchmarks).
+Please refer to `storage_rawperf.yaml` for more details.
+All benchmarks on run on AWS m5.8xlarge with default EBS volume attached with 
+4 parallel processes reading/writing to files.
+
+### Disk Bandwidth (MB/s)
+
+|                  | EBS | Sky Storage<br/>(S3, MOUNT mode) |
+|------------------|-----|----------------------------------|
+| Sequential Read  | 130 | 642                              |
+| Sequential Write | 129 | 1828                             |
+
+
+### Disk IOPS
+
+|                  | EBS  | Sky Storage<br/>(S3, MOUNT mode) |
+|------------------|------|----------------------------------|
+| Sequential Read  | 2051 | 8462                             |
+| Sequential Write | 2055 | 27899                            |
+
+### S3 -> EBS copy bandwidth (MB/s)
+
+We also run `aws s3 cp s3://<bucket> <local_path>` to measure S3->EBS bandwidth.
+We copy a 1 GB file from S3 to EBS using `aws s3 cp` command.
+
+|           | `aws s3 cp` bandwidth |
+|-----------|-----------------------|
+| S3 -> EBS | 92 MB/s               |
+
+### Discussion
+* These benchmarks use the default EBS gp3 volume with the standard 125MB/s 
+  throughput and 16000 IOPS. It is possible to achieve higher performance by 
+  paying more.
+* Sky Storage offers higher **read** performance than EBS because it can directly
+  stream files from S3 to memory.
+* Sky Storage **write** performance is significantly higher than EBS because it
+  writes to memory and then asynchronously uploads to S3. Sky Storage offers 
+  only eventual consistency, so a write operation to sky storage may not reflect 
+  immediately on the S3 storage.
+* These benchmarks are run on single large files. The performance of sky storage
+  degrades when using many small files.
diff --git a/examples/benchmarks/storage_rawperf.yaml b/examples/benchmarks/storage_rawperf.yaml
new file mode 100644
index 00000000000..82e4de3218b
--- /dev/null
+++ b/examples/benchmarks/storage_rawperf.yaml
@@ -0,0 +1,67 @@
+# Sky Storage benchmarks using fio.
+#
+# Uses FIO to run benchmarks on sky storage. We use Azure's recommended
+# parameters for fio to measure storage performance.
+# See https://docs.microsoft.com/en-us/azure/virtual-machines/disks-benchmarks
+#
+# Also measures S3->EBS bandwidth using aws s3 cp, which is used in COPY mode.
+#
+# Note that random writes are not supported by sky storage, and thus
+# not benchmarked.
+#
+# Usage:
+#   sky launch -y -c bench storage_rawperf.yaml
+#   sky down bench
+#   sky storage delete <storage_name>
+
+name: storage-demo
+
+resources:
+  cloud: aws
+  instance_type: m5.8xlarge
+
+file_mounts:
+  /skystorage-mount:
+    name: sky-romil-benchmark # Make sure this name is unique.
+    mode: MOUNT
+
+setup: |
+  sudo apt update
+  sudo apt install -y fio
+
+run: |
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
+  echo "Running EBS read benchmark..."
+  fio --name=64kseqreads --rw=read --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128 --size=1G --group_reporting --directory=/tmp/ --output-format=json > /skystorage-mount/perf_read_ebs.json
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
+  echo "Running S3 read benchmark..."
+  fio --name=64kseqreads --rw=read --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128 --size=1G --group_reporting --directory=/skystorage-mount/ --output-format=json > /skystorage-mount/perf_read_s3.json
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
+  echo "Running EBS write benchmark..."
+  fio --name=64kseqwrites --rw=write --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128 --size=1G --group_reporting --directory=/tmp/ --output-format=json > /skystorage-mount/perf_write_ebs.json
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
+  echo "Running S3 write benchmark..."
+  fio --name=64kseqwrites --rw=write --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128 --size=1G --group_reporting --directory=/skystorage-mount/ --output-format=json > /skystorage-mount/perf_write_s3.json
+  
+  # Change the s3 path here to the storage bucket name used above
+  echo "Running S3 read benchmark with aws s3 cp (COPY mode setup bandwidth)..."
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
+  /usr/bin/time -o /skystorage-mount/perf_copymode_time.txt -f "%e" aws s3 cp s3://sky-romil-benchmark/64kseqreads.0.0 ~/ > /skystorage-mount/perf_copymode_log.txt
+  sed -i -e 's/\r/\n/g' /skystorage-mount/perf_copymode_log.txt
+  
+  echo -e '\n===== Benchmark Results ====='
+  echo 'All results are reported as (bandwidth, IOPS)'
+  echo -e '\n##### Sequential Read Results #####'
+  cat /skystorage-mount/perf_read_ebs.json | python3 -c "import sys, json; data = json.load(sys.stdin)['jobs'][0]; print('EBS:\t{:.2f} MB/s\t{:.2f} IOPS'.format(data['read']['bw_bytes']/(1000*1000), data['read']['iops']))"
+  cat /skystorage-mount/perf_read_s3.json | python3 -c "import sys, json; data = json.load(sys.stdin)['jobs'][0]; print('S3:\t{:.2f} MB/s\t{:.2f} IOPS'.format(data['read']['bw_bytes']/(1000*1000), data['read']['iops']))"
+  echo -e '\n##### Sequential Write Results #####'
+  cat /skystorage-mount/perf_write_ebs.json | python3 -c "import sys, json; data = json.load(sys.stdin)['jobs'][0]; print('EBS:\t{:.2f} MB/s\t{:.2f} IOPS'.format(data['write']['bw_bytes']/(1000*1000), data['write']['iops']))"
+  cat /skystorage-mount/perf_write_s3.json | python3 -c "import sys, json; data = json.load(sys.stdin)['jobs'][0]; print('S3:\t{:.2f} MB/s\t{:.2f} IOPS'.format(data['write']['bw_bytes']/(1000*1000), data['write']['iops']))"
+  echo -e '\n##### Sky Storage COPY mode setup bandwidth #####'
+  echo -n 'aws s3 cp reported bandwidth: ' && ( tail -2 /skystorage-mount/perf_copymode_log.txt | head -1 | grep -o '([^)]* MiB/s)' )
+  echo -n 'Actual aws s3 cp -> EBS bandwidth (MB/s): ' && ( bc <<< $(stat -c %s ~/64kseqreads.0.0)/$(cat /skystorage-mount/perf_copymode_time.txt)/1000000)


Code review comments:
diff_hunk: @@ -0,0 +1,48 @@
+# Sky Benchmarks
path: examples/benchmarks/results.md
text: @user1:
Can we change
- directory name to `examples/perf/`?
- this line to "Performance benchmarks for Sky"?

This is to avoid confusion with the Sky Benchmark PR which is adding `examples/benchmark`.

@author:
fixed!

diff_hunk: @@ -0,0 +1,67 @@
+# Sky Storage benchmarks using fio.
+#
+# Uses FIO to run benchmarks on sky storage. We use Azure's recommended
+# parameters for fio to measure storage performance.
+# See https://docs.microsoft.com/en-us/azure/virtual-machines/disks-benchmarks
+#
+# Also measures S3->EBS bandwidth using aws s3 cp, which is used in COPY mode.
+#
+# Note that random writes are not supported by sky storage, and thus
+# not benchmarked.
+#
+# Usage:
+#   sky launch -y -c bench storage_rawperf.yaml
+#   sky down bench
+#   sky storage delete <storage_name>
+
+name: storage-demo
+
+resources:
+  cloud: aws
+  instance_type: m5.8xlarge
+
+file_mounts:
+  /skystorage-mount:
+    name: sky-romil-benchmark # Make sure this name is unique.
+    mode: MOUNT
+
+setup: |
+  sudo apt update
+  sudo apt install -y fio
+
+run: |
+  echo "Purging I/O caches..."
+  echo 3 | sudo tee /proc/sys/vm/drop_caches
path: examples/benchmarks/storage_rawperf.yaml
text: @user1:
nit: Use
```
sync && echo 3 > /proc/sys/vm/drop_caches
```
currently there are a bunch of `3`'s printed. Better: make the echo & this line into a bash func.

@author:
fixed!