# Analyzing the Performance of the S3 Object Storage API for HPC Workloads
## 4 S3 Embedded
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**System** & **Unit** & **MinIO-Local-Gw** & **Lustre** & **S3 Embedded** & **S3 Remote** \\ \hline ior-easy-write & GiB/s & 0.14 & 5.47 & 0.61 & 0.69 \\ mdtest-easy-write & kIOPS & 0.09 & 7.97 & 2.42 & 3.13 \\ ior-easy-read & GiB/s & 0.32 & 2.78 & 0.48 & 0.42 \\ mdtest-easy-stat & kIOPS & 0.85 & 13.82 & 8.02 & 6.94 \\ ior-hard-read & GiB/s & 0.019 & 0.139 & 0.046 & 0.042 \\ mdtest-hard-stat & kIOPS & 0.86 & 5.10 & 7.25 & 6.65 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: IO500 results for S3 Embedded and S3 Remote compared to MinIO-local-gw and lustre using 2N-5 PPN. Figure 9: Read throughput of S3 Embedded vs. Lustre vs. MinIO for 5N-20 PPN. We notice that Lustre's performance with POSIX is often more than 10x faster than MinIO-local-gw, and that the error rate increases along with the number of Nodes/PPN. In contrast, the S3 API wrappers deliver much better performance, which is closer to Lustre native performance, and are more resilient to the number of clients, as shown in Table 4.  
Even with 10 or 50 Nodes, as seen in Tables 5 and 6, the S3 embedded library yields a performance closer to Lustre, but a performance gap remains. Some MDTest results show a better performance in the case of S3 Embedded than Lustre, although for both Lustre and S3 Embedded, the stat() call is used. This might be due to the way S3 Embedded implements S3_test_bucket(), where the size and rights for the directory and not of the actual file are captured, which seems to be faster. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Benchmark/System** & **Unit** & **Lustre** & **S3 Embedded** & **S3 Remote** \\ \hline ior-easy-write & GiB/s & 16.262279 & 5.363676 & 1.516302 \\ mdtest-easy-write & kIOPS & 18.104786 & 15.952838 & 1.111315 \\ ior-hard-write & GiB/s & 0.030967 & 0.375326 & 0.032984 \\ mdtest-hard-write & kIOPS & 13.705966 & 4.337179 & 0.361589 \\ ior-easy-read & GiB/s & 43.390676 & 2.817122 & 1.309052 \\ mdtest-easy-stat & kIOPS & 46.662299 & 45.878814 & 0.620202 \\ ior-hard-read & GiB/s & 0.218977 & 0.128423 & 0.025549 \\ mdtest-hard-stat & kIOPS & 43.834921 & 44.443974 & 0.62586 \\ mdtest-easy-delete & kIOPS & 9.322632 & 9.262801 & 0.585572 \\ mdtest-hard-read & kIOPS & 4.079555 & 6.673985 & 1.246393 \\ mdtest-hard-delete & kIOPS & 8.457431 & 6.377992 & 1.246105 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: IO500 results for Lustre vs. S3 Embedded using 50N-1 PPN. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Benchmark/System** & **Unit** & **MinIO-Local-Gw** & **Lustre** & **S3 Embedded** & **S3 Remote** \\ \hline ior-easy-write & GiB/s & 0.75 & 23.49 & 3.40 & 1.99 \\ mdtest-easy-write & kIOPS & 0.39 & 17.11 & 7.52 & 1.19 \\ ior-hard-write & GiB/s & 0.01 & 0.04 & 0.30 & 0.05 \\ mdtest-hard-write & kIOPS & 0.10 & 7.25 & 3.41 & 0.55 \\ ior-easy-read & GiB/s & 2.46 & 15.87 & 2.40 & 1.36 \\ mdtest-easy-stat & kIOPS & 5.09 & 42.59 & 28.53 & 0.62 \\ ior-hard-read & GiB/s & 0.11 & 0.38 & 0.11 & 0.02 \\ mdtest-hard-stat & kIOPS & 4.37 & 31.49 & 26.66 & 0.60 \\ mdtest-easy-delete & kIOPS & - & 9.15 & 5.98 & 0.41 \\ mdtest-hard-read & kIOPS & - & 6.34 & 3.82 & 0.29 \\ mdtest-hard-delete & kIOPS & - & 6.27 & 5.04 & 0.41 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: IO500 results for S3 Embedded and S3 Remote compared to MinIO-local-gw and lustre using 5N-20 PPN.