# Analyzing the Performance of the S3 Object Storage API for HPC Workloads
## 4 S3 Embedded  
Because of the scalability limitation of the introduced local-gw mode, we create an I/O library called S3 Embedded based on libs3, where parts of the S3 stack are replaced or removed to optimize the performance. By easily linking the S3 Embedded library at compile time or at runtime to a libS3 compatible client application, it is possible to use the capabilities of this library. Assuming the availability of a globally accessible shared file system, S3 Embedded provides the following libraries:  
* libS3e.so: This is an embedded library wrapper that converts libs3 calls to POSIX calls inside the application address space. * libS3r.so: This library converts the libs3 calls via a binary conversion to TCP calls to a local libS3-gw application that then executes these POSIX calls, bypassing the HTTP protocol. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Benchmark/System** & **Unit** & **Wasabi** & **IBM** & **Google** & **MinIO-Local-Gw** \\ \hline
**Score Bandwidth** & MiB/s & 0.007 & 1.642 & 0.46 & 12.62 \\ ior-easy-write & MiB/s & 2.35 & 35.00 & 13.35 & 46.39 \\ mdtest-easy-write & IOPS & 13.04 & 81.72 & 21.79 & 27.96 \\ ior-rnd-write & MiB/s & 0.01 & 0.23 & 0.07 & 1.231 \\  mdworkbench-bench & IOPS & 5.75 & 47.23 & 12.83 & 15.25 \\  ior-easy-read & MiB/s & 1.20 & 45.37 & 7.81 & 73.86 \\  mdtest-easy-stat & IOPS & 20.92 & 145.09 & 51.10 & 260.97 \\  ior-hard-read & MiB/s & 0.05 & 5.59 & 1.38 & 6.01 \\  mdtest-hard-stat & IOPS & 20.74 & 149.64 & 49.48 & 297.62 \\  mdtest-easy-delete & IOPS & 10.35 & 35.02 & 9.37 & 81.06 \\  mdtest-hard-read & IOPS & 8.54 & 70.06 & 18.90 & 130.36 \\  mdtest-hard-delete & IOPS & 10.28 & 35.25 & 9.48 & 94.32 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: IO500 results comparing S3 Cloud providers. In Figure 9, we display the results of the **IOR** benchmark while using the libraries mentioned above, in comparison with direct Lustre access and MinIO operating in the local-gw mode already described in Section 3. Note that some values are missing in the MinIO-local-gw results, despite the benchmark being repeated several times. This is because this setup does not scale well with the number of clients, as noted in Section 3. Using S3 Embedded helped us to pinpoint a performance problem in the IOR S3 plugin: we noticed that the delete process in IO500 is time-consuming since when trying to delete a bucket, our developed IOR S3 plugin tries to list the content of the entire bucket--calling S3_list_bucket()--for each file to be deleted to clean the fragments; however, since, in case of S3 Embeded, all files are actually placed in a single directory, this ought to be very time-consuming. One workaround is to use the option bucket-per-file that effectively creates a directory per file. However, since this workaround does not cover all test workloads in the IO500, we proceed and introduce an environment variable called \"S3 LIB_DELETE_HEURISTICS\", specific to the IOR S3-plugin. It defines at which file size of the initial fragment the list_bucket operation is to be executed; otherwise, a simple S3_delete_object is performed. While this optimization is not suitable for a production environment, it allows us to determine best-case performance for using S3 with the IO500 benchmark. The results delivered by S3 Embedded are very close to the ones obtained for the Lustre direct access--mainly for files larger than 32 MB--far superior to the ones supplied by MinIO-local-gw, they are also free from timeout errors. Benchmarking with **IO500** reflects the performance improvement delivered by S3 Embedded/S3 Remote, as shown in Table 3.