Investigating the Overhead of the REST Protocol to Reveal the Potential for  Cloud Services in HPC
Channel: The Virtual Institute for I/O
Date: 26/06/2020

Title: Investigating the Overhead of the REST Protocol to Reveal the Potential for Using Cloud Services for HPC Storage 

Presented by Frank Gadban (University of Hamburg)

In this paper, we investigate the overhead of the REST protocol via HTTP compared to the HPC-native communication protocol MPI when storing and retrieving objects. Albeit we compare the MPI for a communication use case, we can still evaluate the impact of data communication and, therewith, the efficiency of data transfer for data access patterns. We accomplish this by modeling the impact of data transfer using measurable performance metrics. Hence, our contribution is the creation of a performance model based on hardware counters that provide an analytical representation of data transfer over current and future protocols. We validate this model by comparing the results obtained for REST and MPI on two different cluster systems, one equipped with Infiniband and one with Gigabit Ethernet. The evaluation shows that REST can be a viable, performant and resource-efficient solution, in particular for accessing large files.