# Performance Analysis of Storage Systems in Edge Computing Infrastructures
## 4 Experimental Evaluation
### Experimental Results  
The evaluation metrics utilized are divided into two categories: (i) resource consumption (total, used)--CPU, RAM, HDD, and network, and (ii) performance--throughput, data request response time, and network time.  
The performance evaluation was executed through Locust [[https://locust.io/](https://locust.io/)]([https://locust.io/](https://locust.io/)), accessed on 5 July 2022, an open-source load-testing framework that enables the definition of user behavior and supports running load tests distributed over multiple machines. In addition, it is able to simulate millions of concurrent user requests. For the purposes of the experiments, 20 users were set performing distributed query requests from four different machines. The results of the experiments are shown in Figure 1, where the average response time in milliseconds of each storage system is recorded. Figure 1a,b visualize the average response time of a single request of read and write operations, respectively. On the other hand, Figure 1c,d illustrate the average response time for all users' requests. The standard deviation of the response time is also illustrated in each figure in a stacked barplot manner on top of each average response time. Overall, as indicated in the above figures, MinIO presents the best performance in both the read and write operations. BigchainDB follows MinIO in all cases except for the write operation of a single request (Figure 1b), where the IPFS outperforms BigchainDB. Due to the object store nature of MinIO, it can be observed that the write operation of all requests is more time-consuming compared to BigchainDB. Moreover, the IPFS exhibits a high standard deviation, indicating a more unstable performance, thus yielding the worst performance out of the three storage systems.  
To further evaluate the storage systems, we also measured the RAM usage, the disk latency, and the disk IO time for a single user's request and for all users' requests, similar to the previous figures. The CPU was also recorded but not plotted because its usage wasnegligible. This proves that the storage is lightweight enough to be deployed on most edge devices, including the Raspberry Pis used for the evaluation. Figures 2 and 3 illustrate the statistics for the read and write operations, respectively. Figures 2(a) and 3(a) indicate the percentage of the RAM usage where, as depicted, MinIO consumes the least amount of RAM in each case. In addition, BigchainDB follows MinIO, only in the case of a single request, with the IPFS is ahead of BigchainDB in all users' requests.  
In the rest of the figures where the disk latency (Figures 2(b) and 3(b)) and the disk IO time (Figures 2(c) and 3(c)) are presented, MinIO achieves the best performance followed by BigchainDB, while the IPFS yields the worst performance results. The disk metrics are increased by a larger degree, proving intense I/O activity.  
Finally, Figure 4 demonstrates the transaction rate (TR) achieved by each storage system. The transaction rate can be defined as:  
\[\text{TR}=\frac{\sum\text{transactions}}{\text{total\_time}} \tag{1}\]  
As the results suggest, MinIO achieves the highest TR followed by BigchainDB, while the IPFS exhibits the worst results. For instance, the TR obtained by MinIO is 3.3 and 1.3 times larger compared to the IPFS and BigchainDB, respectively.  
Figure 3: Statistics for the write operation of each database.  
Figure 2: Statistics for the read operation of each database.