# Performance of Communication- and Computation-Intensive SaaS on the OpenStack Cloud
## 4 Results and Discussion
### Communication  
Interprocess communication highly influences the parallel performance of DEM software. Generally, computations are performed significantly faster than MPI communications between nodes, especially over high-latency and low-bandwidth Ethernet networks. The DEM SaaS is communication-intensive software because of the necessary exchange of ghost particle data at each time step. Thus, intensive interprocess communication can drastically decrease the parallel performance if the communication-to-computation ratio is not low enough. Figure 4 shows DEM SaaS data transfer between the cores and nodes of the cloud infrastructure. The data transfer between processes gradually increased with the number of processes. In the case of GL1, data transfer between 16 processes was 3.3 times more than that between 4 processes. We observed the nearly linear dependency of data transfer on the number of particles. However, a sudden increase in data transfer between nodes was obtained in the case of benchmarks solved by three nodes (12 cores), which was not observed in communication between processes (cores). Figure 4: Communication: data transferred between (**a**) cores and (**b**) nodes. Figure 3: Load imbalance expressed by the maximum and mean computing times. Figure 5a shows the MPI communication time measured solving benchmarks with ghost layers of different thicknesses. The GL1, GL2, and GL3 solid curves represent usual benchmarks with ghost layers of thickness = 2\(R\), 4\(R\), and 6\(R\), respectively. The GL1, GL2, and GL3 remapped curves represent communication times obtained by using the improved mapping of subdomains to cloud resources based on the multicore architecture. It is worth noting that a twofold increase in data transfer between nodes (Figure 4b) caused up to a threefold increase in the communication time for a larger number of processes. For benchmarks solved by three nodes (12 cores), a sudden increase in communication time was observed, which is relevant to data transfer between nodes presented in Figure 4b. In this particular case, the communication time made up 9.1% of the computing time, which significantly reduced the parallel performance. In the case of four nodes (16 cores), the communication time made up only 3.9% of the computing time. Figure 5b shows the default particle distribution among three nodes, which illustrates unsuccessful subdomain mapping to cloud resources based on the multicore architecture. The RCB method of the Zoltan library divides particles into nearly equal subsets but does not optimize internode communication or perform relevant mapping of particle subsets to multicore nodes. As a result, four spatially scattered subdomains were mapped to one node, which had a large number of ghost particles requiring data exchange with other nodes. Spatially connected subdomains were correctly remapped to nodes, reducing the MPI data transfer. In Figure 4a, dotted lines show the significantly reduced communication time due to improved mapping. However, the communication time measured on 12 cores (three nodes) was still larger than that obtained on 16 cores (four nodes), which could be easily observed in the cases of benchmarks with thicker ghost layers. It is natural that the best performance of the method based on recursive bisection can be observed by dividing particles into 2\({}^{p}\) subsets. Figure 6 shows the contribution of computation, communication, and waiting to the total benchmark time in the case of the GL1 benchmark solved by 12 processes on three nodes. Performing code profiling, two MPI barriers were placed before and after communication routines to measure wait times caused by computational load imbalance and communication imbalance, respectively. The Calc, Comm, Wait1, and Wait2 columns represent the computing time, communication time, wait time due to computational load imbalance, and wait time due to communication imbalance, respectively. Figure 5: Communication time (**a**) and default particle distribution among three nodes (**b**).