# Adaptation of Parallel SaaS to Heterogeneous Co-Located Cloud Resources
## 3 Cloud Infrastructure and Developed SaaS
### Parallel DEM SaaS
It is worth noting that these weights also consider variations in application load and system load on virtualized hardware. Sometimes load oscillations cause high variations in weight values, which can lead to drastic changes in topology and the size of subsets of particles after repartitioning. Thus, the previous weights help reduce significant oscillations that often occur in dynamic simulations on co-located resources. Usually, several applications of the repartitioning procedure are required for the computational load and weights to stabilize. In the case of over-subscribed nodes, when several containers are co-located on the same node and compete for the same resources, the algorithm evaluates the runtime measured computational load and reduces the values of relevant weights. Finally, RCB-based repartitioning adapts subsets of particles to heterogeneous resources according to the runtime measured load of application-specific computations. In line 8, data of particles are transferred from the OpenCL device memory to the host memory for following repartitioning. In line 9, the RCB method from the Zoltan library is used to perform parallel repartitioning for distributed-memory architectures. In line 10, some particles migrate from the old partition to a new one, which requires internode communication handled by MPI. The number of redistributed particles depends on changes in load configuration. However, incremental partitions produced by RCB limit particle transfer between nodes. In line 11, the subroutine Registration_Ghost_Particles() defines ghost layers and registers ghost particles. In line 12, MPI processes exchange information of ghost particles, which is necessary for internode communications performed after repartitioning. At the end of the repartitioning procedure, data of internal particles are transferred from the host memory to the OpenCL device memory for the main shared-memory computations. In lines 15-17, the main communications are made before the main DEM computations. Ghost particle data are copied from the OpenCL device memory to the host memory and exchanged between neighboring partitions by MPI. The internode exchange of positions and velocities of particles is a popular approach in parallel DEM software [35] because it allows nodes to independently perform a contact search and computation of forces. It is worth noting that data of ghost particles, transferred by MPI from other nodes at each time step, also should be copied to the OpenCL device memory, which makes internode data exchange very expensive. The main computational procedures of the DEM are implemented by using OpenCL kernels to run the same software on all shared-memory architectures, including CPUs and GPUs of various vendors. Main kernels are performed on the thread-per-particle basis, which takes advantage of the massive parallel computation capabilities of modern GPUs and can be considered to be the most suitable parallelism in the case of DEM computations. In lines 18-23 of the pseudocode, the main computational procedures of the DEM performed by the OpenCL device are outlined. The contact search, the contact history, the computation of contact forces and moments, the boundary conditions, the external gravitational force, and the time integrator are implemented as separate OpenCL kernels to avoid overfill of the private memory. In lines 24-27, at the end of the time step, the particle data can be copied from the OpenCL device memory to the host memory and stored on the hard disk drive in HDF5 format. It is recommended to transfer the data to the host memory as seldom as possible because it is a time-consuming process.