# Overset Grid Assembler and Flow Solver with Adaptive Spatial Load Balancing
## 8 Results  
Test cases are run on TRUBA's [21] clusters. In order to reduce queue time, computations are run on different set of nodes with different specifications, as shown in Table 2. In this work, hyperthreading is not used and virtual processors are mapped to cores; therefore, processor is an alias for core for the rest of the paper. Since a node can have a maximum of 20 cores, for a job requesting more than 20 processors, cores cannot fit into a single node but they will belong to different nodes. Therefore, data transfer takes place between cores having both shared memory and distributed memories. Open MPI's byte transfer layer (BTL) framework uses the best component such as shared memory, TCP, Infiniband, and so on for data transfer between cores. Again, in order to reduce queue time, the number of nodes is left to be determined by the resource manager. The results in this section are obtained by using maximum allowable number of processors per user (128) usually distributed over 7-10 nodes. The most intensive memory usage is observed during data exchange between processors. Exchange of spatial partitions during load (re)balance and exchange of mesh cells after mesh motion are two major tasks involving intensive MPI communication. Although it is verified by [PERSON] [22] that no memory leak occurs, memory usage increases after the mentioned tasks. It is expected that the reason for increasing memory usage is memory fragmentation caused by frequent allocation/deallocation of buffers leaving fragmented chunks of memory space. An obvious solution is using fixed buffers for data exchange. However, after load rebalance partitioning layout changes, therefore, the buffers used for data exchange have to be deallocated. Once memory usage hits the memory limit (8 GB per core), the simulation is halted by resource manager. In order not to lose progress, crucial data structures of assembler and solver are serialized with Boost's serialization library (which is also used by Boost MPI for data transfer), and written to binary archives for every quarter (90\({}^{\circ}\)) rotation. Processors read deserialized data from respective archives and restart the simulation with reset memory space. \begin{table}
\begin{tabular}{c c c c} \hline
**Group** & **Processor Model** & **Core/Node** & **RAM (GB)** \\ \hline Barbun & Intel Xeon Scalable 6148 & 20 & 384 \\ Sardalya & Intel Xeon E5-2690 V4 & 14 & 256 \\ Levrek & Intel Xeon i5-2690 & 16 & 256 \\ \hline \end{tabular}
\end{table} 
Table 2: Hardware specifications. Figure 9: Slice of fuselage mesh before and after overlap minimization: (**a**) before; (**b**) after. The most time consuming tasks per time step are shown in Figure 10. These tasks remain as the most time consuming in all time steps. However, note that Rebalance may not be needed for all time steps. The execution time of Solve is based on the maximum number of pseudo time steps (100) to solve Euler equations. Since the assembler constitutes less time compared to the solver, it is unnecessary to optimize load balance for the assembler. As discussed in Section 2, it is possible to have separate partitions for the assembler and the flow solver in order to optimize load balance differently. However, having two separate partitions requires mapping of data across different partitioning layouts. The case with two different partitions is also tested, and it is found that mapping of data increases the total run time of simulation. The frequency of load rebalance depends on relative time-cost of load rebalancing and the solver. In the case of 100 pseudo time steps for the solver, rebalancing is more costly than the solver and, therefore, Rebalance is called when the ratio of the maximum to the minimum number of cells across processors exceeds 1000. The rebalance threshold is not the optimum and can be improved by an algorithm involving relative costs of tasks. Figure 11 shows the run time per time step with and without rebalance. Areas under each curve represent total run time over quarter rotation of rotor.