## 4 Theory and Calculation
### Temporary Array Method
In the translator scripts these are: reproducible--needed for all methods, repr_temp_array--for using temporary arrays, repr_coloring--for using reproducible coloring method and trivial_coloring which will produce the trivial coloring version. To enable the greedy coloring technique, the -op_repro_greedy_coloring command line flag must be used with the application. ## 5 Performance Results

We measured our techniques with four test applications, introduced in Section 3.8. All results are the average of 10 measurements. Table 1 summarizes the details of the different machine setups we used for our measurements. All of our methods provide full reproducibility at the expense of additional computations, suboptimal scheduling, or redundant memory usage. The overall cost of these techniques is visualized in Figures 7 and 8 and in Table 2. We compare each run with its original, non-reproducible version. On CPU systems, slowdowns are between 1 and 3.21 \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Name** & **CPU** & **GPU** & **\#Processes per Node** & **Compiler** & **OS** \\ \hline \multirow{3}{*}{Cirrus-CPU} & Intel Xeon E5-2695 & \multirow{3}{*}{n.a.} & 18 cores, 2 threads per & \multirow{3}{*}{icc (ICC) 19.0.0.117} & Red Hat Enterprise \\  & (Broadwell) @ & & & core per node & Linux 8.1 (Ootpa) \\ \cline{1-1}  & 2.5 GHz & & & & \\ \cline{1-1}  & 2.5 GHz, Intel Xeon & NVIDIA Tesla & 20 cores, 2 threads per & \multirow{3}{*}{nvc++ 21.9-0} & Red Hat Enterprise \\ \cline{1-1}  & Gold 6248 & V100-SXM2-16 GN & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Details of the different machine setups. Figure 6: An example of a second ghost layer to determine the edge\(\rightarrow\)edge neighbors on the partition borders. The numbers on the edges indicate their unique ID. times. The difference between the greedy and distributed coloring methods comes down to data reuse and cache line utilization, because of the different number of colors used. The main reason for that is that the data for neighboring elements are located close in memory, but when using coloring, adjacent elements will have different colors, leading to poor utilization. A few examples of the number of colors used are shown in Table 3. While the greedy scheme leads to near-optimal color counts, the parallel scheme yields much higher color counts particularly in 3D. The performance of the trivial coloring scheme is close to the reference, since it uses a similar order of execution to the nonreproducible version, with the only differences around the borders of MPI partitions. Since with the trivial scheme we still require sequential execution within a process, we cannot use additional parallelization techniques, such as CUDA or OpenMP. In contrast, the slowdown on GPUs is more significant, because they are even more sensitive to data access patterns and cache locality than CPUs. In particular, with the usage of the temporary arrays, we have to iterate through the increment data twice, once when populating it and once when gathering the results, each time with a different access pattern. If we optimize for one stage, then the other will suffer from the non-coalesced data accesses. This is even true for the coloring methods. If we reorganize the data in a set according to one map, then later, using another map to the same set, we again obtain inefficient access patterns. Figure 8: Slowdown of Hydra measured on an 8M mesh, 20 iterations, using the Cirrus-CPU machine. Figure 7: Slowdown effect of the different methods compared to the non reproducible version. (**a**) Using 40 MPI-only processes on the Cirrus machine; (**b**) Using one MPI+CUDA GPU process on the Cirrus-GPU machine. The runtime overhead of the preprocessing preparations of the temporary array and coloring methods against the number of MPI processes are detailed in Figure 9 and using only one process in Table 4.

Figure 10 shows how well the test applications scale with the different methods using one, two, four, and eight nodes on the Cirrus cluster.