## 4 Theory and Calculation
### Temporary Array Method
On the CPU side, all methods have the same parallel efficiency on each application, except the distributed and greedy coloring methods on Airfoil, where we can observe superlinear scaling (Figure 10a) since much of the data used can fit into the cache if they are divided between at least four nodes. We cannot observe this on the temporary array method, because it uses extra memory to store increments separately. Apart from the reductions (discussed in detail below), MPI communications and communication times do not differ between reproducible and non-reproducible. For non-reproducible execution, the communication overhead (as a fraction of total runtime) will become higher using multiple nodes. In the case of reproducible execution, because we spend more time in the colored execution, we spend a smaller fraction of the total time in communications. Therefore, the relative difference is decreasing and the slowdown effect with any method compared to the non-reproducible is less when more nodes are used. \begin{table}
\begin{tabular}{l c} \hline \hline
**App** & **Runtime** \\ \hline Airfoil & 4.65 s \\ Aero & 1.79 s \\ MG-CFD & 128.88 s \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Reversed map creation and greedy coloring time. \begin{table}
\begin{tabular}{l c c} \hline \hline
**App (Map)** & **Greedy** & **Distributed** \\ \hline Airfoil (pecell1) & 4 & 14 \\ Aero (pcell1) & 5 & 17 \\ MG-CFD (edge\(\rightarrow\)node0) & 7 & 19 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Number of colors with the different methods on the applications main map. Figure 9: Scaling of preprocessing overhead. (**a**) reversed map and temporary array creation time for the temporary array method; (**b**) reversed map creation and distributed coloring time. We can observe the strong scaling of a reduction kernel in Figure 11. Since all reproducible methods use the same reduction technique, there is no separate measurement for them. Again on CPUs, we can see the superlinear effect as the application fits more and more into the cache. We can also observe that there is an additional cost of the reduction caused by the reproBlas functions. The most significant factor in the cost of reproducible reduction is that we must write all the values to be reduced into a separate array and perform a reduction on it within a process. This leads to extra memory movement compared to the reference version. This is particularly expensive on GPUs because this array must be copied to the host to perform the local summation. MPI_reduce is not significantly more expensive. Using only MPI parallelization, the overhead is quite small (between 1 and 1.12 times). Using shared memory parallelism, it is a bit greater due to the bad cache locality. In some extreme cases, we can even lose the speedup gain from GPUs, our reproducible methods work better on CPU-only systems. Figure 10: Strong scaling measurement of the different methods, using 1,2,4,8 nodes; (**a**) Airfoil, using 36 MPI Intel Xeon CPU processes per node; (**b**) Airfoil, using four Nvidia V100 GPU processes per node; (**c**) Aero, using 36 MPI Intel Xeon CPU processes per node; (**d**) Aero, using four Nvidia V100 GPU processes per node; (**e**) Mg-cfd, using 36 MPI Intel Xeon CPU processes per node; (**f**) Mg-cfd, using four Nvidia V100 GPU processes per node. ## 6 Conclusions

In this paper, we examined the non-reproducibility phenomenon that occurs due to the non-associative property of the floating-point number representation on applications defined on unstructured meshes. We compared the differences in results without reproducibility across a range of applications, including Rolls-Royce's production application, Hydra. Non-reproducibility is a widely studied problem; however, we have not yet found an effective solution for distributed systems that could also be applied to arbitrarily partitioned meshes. In this work, we developed a collection of parallel and distributed algorithms to create a plan and then execute it, guaranteeing the reproducibility of the results.