# _teaching_ history = model.fit(train_dataset,epochs=epochs, callbacks= callbacks)
#### 4.2.4 Methodology Supplement
The participation of inter-process communication in the context of the overall program is significant in the case of the currently analyzed application. This mainly results from the fact that between each training epoch, there is synchronization of processes and averaging of weights in distributed neural networks using the allreduce method. It should be remembered that the time cost of the allreduce algorithm performed between training epochs increases with the addition of more processes, because the neural network is not divided among sub-processes. Each of them has an identically sized structure. Therefore, the more processes, the more time needs to be allocated for averaging weights. \begin{table}
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{\(n_{p}\)} & \multicolumn{2}{c}{\(t_{agg}\) [s]} & \multicolumn{2}{c}{**An Absolute**} & \multicolumn{2}{c}{**Relative**} \\ \cline{2-5}  & **OpenMPI** & **MPICH** & **Improvement [s]** & **Improvement [\%]** \\ \hline
1 & 89.761 & 53.270 & 36.491 & 40.654 \\
2 & 74.169 & 44.024 & 30.145 & 40.643 \\
3 & 45.129 & 47.449 & 2.320 & 4.889 \\
4 & 57.258 & 58.395 & 1.137 & 1.948 \\
5 & 64.139 & 67.359 & 3.221 & 4.781 \\
6 & 71.675 & 80.180 & 8.505 & 10.607 \\
7 & 105.357 & 99.827 & 5.529 & 5.248 \\
8 & 110.490 & 99.555 & 10.936 & 9.897 \\
9 & 146.067 & 106.561 & 39.506 & 27.046 \\
10 & 125.540 & 102.463 & 23.077 & 18.382 \\
11 & 142.171 & 108.794 & 33.377 & 23.477 \\
12 & 125.471 & 118.918 & 6.553 & 5.223 \\ \hline \end{tabular}
\end{table} 
Table 2: Average execution times of a neural network training application. One process per container. Performance comparison of MPICH and OpenMPI implementations. Since the mediocre scalability resulted partly from the overload of resources such as the processor, it might be possible to improve this by imposing restrictions on its use. Docker allows setting limits on the usage of processor time for containers. Listing 11 shows how to enable this function for a virtual cluster. Conventionally, the value of the cpus option denotes the maximum number of cores allocated to the container. In practice, the level of restriction affects the time allocation of the container's tasks to the processor. For example, if there are two cores in the computer and the modification from Listing 11 is applied, the container will be able to load the processor to a maximum of 50% of its capacity per second. After applying the changes from Listing 11, tests were repeated for both implementations. Thanks to the limitation on processor time with MPICH, the scalability was artificially slightly improved, as can be inferred from Table 5. However, the results obtained were still not satisfactory. The minimum execution time shifted to five processes, and the expected characteristic became slightly more uniform. The greatest relative deviation in the expected characteristic from the average time dropped to about 28%. Unfortunately, the same could not be demonstrated with the OpenMPI implementation, as the time for each subsequent process always increased for unexplained reasons. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \(n_{p}\) & \(t_{avg}\) [s] & \(t_{pred}\) [s] & \(t_{id}\) [s] & \(\frac{|t_{avg}-t_{pred}|}{t_{avg}}\) **\(\cdot\) 100 [\%]** & \(\frac{|t_{avg}-t_{id}|}{t_{avg}}\)**\(\cdot\) 100 [\%]** \\ \hline
2 & 44.024 & 26.635 & 26.635 & 39.500 & 39.500 \\
3 & 47.449 & 29.349 & 17.757 & 38.145 & 62.577 \\
4 & 58.395 & 35.587 & 13.317 & 39.059 & 77.194 \\
5 & 67.359 & 46.716 & 10.654 & 30.646 & 84.183 \\
6 & 80.180 & 56.133 & 8.878 & 29.992 & 88.927 \\
7 & 99.827 & 68.726 & 7.610 & 31.155 & 92.377 \\
8 & 99.555 & 87.349 & 6.659 & 12.260 & 93.312 \\
9 & 106.561 & 88.493 & 5.919 & 16.956 & 94.446 \\
10 & 102.463 & 95.905 & 5.327 & 6.400 & 94.801 \\
11 & 108.794 & 93.148 & 4.843 & 14.381 & 95.549 \\
12 & 118.918 & 99.728 & 4.439 & 16.137 & 96.267 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Average execution times of the neural network training application. One process per container. MPICH implementation.