## 4 Parallel Version of the Improved Algorithm and Its Software Implementation
### Taking into Account the Capabilities of the Standard MPI-4
For test calculations, we limited ourselves to \(s=400\) iterations of the conjugate gradient method. In this case, the running time \(T_{1}\) of the sequential version of the function conjugate_gradient_method() on one computing node was \(\sim\)755 s (in the case of \(N\) iterations using the \"classic\" stopping criteria, this time would increase to \(\sim\)1.5 days). The parallel version of the program was launched with each MPI process bound to exactly one computing node. Recall the feature of the function dot() from the package numpy, which performs the bulk of the calculations when implementing the algorithm: this function is implemented in the C++ programming language and automatically uses multithreading in calculations if the program is running on a multi-core processor--all processor cores are used through the use of OpenMP parallel programming technology. Thus, by linking each MPI process to a separate computing node, the MPI+OpenMP hybrid parallel programming technology was automatically implemented. Python packages used were: (1) numpy version 1.24.3, (2) mpi4 py version 4.0.0.dev0, (3) mpi4 version 4.1.1. The program was run on the number of processes \(n\equiv\texttt{numprocs}\), which is the square of a natural number (1, 4, 9, 16, 25, 36, 49, etc.). This is due to the fact that the test programs used a virtual topology of processes with a two-dimensional Cartesian grid with parameters num_row \(=\texttt{num\_col}=\sqrt{n}\). The running time \(T_{n}\) of the parallel version of the function conjugate_gradient_method() was detected for each value \(n\) from the specified number of processes. Using the estimated running time \(T_{1}\) of the sequential version of the function conjugate_gradient_method(), the acceleration \(S_{n}\) was calculated using the formula \(S_{n}=\dfrac{T_{1}}{T_{n}}\), and then the efficiency \(E_{n}=\dfrac{S_{n}}{n}\) was also calculated. Figure 5 shows graphs of the parallelization efficiency \(E_{n}\) depending on the number of computing nodes \(n\) involved in the calculations. Graphs of such dependencies are presented for three software implementations of the algorithm using different MPI standards. This shows the undeniable advantage of using the MPI-4 standard. The charts fully correspond to the previously stated expectations. It is important to note the following. The design of the parallel algorithm assumes that the MPI processes involved in the calculations form a two-dimensional Cartesian grid (see Figures 3 and 4). The MPI function Allreduce() used in the parallel implementation of the

Figure 5: Plots of parallelization efficiency depending on the number of computing nodes for various software implementations of the parallel algorithm. Strong scalability is present only for software implementation using the standard MPI-4. algorithm will work most efficiently if this two-dimensional Cartesian grid of processes is periodic in both directions. Therefore, for calculations, the virtual topology of processes of the \"two-dimensional torus\" type was used. Thus, comm\(:=\) comm\(\_\)cart, where comm\(\_\)cart:

comm\(\_\)cart = comm\(\_\)Create\(\_\)cart(dims=(num\(\_\)row, num\(\_\)col),  periods=(True, True),  reorder=True)

However, the virtual topology of MPI processes cannot always be effectively mapped onto a real physical communication network connecting the computing nodes of a supercomputer system. Therefore, situations are possible in which MPI processes, neighboring in the virtual topology, work on computing nodes located far from each other in the communication environment. This will lead to a significant increase in message passing time between some MPI processes (see Figure 6). This, as a consequence, can lead to a catastrophic drop in efficiency. Note that the results shown in Figure 5 correspond to the averaged values over a series of parallel program launches on arbitrary distributions of MPI processes over the computing nodes of the supercomputer \"Lomonosov-2\".