## 3 Parallel Computing Based on a Hybrid OpenMP-MPI Approach
### Parallelization Strategy Based on MPI Approach: Description and Limits
```
Equally distribute element-element interactions among MPI processes // Initialization for each MPI process  Allocate dummy memory MdMesh  Allocate dummy memory MdEE  Allocate dummy memory MdL endfor Broadcast the geometrical information // Parallel pure MPI computation for each MPI process do for each element iell do for each element iel2 do  Compute local iell-iel2 interactions  Accumulate local interactions in MdL endfor endfor //Final Communications Step for each MPI process do  Allocate local memory Mloc endfor for each MPI process do  Send and receive the local matrices MdL  Accumulate in Mloc endfor  deallocate(MdMesh, MdEE,MdL)
```  
**Algorithm 2** Assembly of \(\mathbf{L}\) matrix, pure MPI approach  
As for task (c), the assembly of the flux density matrix, \(\mathbf{Q}\), is obtained through a double loop, the first on the mesh elements, and the second one on the requested field points, as shown in Algorithm 3. In a pure MPI approach, the set of field points are equally distributed among MPI processes. Memory is then allocated in each MPI process. As a general comment, we stress that with the pure MPI paradigm, due to the limited size of per node memory and to the need of each MPI process to allocate not only the memory required to store the local portion of the matrices but also the dummy memory required to carry on the computation, it may not be possible to launch a number of MPI process equal to the number of the physical cores available at the node. For the above reasons, many of these cores are forced to stay idle during the job execution because there is no more space to allocate. This fact limits the maximum achievable speed-up. For example, to compute the \(\mathbf{L}\) matrix, the mesh information should be available at each MPI process in the dummy memory _MdMesh_ defined above: in practical applications, this memory likely reaches the dimensions of some GBs. In the following, we refer to the typical case of homogeneous supercomputing systems, made by nodes that are equal in terms of core number, memory, and operating frequency. In order to assess the performance, let us here define the following quantities:  
- \(N_{MPI}\)--number of the MPI processes;  
- \(N_{N}\)--number of the cluster nodes;  
- \(M\)--total memory required to store the global matrix (for example, \(\mathbf{L}\));  
- \(M_{N}\)--total memory available at any node;  
- \(M_{D}\)--dummy memory per MPI process;  
- \(M_{DT}\)--dummy memory per thread;  
- \(M_{LOC}\)--memory available at each node (for all is needed at the node);  
- \(M_{AV}\)--actual available memory at each node. In the ideal case, the maximum computation speed-up would be equal to the total number of MPI processes, \(N_{MPI}\). At each node, the number of running processes is \(N_{MPI}/N_{N}\), and thus the available memory is given as \(M_{LOC}=M/N_{N}\). Note that the available memory must be used not only the matrix storage (chunk memory) but for any other data required by the process. Unfortunately, the dummy memory, \(M_{D}\), is replicated \(N_{MPI}/N_{N}\) times, due to the distributed memory approach adopted here. Therefore, the actual available memory at each node, \(M_{AV}\), is reduced with respect to \(M_{N}\), and must be larger than the requested chunk memory, \(M_{LOC}\). These conditions are summarized in (11), that sets the limit for the pure MPI approach by imposing a maximum value to \(N_{MPI}\):  
\[M_{AV}=M_{N}-\frac{N_{MPI}}{N_{N}}M_{D}\geq M_{LOC}=\frac{M}{N_{N}} \tag{11}\]