## 4 Experiments
Unless a section of code was sequential, such as the deflation process, we employed single instruction and multiple data sentences. Our decision is based on the granularity of the task and keeping communication between nodes at a minimum, to decide whether computation takes place in shared or distributed memory. To parallelize the secular equation solution, first, we calculate \(\mathbf{v}^{2}\) on the GPU, eorfn-nug a one-dimensional parallelization because we will use those values later. We calculate \(d_{r}-\lambda_{c}\) and \(\lambda_{c}\) for each secular equation root, executing a CUDA kernel over secular equation intervals. As we will use results on the GPU, we keep them in the GPU's memory to avoid unnecessary copies and make the execution faster. We compute the secular equation solution in GPU for matrices of considerable size, which is performed sequentially in other implementations or using OpenMP. However, we have seen time gains because we keep data in GPU memory for the whole rank of one modification. So, our implementation is one of the first to use this approach. To parallelize orthogonality eigenvalue correction, we use Equation (8) (in p. 8). We use \(d_{r}-\lambda_{c}\) values, which form a matrix of size \(n\times n\); we apply a one-dimension parallelization to compute each value of vector \(\mathbf{v}^{\prime}\). Again, we keep data in GPU memory to avoid unnecessary memory copies. We can calculate the rank one modification eigenvectors with the corrected values of \(\mathbf{v}^{\prime}\). We perform a one-dimensional parallelization to calculate the norm, which divides each column vector of the eigenvector matrix \(Q_{r1}\) using a CUDA kernel. Then, we use a two-dimensional parallelization to obtain each matrix element \(Q_{r1}\) using another CUDA kernel. To parallelize Givens' rotations application, we use a loop that crosses each rotation and creates OpenMP threads so each rotation is applied to the corresponding column affected by rotations. Initially, we thought about using CUDA, but the number of rotations was insufficient, so the time to transfer to the GPU memory was longer than the time to apply rotations. To parallelize the application of permutations, we use one-dimensional parallelization using OpenMP. Each thread calculates a new index of the composed permutation and copies the elements from one column to its new corresponding location. Again, we do not use CUDA because copy time is more than permutation time. Matrix multiplication is used to produce the eigenvectors of the scaled matrix from the multiplication of subproblem eigenvectors and rank one modification. We use the cublasDgemm CUBLAS function because it is the fastest version we know. Since multiplication is one of the most time-consuming operations, we also develop a version that uses MPI to split the problem. Hence, each node calculates half of the final result columns. Other parallelizations that are not part of the algorithm's main functions, such as escalations and error computation, were parallelized, identifying loops in which we could use OpenMP. With this implementation, we can manage matrix sizes up to 46,000, only limited by the index we use to access an element in an array, which maintains the solution eigenvectors. The next step is to know how good our solutions are. To answer this question, we have used

\[A = \frac{1}{\alpha}B, \tag{11}\]4 Process flow. the Forbenious matrix norm. We use this norm because it has a high degree of parallelism. We developed multi-core parallel error routines, which show an almost linear speedup in the number of cores. These implementations allowed us to obtain results about ten times faster than sequential versions. Although we use other libraries to solve subproblems and matrix multiplications, we implemented all the different parts of the algorithm very efficiently, starting from zero.