## 3 Optimization of Parallelization
### Optimization B: Communication Optimization  
In order to reduce the time consumed, we optimized the communication procedure. The method of data packing was employed to improve the communication time. In addition, the algorithms were improved by replacing communication with calculation.  
(I) For a high-latitude area, a one-dimensional horizontal smoothing algorithm needs to conduct smoothing more than once. Since the times of smoothing on different latitudes are not the same, the original algorithm conducts smoothing once on each latitude, while the new algorithm involves just one smoothing and two-dimensional communication at all latitudes. Algorithms 2 and 3 show the representative code alterations. Algorithm 2 provides the original code, and Algorithm 3 describes the optimized version.  
Figure 3: Ocean grid boundary communication.  
(II) Based on previous test results when using Intel Vtune, the hotspots of LICOM are the communication functions. Therefore, the optimization of communications is of great importance. During the process of integration, the least communication algorithm was employed. Some communications were replaced by adding boundary calculations. It was beneficial to eliminate communication by adding some calculations. Because of the large number of PEs used, the total calculation job was divided by the number of PEs, while the cost of communication was multiplied by the number of PEs. Algorithms 4 and 5 show the representative code alterations. The original code in Algorithm 4 was optimized into the code in Algorithm 5 to eliminate the cost of communication.  
Figure 4: Optimization of the parallelization scheme.  
```
forNCY\(\leftarrow\)1 toMAX_NNdo forj\(\leftarrow\)jst to jmtdo ifNN(j)\(\geq\)NCYthen fork\(\leftarrow\)1 toKKdo forI\(\leftarrow\)1 toIMTdo  CalculateXS(I);  end forforI\(\leftarrow\)2 toIMMdo  CalculateX(I,J,K);  end for  end for  end  Boundary Exchange 2D;  end for
```  
**Algorithm 3**Communication Optimization (I-B)  
```
forJ\(\leftarrow\)JSM to JEMdo forI\(\leftarrow\)2 toIMMdo  CalculateUB(I,J), VB(I,J), and H0(I,J) ;  end for  end for  Exchange boundary ub, vb, and h0 ; forJ\(\leftarrow\)JSM to JEMdo forI\(\leftarrow\)2 toIMMdo  CalculateWKA(I,J,1), WKA(I,J,2), and WORK(I,J) ;  end for  end for
```  
**Algorithm 4**Communication Optimization (II-A)  
```
forJ\(\leftarrow\)JST to JETdo forI\(\leftarrow\)1 toIMTdo  CalculateUB(I,J), VB(I,J), and H0(I,J) ;  end for  end for forJ\(\leftarrow\)JST to JETdo forI\(\leftarrow\)1 toIMTdo  CalculateWKA(I,J,1), WKA(I,J,2), and WORK(I,J) ;  end for  end for
```  
**Algorithm 5**Communication Optimization (II-B)  
(III) The grid matching information was obtained via communication in the original code in Listing 1. In contrast, in the optimized algorithm, it was obtained by conducting a calculation in Listing 2. Therefore, point-to-point communication was reduced by a considerable amount.  
```
if(mytid==0)then i_start(1)=i_global(1) j_start(1)=j_global(1) don=1,nproc-1 callmpi_recv(j_start(n+1),1,mpi_integer,n,tag_1d,& mpi_comm_ocn,status,ierr) callmpi_recv(i_start(n+1),1,mpi_integer,n,tag_2d,& mpi_comm_ocn,status,ierr) enddo else j_start(1)=j_global(1) i_start(1)=i_global(1) callmpi_send(j_start(1),1,mpi_integer,0,tag_1d,& mpi_comm_ocn,ierr) callmpi_send(i_start(1),1,mpi_integer,0,tag_2d,& mpi_comm_ocn,ierr) endif
```  
Listing 1: Communication Optimization (III-A)  
```
doi=1,nproc iix=mod(i-1,nx_proc) iiy=(i-1-iix)/nx_proc i_start(i)=iix*(imt-num_overlap)+1 j_start(i)=3 if(iiy/=0)then doj=1,iiy j_start(i)=j_start(i)+jmt-num_overlap enddo endif enddo
```  
Listing 2: Communication Optimization (III-B)