## 4 Parallel Performance and Application in Actual Scenarios
### Performance on Era  
The CPU on the computing nodes of Era was an Intel(R) CPU E5-2680V32.5 GHz. The operating system was Linux CentOS release 6.4 (Final). The compilers were Intel composer_xe_2013_sp1.0.080 and Intelmpi 4.1.3.049. Based on the above environment, we conducted tests using up to 4800 processor cores. Since LICOM is so complicated, the elapsed time of LICOM when simulating one model day is used as an indicator of performance. Figure 6 shows the elapsed time and running speed of the three editions of LICOM, while Figure 7 shows the speedups. The simulation year per day (SYPD) is usually used to measure the computational performance of models. The speedups for the three editions of code were calculated separately. The elapsed times for each edition of code on 1200 PEs were chosen as references. For instance, the speedup of the original code on 4800 PEs over 1200 PEs was the elapsed time of the original code on 1200 PEs divided by the elapsed time of the original code on 4800 PEs. The speedup of the fully optimized code on 4800 PEs over 1200 PEs was the elapsed time of the fully optimized code on 1200 PEs divided by the elapsed time of the fully optimized code on 4800 PEs. The reason for why the elapsed times of 1200 PEs were chosen as references was that, with a resolution of 10 km \(\times\) 10 km, LICOM needed a very large memory space. Thus, we needed more PEs so that after decomposition, the limited memory space on each PE could meet the demands  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**No.** & **Description** & **Machine** \\ \hline
1 & Performance test with three versions of code & Era and Tianhe II \\
2 & Scalability test & Tianhe III \\
3 & Real scenario test & Era \\ \hline \hline \end{tabular}
\end{table}
Table 1: Setup of Tests.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Module** & **LICOM2.1** \\ \hline Horizontal resolution & 0.1\({}^{\circ}\) (lat) \(\times\) 0.1\({}^{\circ}\)(lon) \\ Vertical levels & 55 levels \\ Grid point & 3600 \(\times\) 1683 \\ Advection scheme & Shape-preserving [30] \\ Vertical mixing & Canuto [31] \\ Mesoscale eddy & Gent and McWilliams [32] \\ Horizontal viscosity & 3 \(\times\) 10\({}^{3}\) m\({}^{2}\)s\({}^{-1}\) \\ Forcing formula and dataset & Large and Yeager [33]; COREs \\ \hline \hline \end{tabular}
\end{table}
Table 2: Configurations of the test cases.  
of LICOM. As shown in Figures 6 and 7, the semi-optimized LICOM was much faster than the original LICOM, but it still suffered from the problem of scalability. However, the fully optimized LICOM with the optimization of the decomposition scheme showed considerably good scalability when 4800 processor cores were used. The computing speed reached 9 model years per day. Additionally, the elapsed times of both the original and semi-optimized code on 3600 PEs were longer than those on 2400 PEs. This was a non-intuitive phenomenon. However, for the fully optimized code, there was no similar non-intuitive phenomenon. We can infer that the problem that caused the non-intuitive phenomenon was tackled by the optimizations in the fully optimized code. Therefore, the possible reasons that caused this non-intuitive phenomenon could have been the communication overhead and load imbalance.