## 6 Discussion and Conclusions  
The KNL system was developed with the advantage of GPGPU having 68 cores per CPU. Although the KNL system has more than twice as many cores per node as the Skylake system, the KNL system performs at approximately 1/3 of the performance rate of the Skylake system due to slow CPU clock speed and low memory. However, the performance rate of the Nurion KNL system was approximately 43% for all experiments, indicating that Nurion KNL system's excellence. There is almost no difference in performance in the several runs of experiments, thereby indicating that the KNL system is very stable.  
WRF, ROMS, and FVCOM were tested for optimization by core configuration, MPI libraries, various compiler options, memory modes and so on. In conclusion, the best approach is to compile with only the \"xMIC-AVX512\" compilation option in the cache memory mode. All 32 cores were used for the real experiments of ROMS and FVCOM because a large access of I/O caused out of memory. The performances of the real experiments were lower than those of the ideal experiments because the distributions of the calculated amounts are different between cores due to the differing I/O loads. Performances were compared between the SKL and the CLS systems based on the KNL system (Figure 20). When the same numbers of cores were used, the average performance of the KNL system was about 43% (highest: 64%, lowest: 31%) of SKL or CLS (128 cores: 46%, 256 cores: 43%, 512 cores: 39%).  
Based on our results, if the computation performance of each model is listed in order from high to low, then it would WRF, ROMS, and FVCOM, and if the parallel efficiency of core numbers is listed in order from high to low, then it would 128, 256, and 512 cores. Ideal experiments with a low I/O showed a higher performance than the real experiments. The ROMS model showed similar performances in both ideal and real experiments compared to the other models.  
It is more efficient to reduce the number of cores per node in a KNL system by half (36 cores) when the total number of cores is less than 256 cores, and it is economical to use all cores when using more than 256. Therefore, it is economical to use all cores of nodes and reduce the number of nodes when simulating ultra-high resolution models. In all experiments, the performance was continuously improved, even in the maximum core experiment (1024 cores), thereby indicating that the KNL system can sufficiently simulate ultra-high resolution models.  
Conceptualization, D.-H.K. and C.L.; data curation, D.-H.K. and C.L.; funding acquisition, M.J., J.A. and I.-J.M.; methodology, D.-H.K. and C.L.; project administration, S.-B.W.; resources, M.J., J.A. and I.-J.M.; supervision, D.-H.K.; writing--original draft preparation, C.L.; writing--review and editing, D.-H.K., C.L. and S.-B.W. All authors have read and agreed to the published version of the manuscript.  
This research was part of the project titled \"Improvements of ocean prediction accuracy using numerical modeling and artificial intelligence technology,\" funded by the Ministry of Oceans and Fisheries, Korea.  
We thank the Korean Institute of Ocean Science and Technology (KIOST).  
The authors declare no conflict of interest.  
Figure 20: Parallelization efficiency by model (WRF, ROMS and FVCOM) depending on the number of cores (128, 256 and 512).