Preparing NEMO4.2, the new NEMO modelling framework for the next generation HPCinfrastructures
Authors: Mele Francesca , Epicoco Italo , Mocavero Silvia , Calvert Daley , Bell Mike 
Published: March 2022

&lt;p&gt;Nowadays one of the main challenges in scientific computational field is developing the next&lt;br&gt;generation of HPC technologies, applications and systems towards exascale. This leads to focus the&lt;br&gt;efforts on the development of a new, efficient, stable and scalable NEMO reference code with&lt;br&gt;improved performances adapted to exploit future HPC technologies in the context of CMEMS systems.&lt;br&gt;On the main factors that limit the current scalability is an inefficient exploitation of the single node&lt;br&gt;performance. Different technical solutions have been tested to fully exploit memory hierarchies and&lt;br&gt;hardware peak performance. Between all, the fusion of DO loops together and by dividing the&lt;br&gt;computation over tiles are the two optimization strategies more efficiently take advantage of the&lt;br&gt;cache memory organization. This work focuses on the first one.&lt;br&gt;The loop fusion is a transformation which takes two adjacent loops that have the same iteration space&lt;br&gt;traversal and combines their bodies into a single loop. This optimization improves data locality so&lt;br&gt;giving a better exploitation of the cache memory and a reduction of the memory footprint because the&lt;br&gt;temporary arrays can be replaced with scalar values.&lt;br&gt;Performance tests have been executed on a domain size of 3002x2002x31 grid points running 1-year&lt;br&gt;GYRE_PISCES simulations with IO disabled on the Zeus Intel Xeon Gold 6154 machine, available at&lt;br&gt;CMCC. An increasing number of cores - from 504 to 2016 &amp;#8211; have been used to test experiments with&lt;br&gt;the different HPC options.&lt;br&gt;The analysis focused on the routines where the optimizations have been applied. The use of the&lt;br&gt;extended halo introduces a penalty in the execution time that grows as the number of processes&lt;br&gt;increases and generally the use of loop fusion optimization slightly improves the performance. For&lt;br&gt;many routines, as subdomains get smaller, the improvements due to optimizations are less significant.&lt;br&gt;The simultaneous application of all optimizations leads to an improvement between 10% and 50%&lt;br&gt;(except for lateral diffusion). Looking at the total elapsed time, the new HPC optimizations speed up&amp;#160;&lt;/p&gt;&lt;p&gt;the elapsed time of a factor 1.25x. Unfortunately, non-optimized routines mitigate this improvement.&lt;/p&gt;&lt;p&gt;The same scalability test has been repeated running 1-month ORCA025 simulations with the output&lt;br&gt;set to be produced at the end of the run. The results show that the use of loop fusion optimization&lt;br&gt;slightly improves the performance. The use of tiling in ORCA025 introduces less benefits with&lt;br&gt;reference to GYRE. The simultaneous application of all optimizations doesn't lead many benefits in&lt;br&gt;ORCA025 since the improvement concerns only a subset of routines with the Tracer lateral diffusion&lt;br&gt;routine getting worse in all cases.&lt;br&gt;In conclusion the impact of the new optimized code behaves differently depending on the&lt;br&gt;configuration. The overhead introduced by the extended halo implies a computation time cost that the&lt;br&gt;proposed optimizations are able to regain difficultly. Tiling is the aspect with the highest impact in&lt;br&gt;these optimizations (especially in GYRE) and loop fusion has in general a low impact. The optimizations&lt;br&gt;should be applied to all the rest of the code to obtain more benefits.&lt;/p&gt;