**Project Report for Blue Gene Watson Consortium Days**

**March 29, 2006**

**Project Title:**

Global Eddy-Resolving Ocean Modeling on Blue Gene

**Investigators**:

> John M. Dennis
>
> Scientist
>
> Computational Science Section
>
> Scientific Computing Division
>
> National Center for Atmospheric Research
>
> Julie McClean
>
> Associate Researcher
>
> Scripps Institution of Oceanography
>
> Physical Oceanography Research Division
>
> University of California, San Diego
>
> Henry M. Tufo
>
> Scientist
>
> Computational Science Section
>
> Scientific Computing Division
>
> National Center for Atmospheric Research

**Project Results:**

We examined the ability of the Blue Gene/L system to perform global
eddy-resolving ocean modeling using the Parallel Ocean Program (POP). We
used the POP 0.1° benchmark to examine the impact of two code
modifications to increase simulation rate. We replaced the current
conjugate gradient solver that uses a 2-dimensional (2D) data structure
with alternative implementation based on a 1-dimensional (1D) data
structure. The 1D data structure removes land points thereby reducing
the amount of data that must be loaded from the memory hierarchy and
exchanged between MPI processes. The second technique uses space-filling
curve (SFC) partitioning to reduce load imbalances. We discovered that
the combination of both techniques enable a doubling of the simulation
rate of the POP 0.1° benchmark on 29K processors of Blue Gene/L. A plot
of our results from the Blue Gene Watson, Earth Simulator, and RedStorm
supercomputers are provide in Figure 3. We expect that the benefit of
our code modifications would not be limited to Blue Gene/L but would
also improve RedStorm results as well.

**Technical Analysis:**

> The goal of our work during BGW Consortium days was to examine two
> techniques to improve the scalability and efficiency of the Parallel
> Ocean Program (POP) on a 0.1° grid on the Blue Gene/L system. The 0.1°
> resolution allows for a more realistic representation of ocean bottom
> bathymetry and costal geometry, and allows for the accounting of the
> majority of ocean energy budget. The computational requirements of a
> 0.1° ocean simulation are staggering and are only possible on a
> limited number of supercomputers. Previous results from the LLNL
> system for the POP 0.1° benchmark indicates that its scalability was
> limited to only 15K processors on Blue Gene/L. We therefore used the
> computer time provided through BGW Days to examine the impact of two
> techniques on the scalability of POP. The first technique involves the
> rewriting of the conjugate gradient solver using a 1-dimensinal (1D)
> data structure. Use of the 1D data structure allows the removal of
> land points thereby reducing the amount of data that must be loaded
> from the memory hierarchy and exchange between MPI processes. The
> second technique involves the use of a space-filling curve (SFC)
> partitioning algorithm within POP. SFC based partitioning has be used
> to eliminate load-imbalances with the High Order Methods Modeling
> Environment (HOMME) which has demonstrated excellent scaling on 32K
> Blue Gene/L processors (Almasi, et. al, 2005). We describe the use of
> space-filling curves for partitioning in greater detail in the next
> section, followed by a complete description of performance results.

**Partitioning for Load Balancing**

POP decomposes the computational mesh into logically rectangular
2-dimensional (2D) blocks (Jones et. al, 2005). The computational mesh
is distributed across multiple processors by placing one or more 2D
blocks on each processor. Blocks that do not contain any ocean points
are discarded.

We apply SFC partitioning to POP by dividing the computation grid into
2D blocks such that the number of blocks in the x-direction (Nb~x~) and
y-direction (Nb~y~) are Nb~x~ = Nb~y~ = 2^n^ 3^m^ 5^p^ where n, m, and p
are integers. A global 1° POP grid with 20 x 24 grid points per block is
illustrated in Figure 1. The space-filling curve that orders each block
into a 1D curve is illustrated in Figure 2. Note that blocks that
correspond entirely to land points are excluded from the curve.
Partitioning across processors is achieved by subdividing the
1-dimensional curve into approximately equal length segments. The SFC
partitioning of the gx1v3 grid on 8 processors (indicated by the various
colors) is illustrated in Figure 2.

![](media/image1.jpeg){width="5.220138888888889in" height="3.70625in"}

Figure 1: A global 1° POP grid with which corresponds to land points,
gray to ocean points, and superimposed lines indicated blocks with 20x24
grid points per block

![](media/image2.png){width="4.229861111111111in"
height="3.441666666666667in"}

> Figure 2: The space-filling curve, which excludes land blocks,
> provides an ordering of the ocean blocks. A partitioning for 8
> processors is indicated by the various colors.

**Performance Results**

> As mentioned previously, POP decomposes the computational grid into 2D
> blocks that are distributed across processors. Traditionally POP has
> been configured such that a single block, whose size is determined by
> dividing the computational grid into a 2D Cartesian grid, is allocated
> per processor. This configuration that we refer to as \`single block'
> has the potential to create load imbalances due to the presence of
> land. We refer to the configuration, described in the previous section
> that allows for the use of multiple 2D blocks per processor as
> \`sub-block'.
>
> We provide a plot of the simulation rates for POP 0.1° benchmark using
> a the single block configuration on the IBM Blue Gene/L, Earth
> Simulator, and RedStorm supercomputers in Figure 3. Note that the
> notation 1D and 2D in Figure 3 refers to the use of the conjugate
> gradient solver based on 1D and 2D data structures. The results for
> the single block + 2D configuration on Blue Gene/L in Figure 3 clearly
> illustrates scaling limitations. Similar scaling difficulties are
> apparent for the single block + 2D configuration on the Earth
> Simulator and RedStorm. Use of the 1D data structure in the conjugate
> gradient solver improves scaling on Blue Gene/L, increasing simulation
> rate from 3.9 to 6.2 simulated years per wall-clock day. The increased
> scalability is likely a result of a reduction in message passing
> volume. Scalability on Blue Gene/L is further enhanced through the use
> of SFC partitioning. Simulation rate is nearly doubled from 3.9 to 7.9
> simulated years per wall-clock day versus the base single-block + 2D
> configuration.

**Conclusions and Future Work:**

> Access to the large Blue Gene Watson system allowed us to examine two
> techniques to address scalability of the POP 0.1° benchmark. We
> discovered that reducing the amount of data both loaded from the
> memory hierarchy and passed between MPI processes and an improved
> partitioning algorithm significantly increases scalability on 29K Blue
> Gene/L processors. Despite the significant increase in scalability, we
> believe that further improvements are still possible. In particular,
> it was discovered that POP does not currently aggregate messages
> potentially increasing communication cost. Regrettably, time
> limitations prevent the examination of alternative process mappings.
> The snake processor mapping was demonstrated to have a significant
> impact on the performance of HOMME (Almasi et. al, 2005) for processor
> counts greater than 12K.
>
> Our scaling study clearly illustrates that it is possible to efficient
> utilize 29K processors for Geoscience applications. Interestingly, the
> communication dominated conjugate gradient solver that typically
> limits scaling of POP only consumes 21% of the total time on 29K
> processors. These results suggest that POP could benefit tremendously
> from access to the second floating-point unit on the Blue Gene
> processor. Scaling to this type of processor counts however requires
> regular access to large-scale systems. For example without early
> access to the LLNL system, we would have been ill prepared to address
> the scaling issues associated with the single-block + 2D
> configuration. Based on the strength of our results, we propose that
> 32K processors of the Blue Gene Watson system be dedicated for
> approximately 9 days to the 55-year hind-casting experiment described
> below.

![](media/image3.jpeg){width="5.988194444444445in"
height="4.492361111111111in"}

Figure 3: Plot of simulated years per day for the POP 0.1° benchmark on
several computer systems. Data courtesy of Y. Yoshida and M. Taylor.

**Resource Requirements for eddy-resolving ocean simulation:**

Based on the improvements made to the POP model, we describe the
resources necessary for 55-year run. A 55-year hind-casting experiment
would involve a 15-year spin-up during the years 1951 -- 1966, followed
by a comparison with the observed state of the ocean from 1967 to
present.

1.  **Computing requirements (Rack-days, rack count vs. time tradeoffs
    possible):**

> Due to time limitations we were unable to measure the cost of disk
> I/O. We however estimate the cost be reducing the simulated years per
> wall-clock day by 20%. The 20% estimate is based on experience with a
> 0.1° POP run on a POWER4 based cluster. The following table contains
> the computational cost of a 55-year run for various processor counts.
> The additional code modifications mentioned in the previous section
> may further reduce the required number of wall-clock days.

1.  **Storage requirements (maximum working set)**

> 2\*710 GB disk space for 15 year spin-up = 1.5 TB
>
> [710 GB disk space per year \* 40 years = 28 TB]{.underline}
>
> Total = 30 TB

**Reference**:

G. Almasi, G. Bhanot, D. Chen, M. Eleftheriou, B. Fitch, A. Gara, R.
Germain, M. Gupta, M.C. Pitman, A. Rayshubskiy, J. Sexton, F. Suits, P.
Vranas, B. Walkup, T.J.C. Ward, Y. Zhestkov, A. Curioni, W. Curioni , C.
Archer , J.E. Moreira, R. Loft, H.M. Tufo, T. Voran, K. Riley. 2005.
*Early Experience with Scientific Applications on the Blue Gene/L
Supercomputer*, Proceedings of Euro-Par 2005, Lisbon, Portugal.

W. D. Collins, C. M. Bitz, M. L. Blackmon , G. B. Bonan, C. S.
Bretherton, J. A. Carton, P. Chang, S. C. Doney, J. J. Hack, T. B.
Henderson, J. T. Kiehl, W. G. Large, D. S. McKenna, B. D. Santer, R. D.
Smith. 2005. The Community Climate System Model: CCSM3, *Journal of
Climate: CCSM Special Issue* 11(6) To appear

E. F. D'Azevedo, V. L. Eijkhout, and C. H. Romaine. 2002. Conjugate
gradient algorithms with reduced synchronization overhead on distributed
memory multiprocessors. *Technical Report 56 Lapack Working Note*

J. M. Dennis and E. R. Jessup. 2006. Applying Automated Memory Analysis
to improve Iterative Algorithms. In preparation.

P. W. Jones, P. H. Worley, Y. Yoshida, J. B. III White, J. Levesque.
2005.

Practical Performance Portability in the Parallel Ocean Program (POP).

*Concurrency Comput. Prac. Exper.* 17(10): 1317-1327

M. E. Maltrud, and J. L. McClean. 2005. An eddy resolving global 1/10
POPdegree ocean simulation *Ocean Modeling* *8(1-2): 31-54*