# Presentation: 567071

## COLLABORATIVE DESIGN AND DEVELOPMENT OF THE COMMUNITY CLIMATE                                                 SYSTEM MODEL FOR  TERASCALE COMPUTING 

**John Drake (ORNL), Phil Jones (LANL)**
**for the  SciDAC CCSM Consortium**

- Bob Malone

## The SciDAC CCSM Consortium consists of PI:  J. Drake5, P. Jones4, Site-Contacts: W. Collins7, P.Cameron-Smith3, C. Ding2, S. Ghan6, J. Larson1, W. Washington7, R. Rood8, Co-Is: J. Baumgardner4, L. Buja7, S. Chu4, M. Vertensten7, P. Duffy3, J. Dukowicz4, S. Elliot4, D. Erickson5, M. Ham5, Y. He2, F. Hoffman5, E. Hunke4, R. Jacob1, P. Jones4, J. Lamarque7, W. Lipscomb4, R. Malone4, M. Maltrud4, D. McKenna7, A. Mirin3, W. Putman8, W. Sawyer8, J. Schramm7, T. Shippert6, R. Smith4, Y.H.Tseng2, P. Worley5 
1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab, 5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center 

- 1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab, 5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center

## http://www.osti.gov/scidac/updates2004

## CCSM3 Released June 2004  - Mission Accomplished?

**SciDAC/ CCSM Collaboration – Drake(ORNL), Jones(LANL), Collins (NCAR)**
**Software Engineering - Worley(ORNL), Vertensten(NCAR)**
**Coupler Developement -  Larson(ANL), Kauffman(NCAR) **
**Atmospheric Dynamical Cores – Drake(ORNL), Hack (NCAR)**
**POP Ocean Code – Jones (LANL), Norton (NCAR)**
**CICE Sea Ice - Lipscomb(LANL), Hunke (LANL)**
**Atmospheric Chemistry – Cameron-Smith (LLNL),  Lamarque (NCAR)**
**Land and River Modeling – Hoffman (ORNL), Bonon (NCAR)**
**Biogeochemistry  --  Erickson(ORNL), Thornton (NCAR)**
**Global Change Working Group –  Washington (NCAR)**
**Software Engineering Working Group - Worley(ORNL) Co-chair**
**Ice Model Working Group - Hunke (LANL) Co-chair**
**Performance Evaluation Research Center (SciDAC PERC ISIC)**
**Climate benchmarking– Worley (ORNL)**
** ****Vectorization (CRAY, NEC)**
**Grids and Frameworks: **
**SciDAC Earth System Grid **
**       – ****Williams (LLNL), Middleton(NCAR)**
**Earth System Modeling Framework **
**       –****Suarez(NASA-GSFC****),**** Deluca(NCAR) **
**Other SciDAC ISICs: CCA, TOPS, TSTT, ...**

## SciDAC Contributions to CCSM3 Release

**SWE design, parallel algorithms**
**Coupler CPL6 and MCT, MPH libraries**
**Performance/ optimization / vectorization**
***POP 1.4.3 ocean code from LANL***
***CICE sea ice code from LANL***
**DOE computer resources for development**
**DOE computer resources for IPCC runs**

## Goals of the Consortium

**Performance portability for the CCSM**
**Open software design process**
**Layered software architecture to insulate modeling**
**Improving SWE practices**
**Readiness for climate change simulations**
**Improvement of methods and models supporting  more comprehensive coupled climate simulations**
**Higher resolution**
**Dynamics algorithms**
**Atmospheric chemistry**
**Ocean and Land ecology**
**Carbon cycle, aerosols **

## The State of the Software

| Performance portability for the CCSM | Ported to ES and X1, what about BG/L, etc..
 |
| --- | --- |
| Open software design process | External developers, much improved design, design discussions and documents, CRB process |
| Layered software architecture to insulate modeling | Only partial, the unified model as clean up
 |
| Readiness for climate change simulations | Used for IPCC runs on DOE computers |
| Improvement of methods and models supporting  more comprehensive coupled climate simulations |  |
| Higher resolution | T170 tuning and downscaling, 0.4 degree AOMIP, |
| Dynamics algorithms | Revisiting SLD, vectorizing, new dycores |
| Atmospheric chemistry | Mechanism development and simulations
New challenge to performance and design |
| Ocean and Land ecology | POP 2.0 and ocean biology, dynamic vegetation |
| Carbon cycle, aerosols | CASA, C4MIP, CN, ... |

## Porting and Vectorization for CRAY X1 and Earth Simulator

- SciDAC Workshop on Vectorization in Feb. 2003 started the effort.
- CCSM3 release includes vector mods and is used for IPCC on Japanese Earth Simulator, validated on Cray X1
- Effort spanned  NCAR, NASA-Goddard, ORNL, LANL, LBNL, Cray, NEC, Fujitsu, CRIEPI

- Pat Worley will speak on vectorization and other performance issues tomorrow.

## Vector Adaptation

-  **Which application characteristics matched the vector architectures  well/poorly**:
- POP ocean code (best)- finite difference with halo-updates and conjugate gradient barotropic solver
- CSIM ice model (excellent) - only minor mods to finite difference with halo-updates, incremental remapping advection
- CAM atmosphere (good) - vector radiation/physics calculations, spectral dynamics, semi-Lagrangian advection with halo updates
- CLM2 (poor) - pointer data structure required a complete re-write to get an explicit loop in each processing routine. After re-write vector performance is excellent and other platforms are faster as well.
- CPL6 - coupler has modest vector operations (distributed sparse multiply for interpolation and field regridding)
- Fast network bandwidth allows good distributed performance on spectral methods and data transpositions especially important in CAM
- Low network latency allows fast collectives and halo updates especially important in POP** **

## Community Land Model (CLM3.0) Vectorized for Cray X1 and Earth Simulator

**In addition to clumps, another set of structures, called "filters," was added to better support vectorized processing of columns and plant functional types (PFTs).**
**Filters group like columns or PFTs based on their process-specific categorization and are used for indirect addressing into the main data structure hierarchy.**
**Filters are created for snow, non-snow, lake, non-lake, and bare soil columns and PFTs by clump. When the Dynamic Global Vegetation Model is enabled, additional filters are created for naturally-vegetated PFTs.**

- Forrest Hoffman poster presentation.

## Physics Load Balancing

**Notes:**
The task id's are
0:  d_p_coupling and p_d_coupling
1:  phys_gmean
2:  phys_driver
3:  phys_bc / phys_ac
4:  atmlnd_driver
5:  chunk_to_clump and clump_to_chunk
6:  scandyn
7:  scanslt
8:  dyndrv
9:  scan2
10: stepon_startup
11: stepon_st  (which includes i/o)
12: grcalc
13: spegrd
14: tfilt_massfix

The graphs are pretty busy - sorry about that. The important
points to note are

1) Looking at the right task graph (lower right corner),
    in the middle of the x-axis is a large gray-blue task.
    In the left task graph, this task is approx. 1/10 the duration,
    and any load imbalance is now inconsequential.

2) While the red (MPI idle) regions in the top graph
    have not gone away, they still appear to be due primarily to load
    imbalances, rather  than MPI overhead.

##  Evolution of Performance in CAMPat Worley, John Drake, Art Mirin, Will Sawyer, Bill Putman

- CAM2.0 Eulerian Spectral Model at T42L26
- Hybrid MPI/OpenMP programming paradigm
- Cache friendly chunks, load balance, improved algorithms
- FV Core on IBM NHII in lat-vert decomposition
- NASA- DAO’s mod_comm replacing PILGRIM

## Performance of POP and CLM

## Simulations

## AOMIPElizabeth Hunke

**Arctic Ocean Model Intercomparison Project run 1948-2002.  **
**It's global, 0.4 deg, coupled POP2.0 and CICE3.1_beta.  **

**Notes:**
Two attached files are from the Cray run, UV_GM_1954.and UV_GM_1978, showing the velocity vectors at 465 m a typical depth for looking at the Atlantic Layer in the Arctic Ocean). 
 Color is bathymetry (m), and this is on the logical grid (N pole in N America), so Siberia/N Europe are at the bottom
Notice that the flow along the continental margins switches directions!  In some AOMIP models, the flow is ALWAYS in one direction and in other models it's always in the OTHER direction. 

phoenix: produces 14.4 model hours per processor- wallclock hour.  (Currently running with 60 processors)

cheetah:  8.4 model hours per processor-wallclock hour. 

## IPCC Fourth Assessment

- Computational support for IPCC simulations using CCSM3

## Plans for C4MIP SimulationsBiogeochemistry Working Group

**Start at a T31 resolution.**
**Simulations will be "F" runs with active atmosphere and land models using data ocean model and thermodynamic ice run through the coupler.**
** ****Software engineering work is needed on the data ocean and CSIM ice models so that they can cycle through 25 years of forcing data for spin-up.**
**As spin-up progresses,  turn on CO2 flux to the atmosphere with advection, complete implementation of land use change for agriculture, add fossil fuel emissions forcing, etc.**
**As time is available on the Cray X1, the CASA' and CN simulations for C4MIP could be run concurrently at ORNL.**

## Eddy-Resolving Ocean

- 0.1 deg

- 0.28 deg

- Obs

- 2 deg

## Atmospheric Chemistry for A2 ScenarioJ-F Lamarque, S. Walters

**Notes:**
I have attached 2 figures.  They are both aerosol optical depth
from CAM3 (full chemistry and aerosols) in 2000 and 2100 (A2 scenario),
annual averages.
Hi Philip (and John)

>> I take it that the plots are for 2000 and 2100?

yes

>>
>> Did the runs include radiative feedback from ozone, other reactive gases,
>> aerosol direct forcing, and aerosol indirect forcing?

no indirect forcing (need to put Steve Ghan's routines)

>>
>> Is this instantaneous output (for 15 Jan), or a monthly mean?

10-yr average

>> How long was the spin-up?

7 years

>> Which aerosols contributed to the optical depth?

sulfate, organic carbon, black carbon, sea-salt and ammonium nitrate.
No dust.

>> Is the optical depth for the shortwave or longwave?

it is for 0.6 micron.

>> I have just started crafting the presentation. I plan to send you a draft
>> late tomorrow for your comment. Will that work for you? If you do have
>> other plots can you send them to me sooner, so that I can incorporate them
>> into the draft I send you?

Here is ozone at the surface.  Units are pptv (ignore
the units in the title), same averaging as
aod.

Jean-Francois

## Subgrid OrographySteve Ghan and Tim Shippert

## Subgrid scheme

## Progress on Subgrid Orography

- Scheme applied to CAM and CLM
- Euler dycore
- Finite-volume dycore (1-D and 2-D domain decomposition)
- Developmental branch updated to cam2_0_2_dev55 (IPCC physics)
- Bit-for-bit agreement between SP, SMP, SPMD
- Restarts bit-for-bit agreement
- Bit-for-bit agreement with dev trunk if subgrid scheme turned off
- Energy conservation demonstrated
- Load balancing within nodes for both dycores
- Load balancing between nodes for both dycores (FV 1-D only)
- Runoff distributed according to elevation of river transport model surface elevation
- Simulations at T42 and 2x2.5o resolution
- Starting IPCC downscaling simulations

## Downscaled Temperature

## March snow

## North American Glaciers

## River Discharge

## SWE and Model Development

## Community Atmospheric Model  Plans (May04)

- Continue Optimizations for IPCC
- Vector version for Cray X1 and NEC SX7  (June 2004)
- IBM p690 improvements
- Increased resolution for spectral cores
- SLT 2-D decompositions  October 2004
- SLD reduced grid   Dec 2004
- SLD lagrangian vertical coordinate June 2005
- SLD new advection (incremental remapping?) October 2005
- Toward a chemical atmosphere model
- Increased resolution (B ->D) for Lin-Rood Oct 2004
- WACCM configurations, chemistry, vectorization  Oct 2004
- Advection optimization for a Chemical atmosphere Dec 2004
- IPCC Simulations with subgrid orography Jan 2005
- T170 Eulerian and FV (D-grid) downscaling of selected IPCC runs
- Implications of SP-CAM
- Revision of software design and the Unified Model
- Block/ESMF dycores Dec 2004
- Utility layer standardization  March 2005
- Single executable CCSM (with NCAR) Completion date: October 2004.
- A prototype code is done.
- Integrating parallel netCDF into Ziolib  Complete: Nov 2004.
- Incorporating ZioLib into CAM:  Completion date: December 2004.

## Unified Model Architecture

***Issues:*****  **
**Model componentization**
**Utility layer**
**Extensibility**
**Performance **
**Single or multiple **
**  ****executables**
**Stand alone execution**
**Unit testing**

**Version 2.0.2 Released**
**11 July 2004**

**CPL6 -- Implemented, Tested, Deployed based on the Model Coupling Toolkit and the MPH3 multi-processor handshaking library**
**ESMF and CCA prototypes**
**Single Executable tests underway**

## Parallel I/OWoo-Sun Yang, Chris Ding

- Parallel decompositions of ZioLib
- Tested implementation in CAM

- Parallel performance of Ziolib

- Parallel NetCDF in conjunction with SciDAC SDM Center

## Ocean BiogeochemistryS. Elliot, S. Chu, M. Maltrud, D. Erickson

**Ocean ecology model additions to Doney-Moore-Lindsey **
**Implemented in POP2.0**
**Replace POP1.4.3 with POP2.0**
**Use to drive DMS flux with atmosphere coupling (other chemicals as well)**

**LANL Ecosystem Model**
**nutrients (nitrate, ammonium, iron, silicate)**
**phytoplankton (small, diatom, coccolithophores)**
**zooplankton**
**bacteria, dissolved organic material, detritus**
**dissolved inorganic carbon (DIC), alkalinity**
**trace gases (dimethyl sulfide, carbonyl sulfide, methyl halides and nonmethane hydrocarbons)**
**elemental cyclings (C,N,Fe,Si,S)**

## Atmospheric ChemistryP. Cameron-Smith, J.F. Lamarque, S. Walters, ... 

**Gas-phase chemistry with emissions, deposition, transport and photo-chemical reactions for 89 species.  **
**Experiments performed with 4x5 degree Fvcore – ozone concentration at 800hPa for selected stations (ppmv)**
**Mechanism development with IMPACT**
**A)    Small mechanism (TS4), using the ozone field it generates for photolysis rates.**
**B)     Small mechanism (TS4), using an ozone climatology for photolysis rates.**
**C)    Full mechanism (TS2), using the ozone field it generates for photolysis rates.**

- Zonal mean
- Ozone, Ratio A/C

- Zonal mean
- Ozone, Ratio B/C

## Polar and THC

## HYPOP

- Arbitrary Lagrangian-Eulerian vertical coordinate
- Keep Lagrangian in deep ocean
- Remap to z-coordinate in mixed layer
- CSU SciDAC
- New time stepping/mode splitting
- Progress
- Model currently working in z-coord mode
- Examining vertical grid generators
- Testing

## CICE

**Incremental Remapping for Sea Ice and Ocean Transport**** **
- Incremental remapping scheme that proved to be three times faster than MPDATA,  total model speedup of about 30% --added to CCSM/CSIM
- CICE3.0 restructered for vector  Community Sea Ice Model
**Sensitivity analysis and parameter tuning test of the CICE code **** **
- Automatic Differentiation (AD)-generated derivative code

## SciDAC DOE Milestones for FY05

- Q1: (LANL)
- Non-interacting ocean biogeochemistry (ecosystem + trace gases) model implemented in the current version of the Parallel Ocean Program  (POP) ocean GCM and executed as part of a coupled experiment within CCSM3 to simulate the response of ocean biogeochemistry to climate.
- Q2: (LLNL)
- Code implemented in CCSM3 coupler and all components to exchange carbon and sulfur species; modify radiation scheme in the atmospheric component of the model to include the effects of sulfur species.
- Q3: (ORNL)
- Initial testing of all model changes to include the sulfur cycle and the carbon cycle in coupled climate model simulations
- Q4: (ORNL)
- Coupled simulations of the climate system including the carbon and sulfur cycles (POP2/biogeochemistry, CAM3/sulfur+carbon) with an interactive land surface model component.

## Block Buster Sequel: SciDAC II is coming ...

## Component Model Architecture