# Presentation: 372795

## Overview of the SciDAC Project: Collaborative Design and Development of the CCSM for Terascale Computers

- PI: Malone(LANL), Drake(ORNL)
- Co-I (DOE): Ding(LBL), Duffy(LLNL), Erickson(ORNL),
-    Foster(ANL), Ghan(PNL), Jacob(ANL), Jones(LANL),
-    Larson(ANL), Mirin(LLNL), Rotman(LLNL), Taylor(ANL),
-    Worley(ORNL)
- Co-I (NCAR&NASA):  Bettge(NCAR), Kiehl(NCAR),
-    Craig(NCAR), Deluca(NCAR), Lin(DAO), Washington(NCAR),
-    Williamson(NCAR)

## Goals for SciDAC CCSM Collaborative Development

- Comprehensive treatment of physical and chemical processes
- High resolution ocean and atmosphere support
- Hybrid vertical coordinate in ocean code
- Tropospheric chemistry package
- Biogeochemistry
- Modular “packages” with well defined interfaces and testing procedures
- Atm, Coupler (Avant Garde), Ice, Ocn, Lnd
- Performance optimized yet portable and adaptable for utilization of emerging architectural features of terascale computers
- Ready for DOE, NSF and NASA applications
- High resolution historical and climate change scenario studies
- Carbon cycle studies and data assimilation for climate

## Climate Science Enabled

***Science***** **13April 2001: “Detection of Anthropogenic Climate Change in the Worlds Oceans,” Barnett, Pierce, Schnur

**Method:**  Ensemble simulations of the DOE Parallel Climate Model (PCM)

**Results:**
- Detection of Anthropogenic Climate Change in the Worlds Oceans
- Ensembles establish 95% confidence intervals of model predictions
- Simulated ocean heat storage matches historical record of rising ocean temperatures

**Enabling Technology:**
- Parallel Climate Model developed in collaborative effort lead by Warren Washington (NCAR)
- Terascale computing resources

**Firsts:**
-  Ensemble study with US model and computers
-  Coupled model reproducing ocean response
-  Establishing new level of US model quality

## Software Infrastructure and Applied Math Challenges

| Challenge | Existing Solutions | Enhanced Solutions |
| --- | --- | --- |
| Project management, Version control and testing infrastructure for competing, distributed development teams | CVS with monitoring procedures | BitKeeper? SourceForge?  Variety of systems and institutions.  |
| Evolving Three layer architecture:  Library(eg. FFT), Utility(eg. orbital calendar, data transpose) and Model ( eg. radiation physics) | F90 modules, vendor math libraries, customized data movement, PILGRM, MCT, ...Open design process | Optimized math libraries, performance monitors, CCA, NASA Earth System Modeling Framework(ESMF) |
| Accurate, fast dynamical methods:  atm, ocn, ice | Spectral horizontal, finite volume, conservative semi-Lagrangian advection, hybrid vertical coordinates, fast Helmholtz solvers, two-time level monotone advection | Scalable algorithms, grid technologies, nonlinear solvers, new discretization techniques, subcycling explicit barotropic modes, new formulations |
| Incorporating new model components and improving coupled model climate simulation | Control simulations, component working group activities, scientific steering and review of non-linear interactions | Available cycles, improved analysis capabilities and extensive data handling capabilities, collaborative technologies |

- Unknown source

- ISIC and external

- Internal to project

## Software Engineering Challenges

## Software Tasks

- Develop comprehensive design documents for each component of the model
- Implement performance-portable, run-time configurable CCSM on target machines
- Incorporate emerging programming paradigms and software design practices
- Develop testing and validation procedures for all component models
- Incorporate new dynamical components for ocean, atmosphere and sea ice.
- Incorporate new modular physical and chemical process models

## Task List Excerpt and Schedule

## Chemical solver technologies

- GHG with transport
- Interactive ozone : sulfur cycle and aerosols
- Upper ocean dimethyl sulfide
- Load balancing
- Off-line chemical simulations (DAM)
- Support C4MIP

- Multiple resolution atm model
- T31, T42, T85 and T170
- Sub-grid orography precip scheme
- Scalable dycores
- 2d, 3d blocks, MLP tests
- Optimized and load balancing physics

- Design documents
- Chunking for cache performance
- Load balanced, m to n transfers
- Land Surface Modeling Toolkit as part of utility layer

## Next Generation Couplers

- MCT version 2
- Higher level abstractions, component model registry
- Scalability to thousands of processors
- Dynamic load balancing
- 3d fields
- Interoperation of climate, weather and data assimilation functions
- MCT as CCA compliant prototype for Earth System Modeling Framework

## SciDAC CCSM – ISIC Collaboration Goals

- Improve performance characterizations
- Accelerate development of mathematical software for climate modeling and analysis
- Explore novel methods and advance the theory and simulation of geophysical flows
- Improve software practices for the scientific endeavor of climate change prediction
- Enhance analysis and data handling methods

## ISIC Projects

- Performance Evaluation(D. Bailey)           02Q1
- PCTM, CCSM Instrumentation - P. Worley (ORNL)
- Earth System Grid (Ian Foster)                  02Q3
- Data archive and analysis grid - Bernholdt(ORNL)
- Data Access (Arie Shoshani)                        02Q3
- Improved efficiencies and monitoring - Samatova (ORNL)

## ISIC Projects 

- Grids (Glimm, Brown)                        03Q1
- Ocean nested grids – Smith (LANL), Brown(LLNL)
- Smooth grid transformations – Drake(ORNL), Khamayseh(ORNL)
- Spectral element refinement strategies – Fournier (NCAR, U.Md), Fisher(ANL),Taylor&Wingate(LANL)
- Adaptive models – Joyce Penner (U. Mich)

## ISIC Projects

- PDE methods (Phil Collela)                      03Q1
- Adaptive diagnostics (Drake-Williamson)
- Static adapted western boundary currents Malone(LANL)
- Solvers (David Keyes)                              03Q4
- Baroclinic model (a new discretization)
- Krylov methods – Balou Nadiga (LANL)
- Common Component Architecture (Rob Armstrong)                                               04Q1
- Model Coupler Toolkit – Larson (ANL)
- CCA compliant CCSM

## SciDAC Climate Projects