# Presentation: 692176

## Introduction to the Earth System Modeling Framework

**C. DeLuca/NCAR, J. Anderson/NCAR, V. Balaji/GFDL, B. Boville/NCAR, N. Collins/NCAR, ****T. Craig/NCAR, C. Cruz/GSFC, A. da Silva/GSFC, R. Hallberg/GFDL, C. Hill/MIT, M. Iredell/NCEP, ****R. Jacob/ANL, P. Jones/LANL, B. Kauffman/NCAR, J. Larson/ANL, J. Michalakes/NCAR, ****E. Schwab/NCAR, S. Smithline/GFDL, Q. Stout/U Mich, M. Suarez/GSFC, A. Trayanov/GSFC, ****S. Vasquez/NCAR, J. Wolfe/NCAR, W. Yang/NCEP, M. Young/NCEP and L. Zaslavsky/GSFC**

**Introduction to the Earth System Modeling Framework**

**NSIPP Seasonal Forecast**

**NCAR/LANL CCSM**

**NCEP Forecast**

**GFDL FMS Suite**

**MITgcm**

**NASA GSFC PSAS**

***Climate***

***Data ******Assimilation***

***Weather***

## Outline

**ESMF Overview**
**ESMF Applications**
**Related Projects**
**ESMF Architecture**
**Timeline**

## Background

**Background**

**NASA’s Earth Science Technology Office proposed the creation of an****Earth System Modeling Framework (ESMF)**** in the September 2000 NASA Cooperative Agreement Notice:**
**“****Increasing Interoperability and Performance of Grand Challenge ****Applications in the Earth, Space, Life and Microgravity Sciences”**
**A large, interagency collaboration with roots in the Common Modeling Infrastructure Working Group proposed three interlinked projects to develop and deploy the ESMF, which were all funded:**
**Part I:  Core ESMF Development (PI:  Killeen, NCAR)**
**Part II:  Modeling Applications (PI:  Marshall, MIT)**
**Part III:  Data Assimilation Applications (PI:  da Silva, NASA GMAO)**

## Motivation 

**Motivation **

**In climate research and NWP... ****increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system**
**In computing technology... ****increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable computing architectures **
**In software ...****development of frameworks, such as FMS, GEMS, CCA and WRF, that encourage software reuse and interoperability**
**The ESMF is a focused community effort to tame the complexity of models and **
**the computing environment.  It leverages, unifies and extends existing software**
**frameworks, creating new opportunities for scientific contribution and collaboration.**

## ESMF Project Description

**GOALS:**** To increase software reuse, interoperability, ease of use and performance portability in climate, weather, and data assimilation applications**
**PRODUCTS:****  **
**Core framework:  Software for coupling geophysical components and utilities for building components **
**Applications:  Deployment of the ESMF in 15 of the nation’s leading climate and weather models, assembly of 8 new science-motivated applications**
**METRICS:**
**RESOURCES and TIMELINE:**** $9.8M over 3 years                **

| Reuse | Interoperability | Ease of Adoption | Performance  |
| --- | --- | --- | --- |
| 15 applications use ESMF component coupling services and 3+ utilities | 8 new applications comprised of never-before coupled components | 2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours | No more than 10% overhead in time to solution, no degradation in scaling |

## ESMF Accelerates Advances in Earth System Science

**Eliminates software barriers to collaboration among organizations**
**Easy exchange of model components accelerates progress in NWP and climate modeling**
**Independently developed models and data assimilation methods can be combined and tested**
**Coupled model development becomes a truly distributed process**
**Advances from smaller academic groups easily adopted by large modeling centers**
**Facilitates development of new interdisciplinary collaborations**
**Simplifies extension of climate models to upper atmosphere**
**Accelerates inclusion of advanced biogeochemical components into climate models**
**Develops clear path for many other communities to use, improve, and extend climate models**
**Many new model components gain easy access to power of data assimilation**

## ESMF Collaborators

**NSF NCAR*****Tim Killeen, PI*****Jeff Anderson***** *****Byron Boville****Nancy Collins ****Cecelia DeLuca****Roberta Johnson****Al Kellie****John Michalakes****David Neckels****Earl Schwab ****Robbie Staufer****Silverio Vasquez ****Jon Wolfe****DOE ANL****Rob Jacob****Jay Larson**

**NOAA GFDL****Ants Leetmaa****V. Balaji****Robert  Hallberg****Shep Smithline**
**NASA GMAO*****Arlindo da Silva, PI*****Michele Rienecker****Max Suarez****Atanas Trayanov****Christian Keppenne****Leonid Zaslavsky****Will Sawyer****Carlos Cruz**

**NOAA NCEP****Stephen Lord****Mark Iredell****Mike Young****Weiyu Yang****John Derber**
**MIT*****John Marshall, PI*****Chris Hill**
**DOE LANL**
**Phil Jones**
**University of Michigan****Quentin Stout**

## Outline

**ESMF Overview**
**ESMF Applications**
**Related Projects**
**ESMF Architecture**
**Timeline**

## Modeling Applications

| SOURCE | APPLICATION |
| --- | --- |
| GFDL | FMS B-grid atmosphere at N45L18 |
|  | FMS spectral atmosphere at T63L18 |
|  | FMS MOM4 ocean model at 2°x2°xL40 |
|  | FMS HIM isopycnal C-language ocean model at 1/6°x1/6°L22 |
| MIT | MITgcm coupled atmosphere/ocean at 2.8°x2.8°, atmosphere L5, ocean L15 |
|  | MITgcm regional and global ocean at 15kmL30 |
| GMAO/NSIPP | NSIPP atmospheric GCM at 2°x2.5°xL34 coupled with NSIPP ocean GCM at 2/3°x1.25°L20 |
| NCAR/LANL | CCSM2 including CAM with Eulerian spectral dynamics and CLM at T42L26 coupled with POP ocean and data ice model at 1°x1°L40 |

## Data Assimilation Applications

| SOURCE | APPLICATION |
| --- | --- |
| GMAO/DAO | PSAS interface layer with 2O0K observations/day |
|  | CAM with finite volume dynamics at 2°x2.5°L55, including CLM |
| NCEP | Global atmospheric spectral model at T170L42 |
|  | SSI analysis system with 250K observations/day, 2 tracers |
|  | WRF regional atmospheric model at 22km resolution CONUS forecast 345x569L50 |
| GMAO/NSIPP | ODAS with OI analysis system at 1.25°x1.25°L20 resolution with ~10K observations/day |
| MIT | MITgcm 2.8° century / millennium adjoint sensitivity |

## Interoperability Experiments:8 New Applications

| COUPLED CONFIGURATION | NEW SCIENCE ENABLED |
| --- | --- |
| GFDL B-grid atm / MITgcm ocn | Global biogeochemistry (CO2, O2), SI timescales. |
| GFDL MOM4 / NCEP forecast | NCEP seasonal forecasting system. |
| NSIPP ocean / LANL CICE | Sea ice model for extension of SI system to centennial time scales. |
| NSIPP atm / DAO analysis | Assimilated initial state for SI. |
| DAO analysis / NCEP model | Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation. |
| DAO CAM-fv / NCEP analysis | Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation. |
| NCAR CAM Eul / MITgcm ocn | Improved climate predictive capability: climate sensitivity to large component interchange, optimized initial conditions. |
| NCEP WRF / GFDL MOM4 | Development of hurricane prediction capability. |

## Outline

**ESMF Overview**
**ESMF Applications**
**Related Projects**
**ESMF Architecture**
**Timeline**

## Related Projects

**PRISM ****is an ongoing European Earth system modeling infrastructure project**
**Involves current state-of-the-art atmosphere, ocean, sea-ice, atmospheric chemistry, land-surface and ocean-biogeochemistry models**
**22 partners: leading climate researchers and computer vendors, includes MPI, KNMI, UK Met Office, CERFACS, ECMWF, DMI**
**ESMF is working with PRISM to merge****frameworks and develop common conventions**

**CCA**** is creating a minimal interface and sets of tools for linking high performance components.  CCA can be used to implement frameworks and standards developed in specific domains (such as ESMF).**
**Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of Tennessee, and many more. Ongoing ESMF collaboration with CCA/LANL on language interoperability.**
**Working prototype demonstrating CCA/ESMF interoperability, to be presented at SC2003.**

***For joint use with PRISM, ESMF***
***developed a component database***
***to store component import/export ******fields and component descriptions***

## ESMF Connections

**CCA**

**WRF**

**SciDAC**

**CCSM**

**GEMS**

**FMS**

**SWMF**

**PRISM**

**ESMF**

**ESMF**** Earth System Modeling Framework**
**CCA**** DOE Common Component Architecture**
**SciDAC**** DOE/NSF CCSM SciDAC Project**
**GEMS**** Goddard Earth Modeling System**
**FMS**** GFDL Flexible Modeling System**
**SWMF**** ****Space Weather Modeling Framework**
**WRF**** Weather Research and Forecast Model**
**CCSM**** ****Community Climate System Model**
**PRISM**** Program  for Int. Earth System Modeling**

## Outline

**ESMF Overview**
**ESMF Applications**
**Related Projects**
**ESMF Architecture**
**Timeline**

## Computational Characteristicsof Weather/Climate

**Computational Characteristics****of Weather/Climate**

**Mix of global transforms and local communications**
**Load balancing for diurnal cycle, event (e.g. storm) tracking**
**Applications typically require 10s of GFLOPS, ****100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs**
**Required Unix/Linux platforms span laptop to ****Earth Simulator**
**Multi-component applications:  component ****hierarchies, ensembles, and exchanges**
**Data and grid transformations between ****components **
**Applications may be MPMD/SPMD, ****concurrent/sequential, combinations **
**Parallelization via MPI, OpenMP, shmem, combinations**
**Large applications (typically 100,000+ lines of source code)**

***Platforms***

**assim**

**sea ice**

**ocean**

**land**

**atm**

**physics**

**dycore**

**assim_atm**

**atmland**

***Seasonal Forecast***

***coupler***

## Architecture

**Low Level Utilities**

**Fields and Grids Layer**

**Model Layer**

**Components Layer:**
**Gridded Components**
**Coupler Components**

**ESMF Infrastructure**

**User Code**

**ESMF Superstructure**

**BLAS, MPI, NetCDF, ...**

**External Libraries**

**ESMF provides an environment for assembling geophysical components into applications.**
**ESMF provides a toolkit that ****components use to**
**increase interoperability**
**improve performance ****portability**
**abstract common services**

***Becoming an ESMF Component***
**Pack model import and export data into ESMF data structures and conform to a standard calendar.  Use ESMF utilities internally as desired.  Organize model using standard ESMF methods:  Initialize, Run, Finalize, ReadRestart, WriteRestart.  Methods may be multi-phase (Run phase=1, Run phase=2).  Method interfaces are prescribed.**
**Instantiate an ESMF Component with name, type, config information.  Register standard model methods with Component.  If desired, register data.**
**Use ESMF AppDriver to sequence and run Components.**
** **

## Design Features

**ESMF enables modeling applications to be:**
**Scalable**** ****Models are built from modular components, and can ****be easily nested within larger applications**
**Performance - portable**** ****ESMF high-performance communication libraries offer a consistent interface across computer architectures**
**Exchangeable**** ****Standard component interfaces enable interoperability **

## ESMF Class Structure

**ESMF Class Structure**

**DELayout**
- Communications

**State**
- Data imported or exported

**Bundle**
- Collection of fields

**GridComp**
- Land, ocean, atm, ... model

***F90***

***Superstructure***

***Infrastructure***

**Field**
- Physical field, e.g. pressure

**Grid**
- LogRect, Unstruct, etc.

***Data***

***Communications***

***C++***

**Regrid**
- Computes interp weights

**CplComp**
- Xfers between GridComps

**Utilities**
- Machine, TimeMgr, LogErr, I/O, Config, Base etc.

**Array**
- Hybrid F90/C++ arrays

**Route**
- Stores comm paths

**DistGrid**
- Grid decomposition

**PhysGrid**
- Math description

## Design Principle:Local Communication

**All inter-Component communication within ESMF is local - all communication is handled within Components.  This allows the architecture of the framework to be independent of the communication strategy.**
** ****	**

**climate_comp**

**ocn_comp**

**atm_comp**

**atm_phys**

**phys2dyn_coupler**

**atm_dyn**

**PE**

**atm2ocn _coupler**

**As a consequence, Coupler Components must be defined on the union of the PEs of all the Components that they couple.**
**In this example, in order to send data from the atmosphere Component to the ocean, the atm2ocn Coupler mediates the send.**

## Design Principle:Scalable Applications

**Since each ESMF application is also a Component, entire ESMF applications can be treated as Gridded Components and nested within larger applications.**
**	**
** ****	**

**climate_comp**

**ocn_comp**

**atm_comp**

**ocn2atm_coupler**

**atm_phys**

**phys2dyn_coupler**

**atm_dyn**

**PE**

**Example: atmospheric application itself composed of multiple Components may be run standalone, or nested within a larger climate application**
** **

## Design Principle:  Modularity

**Design Principle:  Modularity**

**Gridded Components don’t have access to the internals of other Gridded Components, and don’t store any coupling information.  Gridded Components may:**
**pass their States to other Components through their argument list or**
**receive user-defined methods through their argument list for transforming and transferring States.  These are called Transforms, and they contain a function pointer and attributes describing frequency and validity criteria.  Transforms may modify the data in States, receive States from other Components, send States to other Components, etc. **

**! From climate comp**
**call ESMF_CompRun(atm, atm_xform)**
**call ESMF_CompRun(ocn, ocn_xform)**
**! From atm comp**
**call ESMF_StateXform(atm_ex_state, &****                     atm_xform)**
**! From ocn comp**
**call ESMF_StateXform(ocn_im_state, &**
**	****	****    ****ocn_xform)**

***EX 1: One-way coupling from atm to ocn***
**atm_xform is a send; ocn_xform a receive**

***EX 2: Standalone atm***
**atm_xform is a no-op**

**! From atm comp**
**call ESMF_CompRun(ocn, ocn_xform)**
**call ESMF_StateXform(atm_ex_state, &****	****	****    atm_xform)**

## Design Principle:Uniform Communication API

**Design Principle:****Uniform Communication API**

**The same programming interface is used for shared memory, distributed memory, and combinations thereof.**
**Machine model**** = abstraction of machine architecture (num_nodes, num_pes_per_node, etc.)**
**DE**** = a decomposition element - may be virtual, thread, MPI process**
**DELayout**** = an arrangement of DEs, in which dimensions requiring faster communication may ****be specified and resources arranged accordingly **

**DELayout: 4 x 3, ****ESMF_COMM_SHR****  in x and ****ESMF_COMM_MP **** in y**
**The data in a Grid is decomposed according to the number and topology of DEs in the DELayout**** **
**	**

**Performance of ESMF AllGather vs. raw MPI**

## Outline

**ESMF Overview**
**ESMF Applications**
**Related Projects**
**ESMF Architecture**
**Timeline**

## Time Line

**Time Line**

| May 2002 | Draft Developer’s Guide and Requirements Document completed
1st Community Requirements Meeting and review held in D.C. |
| --- | --- |
| July 2002 | ESMF VAlidation (EVA) suite assembled  |
| August 2002 | Architecture Document:  major classes and their relationships
Implementation Report:  language strategy and programming model
Software Build and Test Plan:  sequencing and validation |
| May 2003 | ESMF Version 1.0 release, 2nd Community Meeting at GFDL |
| November 2003 | First 3 interoperability experiments completed |
| April 2004 | Second API and production software release, 3rd Community Meeting |
| November 2004 | All interoperability experiments complete; all testbed applications compliant |
| January 2005 | Final delivery of source code and documentation |

## More Information

**ESMF website:****  ****http://www.esmf.ucar.edu**

***Acknowledgements***
**The ESMF is sponsored by the NASA **
**Goddard Earth Science Technology Office.**