# Presentation: 131007

## V. Balaji (vb@gfdl.gov), SGI/GFDL
First ESMF Community Meeting
                 Washington, 30 May 2002NASA/GSFC

*V. Balaji (vb@gfdl.gov), SGI/GFDL*
*First ESMF Community Meeting*
*                 **Washington, 30 May 2002****NASA******/GSFC***

***A High-Performance Framework for Earth Science Modeling and****** Data Assimilation ***

## Outline

**Background**
**ESMF Objectives and Scientific Benefits**
**ESMF Overview**
**Final Milestones**
**Development Plan**
**Beyond 2004: ESMF Evolution**

## Technological Trends

** ****In climate research and NWP... ****increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall coupled system**
**  **
**In computing technology... ****increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable computing architectures**

## Technological Trends

** ****In software design for broad communities...**** ****The open source community provided a viable approach to constructing software to meet diverse requirements through “open standards”.  The standards evolve through consultation and prototyping across the user community.**
**    ****“****Rough consensus and working code.”**** (IETF)**

## Community Response

**Modernization of modeling software**
- 	Abstraction of underlying hardware to provide uniform programming model across vector, uniprocessor and scalable architecturesDistributed development model characterized by many contributing authors; use of high-level language features for abstraction to facilitate development processModular design for interchangeable dynamical cores and physical parameterizations, development of community-wide standards for components
**Development of prototype frameworks**
**    **GFDL (FMS), NASA/GSFC (GEMS).
-   Other framework-ready packages: NCAR/NCEP (WRF), NCAR/DOE (MCT)
**The ESMF aims to unify and extend these efforts**

## Framework examples

**Atmosphere and ocean model grids.**	Shared the same model grid, fluxes passed by common block.	Under a framework, independent model grids are connected by a coupler.
**Parallel implementation of legacy code.**	Take the old out-of-core solver, replace write-to-disk, read-from-disk with shmem_put, shmem_get.
- 		Under a framework, there is a uniform way to specify and fulfil data dependencies through high-level calls.

## The ESMF Project

**The need to unify and extend current frameworks into a community standard achieves wide currency.**
**NASA offers to underwrite the development of a community framework.**
**A broad cross-section of the community meets and agrees to develop a concerted response to NASA, uniting coupled models and data assimilation in a common framework.**
**Funding began February 2002: $10 million over three years.**

## Project Organization

- NASA DAO PI

**Data Assimilation**
**Deployment**

- Part III Proposal Specific Milestones

**Part III**

**Prognostic Model**
**Deployment**

- MIT PI

- Part II Proposal Specific Milestones

**Part II**

***Project Organization***

**NASA ESTO/CT**

**Joint Specification Team**
- Requirements Analysis
- System Architecture
- API Specification

## Design Principles

- Modularity data-hiding, encapsulation, self-sufficiency;
- Portability adhere to official language standards, use community-standard software packages, comply with internal standards
- Performance minimize abstraction penalties of using a framework
- Flexibility address a wide variety of climate issues by configuring particular models out of a wide choice of available components and modules
- Extensibility design to anticipate and accommodate future needs
- Community encourage users to contribute components, develop in open source environment

## Application Architecture

**External Libraries**

**Low Level Utilities**

**Fields and Grids Layer**

**Model Layer**

**Coupling Layer**

**ESMF Infrastructure**

**User Code**

**ESMF Superstructure**

**BLAS, MPI, netCDF, ...**

## Sample call structure

**ESMF Infrastructure**
**provides halo update, I/O**

**User Code**

**Initialize ESMF**
**initialize atmos, ocean**
**get grids for atmos, ocean**
**initialize regrid**
**time loop**

**Atmos and ocean models**
**must provide return_grid**
**call ESMF halo update, I/O etc**

## Framework Architecture

| Components and Coupling |  | gridded component interfacecollective data transfers/couplingcollective I/O |
| --- | --- | --- |
| Fields and Grids | Fields | field metadatafield and field bundle datafield I/O |
|  | Grids | grid metadatagrid decomposition |
|  | Parallel Utilities | transpose, halo, etc.abstract machine layout |
| Low-Level Utilities |  | event alarmsperformance profilingI/O primitivescommunication primitives, etc. |

## An MPMD ocean and atmosphere register their presence (“call ESMF_Init”). A separate coupler queries them for inputs and outputs, and verifies the match.
“In an 80p coupled model, assign 32p to the atmosphere component, running concurrently with the ocean on the other 48. Run the land model on the atmos PEs, parallelizing river-routing across basins, and dynamic vegetation on whatever’s left.”
Possible extension of functionality: “Get me CO2 from the ocean. If the ocean component doesn’t provide it, read this file instead.” 

- “In an 80p coupled model, assign 32p to the atmosphere component, running concurrently with the ocean on the other 48. Run the land model on the atmos PEs, parallelizing river-routing across basins, and dynamic vegetation on whatever’s left.”
- Possible extension of functionality: “Get me CO2 from the ocean. If the ocean component doesn’t provide it, read this file instead.”

***Superstructure: CONTROL***

## “How many processors are you running on?”
“What surface fields are you willing to provide?”
“Here are the inputs you requested. Integrate forward for 3 hours and send me back your surface state, accumulated at the resolution of your timestep.”
Possible extension of functionality: define a standard “ESMF_atmos” datatype.

- “What surface fields are you willing to provide?”
- “Here are the inputs you requested. Integrate forward for 3 hours and send me back your surface state, accumulated at the resolution of your timestep.”
- Possible extension of functionality: define a standard “ESMF_atmos” datatype.

***Superstructure: COUPLER***

## Non-blocking halo update:						call halo_wait()						...								call halo_update()
Bundle data arrays for aggregate data transfer.
Redistribute data arrays on a different decomposition.
Possible extension of functionality: 3D data decomposition.

- Bundle data arrays for aggregate data transfer.
- Redistribute data arrays on a different decomposition.
- Possible extension of functionality: 3D data decomposition.

***Infrastructure: GRID***

## Given this function for the grid curvature, generate all the metrics (dx, dy, etc) needed for dynamics on a C-grid.
Refine an existing grid.
Support for various grid types (cubed-sphere, tripolar, ...)
Possible extension of functionality: Standard differential operators (e.g grad(phi), div(u) )

- Refine an existing grid.
- Support for various grid types (cubed-sphere, tripolar, ...)
- Possible extension of functionality: Standard differential operators (e.g grad(phi), div(u) )

***Infrastructure: GRID***

## ESMF will set a standard for writing grid overlap information between generalized curvilinear grids.
Clip cells as needed to align boundaries on non-aligned grids (e.g coastlines).
Support for various grid types (cubed-sphere, tripolar, ...)
Possible extension of functionality: efficient runtime overlap generation for adaptive meshes.

- Clip cells as needed to align boundaries on non-aligned grids (e.g coastlines).
- Support for various grid types (cubed-sphere, tripolar, ...)
- Possible extension of functionality: efficient runtime overlap generation for adaptive meshes.

***Infrastructure: REGRID***

## 
ESMF will be usable by models written in F90/C/C++.
ESMF will be usable by models requiring adjoint capability.
ESMF will be usable by models requiring shared or distributed memory parallelism semantics.
ESMF will support SPMD and MPMD coupling.
ESMF will support several I/O formats (including GRIB/BUFR, netCDF, HDF).
ESMF will have uniform syntax across platforms.

- ESMF will be usable by models written in F90/C/C++.
- ESMF will be usable by models requiring adjoint capability.
- ESMF will be usable by models requiring shared or distributed memory parallelism semantics.
- ESMF will support SPMD and MPMD coupling.
- ESMF will support several I/O formats (including GRIB/BUFR, netCDF, HDF).
- ESMF will have uniform syntax across platforms.

***General features***

## Building blocks

- ESMF Infrastructure
- Distributed grid operations (transpose, halo, etc.)
- Physical grid specification, metric operations.
- Regridding: interpolation of data between grids, ungridded data.
- Fields: association of metadata with data arrays. Loose and packed field bundles.
- I/O: on distributed data
- Management of distributed memory, data-sharing for shared and distributed memory.
- Time management, alarms, time and calendar utilities
- Performance profiling and logging, adaptive load-balancing.
- Error handling

- ESMF Superstructure
- Control: assignment of components to processor sets, scheduling of components and inter-component exchange. Inter-component signals, including checkpointing of complete model configurations.
- Couplers and gridded components: Validation of exchange packets. Blocking and non-blocking transfer of boundary data between component models. Conservation verification. Specification of required interfaces for components.

## Target Platforms

**ESMF will target broad range of platforms**
- Major center hardware, *e.g**.*
**SP, SGI O3K, Alpha**
**1000+ processors**
** **Commodity hardware, *e.g.*
**Linux clusters, desktops**

## Joint Milestone Codeset I

| ID | Part I JMC: EVA Suite |
| --- | --- |
| a | spectral simulation at T42 |
| b | spectral simulation at T170 |
| c | gridpoint simulation at 1/4° x 1/4° or equivalent |
| d | component based on a physical domain other than the ocean or atmosphere, 2° x 2.5° or equivalent |
| e | simplified 3D-VAR system with 200K observations/day |
| fc | synthetic coupled SPMD system |
| gc | synthetic coupled MPMD system  |

## Joint Milestone Codeset II

| Source | ID | Part II JMC: Modeling Applications |
| --- | --- | --- |
| GFDL | h | FMS B-grid atmosphere at N45L18 |
|  | i | FMS spectral atmosphere at T63L18 |
|  | j | FMS MOM4 ocean model at 2°x2°xL40 |
|  | k | FMS HIM isopycnal C-language ocean model at 1/6°x1/6°L22 |
| MIT | lc | MITgcm coupled atmosphere/ocean at 2.8°x2.8°, atmosphere L5, ocean L15 |
|  | m | MITgcm regional and global ocean at 15kmL30 |
| NSIPP | nc | NSIPP atmospheric GCM at 2°x2.5°xL34 coupled with NSIPP ocean GCM at 2/3°x1.25°L20 |
| NCAR/LANL | oc | CCSM2 including CAM with Eulerian spectral dynamics and CLM at T42L26 coupled with POP ocean and data ice model at 1°x1°L40 |

## Joint Milestone Codeset III

| Source | ID | Part III JMC: Data Assimilation Applications |
| --- | --- | --- |
| DAO | p | PSAS based analysis system with 2O0K observations/day |
|  | qc | CAM with finite volume dynamics at 2°x2.5°L55, including CLM |
| NCEP | r | Global atmospheric spectral model at T170L42 |
|  | s | SSI analysis system with 250K observations/day, 2 tracers |
|  | t | WRF regional atmospheric model at 22km resolution CONUS forecast 345x569L50 |
| NSIPP | uc | ODAS with OI analysis system at 1.25°x1.25°L20 resolution with ~10K observations/day |
| MIT | v | MITgcm 2.8° century / millennium adjoint sensitivity |

## Interoperability Demo

|  | MODEL | MODEL  | SCIENCE IMPACT |  |
| --- | --- | --- | --- | --- |
| 1 | GFDL FMS B-grid atm | MITgcm ocean | Global biogeochemistry (CO2, O2), SI timescales. |  |
| 2 | GFDL FMS MOM4 | NCEP forecast | NCEP seasonal forecasting system. |  |
| 3 | NSIPP ocean | LANL CICE | Sea ice model for SI, allows extension of SI system to centennial time scales. |  |
| 4 | NSIPP atm | DAO analysis | Assimilated initial state for SI. |  |
| 5 | DAO analysis | NCEP model | Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation. |  |
| 6 | DAO CAM-fv | NCEP analysis | Intercomparison of systems for NASA/NOAA joint center for satellite data assimilation. |  |
| 7 | NCAR CAM Eul | MITgcm ocean | Improved climate predictive capability: climate sensitivity to large component interchange, optimized initial conditions. |  |
| 8 | NCEP WRF | GFDL MOM4 
 | Development of hurricane prediction capability. |  |

-    3 interoperability experiments completed in 2004, 5 by 2005.

## Beyond 2004:ESMF Evolution

- Maintenance, support and management
- NCAR commitment to maintain and support core ESMF software
- Seeking inter-agency commitment to develop ESMF
- Technical evolution
- Functional extension:
- Support for advanced data assimilation algorithms: error covariance operators, infrastructure for generic variational algorithms, etc.
- additional grids,  new domains
- Earth System Modeling Environment, including web/GUI interface, databases of components and experiments, standard diagnostic fields, standard component interfaces.