# Presentation: 877643

## Status Review of NOAA Implementation of ESMF Architecture (i.e. NEMS) Mark IredellMarch 26 (#1-12) & April 2 (#12-32), 2008

- NEMS People
- NEMS Projects
- NEMS Issues

## NEMS People

- Tom Black	_Atm_, NAM
- Huiya Chuang	Post
- Ed Colon	Infrastructure
- Mike Ek	Land
- Jim Geiger	Land (NASA)
- Bob Grumbine	Sea ice
- Henry Juang	Atm, _Dyn_, GFS
- Sujay Kumar	Land (NASA)
- Sarah Lu	AQ
- Avichal Mehra	Ocean

- S. Moorthi	Atm, _Phy_, GFS
- Ken Mitchell	Land
- Christa Peters	Land (NASA)
- Youhua Tang	AQ
- Hendrik Tolman	Wave
- Ratko Vasic	Infrastructure
- Jun Wang	Infrastructure
- Weiyu Yang	Infrastructure
- Mike Young	Infrastructure
- Shujia Zhou	Land (NASA)
- Yanqiu Zhu	GSI

- _Biweekly UMIG (Unified Modeling Infrastructure Group) meeting_

## NEMS People

- V. Balaji	GFDL
- Brian Gross	GFDL
- Niki Zadeh	GFDL

- Stan Benjamin	GSD
- Tom Henderson	GSD
- Jin Lee	GSD
- Jacques Middlecoff	GSD

- _Monthly UMIT (Unified Modeling Infrastructure Telecon) meeting_
*Includes local UMIG group plus...*

*Occasionally the telecon invites a much wider group, including DTC and Navy.*

## NEMS Projects

- ESMF
- NEMS atmosphere
- Write history and Post processor
- Nesting
- Aerosols and Chemistry
- Land
- Ocean, waves and sea ice
- Ionosphere
- Ensemble
- Data assimilation

## ESMF

- Earth System Modeling Framework
- Community effort, partially supported by NOAA
- Proper ESMF Superstructure required for all NEMS components
- ESMF Infrastructure optional
- NEMS will require ESMF 3.1.0r at this time

## An ESMF “Component”

- An ESMF Component

**“****Parent” Driver (User Code)**

** ****SetServices()**

** ****Initialize()**

** ****Run()**

** ****Finalize()**

**   ****ESMF library code**

** ****Run()**

- Import State

- Export State

- Internal State

- Data flow in an ESMF Component Run method

## ESMF Component rules

- Only SetServices is public; Initialize, Run, and Finalize are private but exposed by SetServices.
- Import State is input; Export State is output. Both should be fully described using ESMF metadata (names, grid, decomposition, etc).
- Coupler Component likely has Import and Export States on different grids. A Coupler Component is user code.
- Internal State is private to the component but persists across calls to Initialize, Run, and Finalize. Even other instances of the same component will have their own private internal state. An access exception may be made to other “friendly” components.

## ESMF Infrastructure capabilities

- 	ESMF offers or will offer capabilities for decomposed halo update and transposes as well as extensive parallel regridding tools (among other capabilities). The user is free to make use of these or not. Thus far NEMS has taken advantage of existing in-house capability and has minimally made use these ESMF capabilities. This is expected in such transition projects, and ESMF capability will be used as needed.

## ESMF Portability, Conventions

- ESMF is designed to run on most systems Earth modeling runs on. Plus, ESMF infrastructure will support some ordinary functions that otherwise often hinder portability.
- ESMF supports but does not require the Climate-Forecast metadata conventions.Should NEMS adopt them?
- Pro: wide community uses it, including NOAA labs
- Con: very long names, but we can make local aliases

## CF convention examples fromhttp://cf-pcmdi.llnl.gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping

**CF convention examples from**http://cf-pcmdi.llnl.gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping

## NEMS Component Library

- Create a NEMS component library (in SVN).
- Each component will have some documentation. **Moreover, each component should have a sample MAIN to run it in stand-alone mode for testing.** For instance, Dynamics components may have Held-Suarez, Physics may have a single-column model.
- Each type of component will have an assigned NCEP librarian to coordinate the sub-library.

## NEMS Atmosphere

- Atmosphere

- Dynamics

- Physics

- Dyn-Phy
- Coupler

- NMM-B

- Spectral

- FIM

**Color Key**

- Component class

- Coupler class

- Completed Instance

- Under Development

- NAM Phy

- GFS Phy

- Simple

- unified atmosphere
- including digital filter

- Future Development

-  The goal is one unified atmospheric component that can invoke multiple dynamics and physics.
-  At this time, dynamics and physics run on the same grid in the same decomposition, so the coupler literally is very simple.

- ARW

- FVCORE

- FISL

- NOGAPS

- WRF Phy

- Navy Phy

- COAMPS

- Regrid,
- Redist,
- Chgvar,
- Avg, etc

## Original FIM run() Call Order

- FIM run:
**do timestep = first, last**
** *****dyn_1*****  ! 1****st**** half of dyn calls**
** ****call physics()**
** *****dyn_2*****  ! 2****nd**** half of dyn calls**
**end do**
- Must re-order operations without changing model results

*Slide courtesy Stan Benjamin*

## FIM run() Call Re-Ordering

- Re-order time stepping loop:
**do timestep = first, last+1**
** ****if (timestep > first) &**
**   ****dyn_2(***timestep-1***)**
** ****if (timestep <= last) then**
**  ****dyn_1(***timestep***)**
**  ****call physics(***timestep***)**
** ****endif**
**end do**

*Slide courtesy Stan Benjamin*

## New FIM run() Call Order

- Combine dyn_1 + dyn_2 into dyn_run and push “if” statements inside
- Push physics() and if statement into phy_run
**do timestep = first, last+1**
** ****call dyn_run()**
** ****call phy_run()**
**end do**

*Slide courtesy Stan Benjamin*

*MI: Note NMM and GFS have gone through the exact same process.*

## Write history and Post processor

- Atmosphere

- Dynamics

- Physics

**Color Key**

- Completed Instance

- Under Development

- Future Development

- Write

- Binary file

- NEMSIO

- Post

- Write

- Write

-  The Write component transfers state from model tasks to “quilt” tasks and then writes out from there.
-  NEMSIO is intended to be a unified optimized parallelized I/O package that can write several formats. Ideally, NEMSIO would be used for all NEMS output data and metadata (not necessarily restart data).
-  The unified post-processor NCEP_POST will run on the quilt tasks.
-  Perhaps in the future, the Dynamics and Physics will write their own history files.

- unified atmosphere

- Component class

- Coupler class

## Nesting

- Atmosphere

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Parent creates children and creates proper boundary conditions in their import states.
-  Recursively, children create grandchildren.
-  Children run on *different* tasks from parent in *concurrent* nesting, *same* tasks in *sequential* nesting.
-  Two-way concurrent nesting would require a different time integration scheme that needs to be tested.

- Atmosphere

- 1-way conc. NAM-NAM

- 1-way conc. GFS-NAM

- 2-way seq.

- 2-way conc.

- moving nest

- Atmosphere

- Atmosphere

- Atmosphere

- Atmosphere

- Component class

- Coupler class

## •   Eventually we will have all forms.  

- •   Eventually we will have all forms.

- ► The parent domain can have any number of children.

- •   Begin with 1-way grid-associated static nests.

- ► Telescoping:  Children can have any number of children.

- ► Domains can run concurrently on unique sets of processors.

*These three criteria require the need for general and*
*repeated splitting of the MPI Communicators.*  --*Done*

- ► One-way / Two-way.

- ► Static / Moving.

- ► Grid-associated / Not grid-associated.

*Slide courtesy Tom Black*

## Nesting – ATM_RUN

- 	DO Timeloop over Fcst_time at interval Δtinternal
- 		 IF(My_Compute_Tasks) PREDICT(Δtinternal)
- 		 IF N_CHILDREN>0
- 			 DO N= 1, N_CHILDREN
- 				 IF(My_Compute_Tasks) ISend BC’s to child
- 				 IF(Child_Compute_Tasks(N)) Recv from parent
- 				 CALL ATM_RUN(Child(N), Fcst_time=Δtinternal)
- 			ENDDO
- 			IF(Two-way) Send children’s state back to me and blend into mine
- 		ENDIF
- 	ENDDO
- END ATM_RUN

***Recursive call***

***“******Mini” timeloop ***

*Slide courtesy Tom Black*

## Aerosols and Chemistry

- Dynamics

- Physics

**Color Key**

- Completed Instance

- Under Development

- Future Development

- Aerosols

- GOCART

-  GOCART is Goddard aerosol model. CBM is VOC-NOX-O3 mechanism (chemistry in CMAQ model).
-  Aerosol and chemistry components compute sources, sinks, and chemical transformation.
-  Aerosol and chemistry components may need to expose internal state to avoid memory copies.
-  Atmosphere must be ready to do convective fluxes and removal as well as advection and diffusion.

-  CBM-5

-  reduced chem

- Atmosphere

-  WRF-chem

- Component class

- Coupler class

## Land (alternative 0)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Dynamics

- Physics

- Land

- LIS-Noah

- others

- Atmosphere

- Component class

- Coupler class

## Land (alternative 0b)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Dynamics

- Physics

- Land

- LIS-Noah

- others

- Atmosphere

- Component class

- Coupler class

- Main

## Land (alternative 1)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Component class

- Coupler class

## Land (alternative 1b)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Dynamics

- Radiation

- Land

- LIS-Noah

- others

- PBL/Moist

- Atmosphere

- Component class

- Coupler class

## Land (alternative 2)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Dynamics

- Atmosphere

- Radiation

- Land

- PBL/Moist

- Physics

- Aerosols/Chemistry

- Component class

- Coupler class

## Land (alternative 3)

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Land is responsible for computing surface fluxes.
-  Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.
-  Implicit solving would require iteration between Land and PBL/Moist.
-  How deep should ESMF go?

- Dynamics

- Atmosphere

- Radiation

- PBL/Moist

- Aerosols/Chemistry

- Land

- Ice

- Waves

- Ocean

- Physics

- Surface

- Component class

- Coupler class

## GEOS-5 AGCM COMPONENT STRUCTURE

- AGCM

- DYNAMICS

- GWD

- FVCORE

- RADIATION

- COLUMN PROCESSES

- SOLAR

- IR

- MOIST

- TURB

- SURFACE

- CHEM

- LAKE

- SALTWATER

- CAP

- (qv, ql, qi, cl,...)

- (u, v, T, p)

- (Tc, qc, Td,...)

- (Ts,Fi...)

**VEGDYN**

**CATCH**

- HISTORY

- ICE

- LAND

- (Ts,Fi...)

- (O3,...,Qdust,...)

- (Tskin,Hskin,...)

- (Ci,...)

- OGCM

*Slide courtesy Max Suarez*

## Ocean, waves and sea ice

- Atmosphere

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Under ESMF NEMS, there should be not much difference between SPMD and MPMD codes. However, the MPMD code might not be as portable. MPMD will allow more flexible optimizations though.
-  Due to some rapidly interacting physics (sea ice thermodynamics and wave roughness), some coupling would have to be frequent.

- Atm-Ocn
- Coupler

- Ocean

- GFS

- Atm-Ocn
- Coupler

- MOM4

**NCEP CFS non-ESMF MPMD**

- Waves

- GFS

- WW III

- MOM4

- NEMS-GFS

- HYCOM

- Ice

- H-ice

- N-ice

- Earth system model

- Component class

- Coupler class

## Ionosphere

- Atmosphere

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  GFS-IDEA is the GFS extended up to 600 km (from 60 km). GIP is Global Ionosphere-Plasmasphere
-  The coupling will be frequent and three-dimensional, providing optimization challenges.

- Atm-Ion
- Coupler

- GFS-IDEA

- GIP

- TBD

- Whole atmosphere

- Component class

- Coupler class

## Ensemble

- Atmosphere

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  The atmosphere ensemble coupler provides stochastic forcing for the ensemble.
-  The current non-NEMS GEFS capability will be converted to NEMS.

- Atm ens
- Coupler

- GFS

- NEMS-GFS

- Atmosphere

- Atmosphere

- NEMS-any

- Component class

- Coupler class

## Data assimilation

- Model

**Color Key**

- Completed Instance

- Under Development

- Future Development

-  Frequent coupling of atmosphere and analysis will be necessary as hourly analyses are required.
-  Whole Earth System Nested Ensemble may include any components in NEMS from previous slides.
-  4DVAR is supported in this scenario. The inner loop of the variational system could be entirely within the analysis component, or it could invoke adjoint methods with the NEMS atmosphere.
-  The model-analysis coupler may change grids and variables. When returning to the model state, only the analysis *increments* will be interpolated.

- Mdl-Anl
- Coupler

- Analysis

- GFS

- GSI

- NAM

- Whole Earth System Nested Ensemble

- Component class

- Coupler class

## NEMS Issues

- ESMF version
- NEMS will require ESMF 3.1.0r, which will be the latest public release in spring 2008. NEMS may later need ESMF 3.1.1.
- All components must use the same version
- MAPL
- GSFC ESMF wrapper
- Used by MOM4, GOCART, FVCORE
- Compatible with NEMS?
- Code Repositories
- Operationally secure
- Collaboration friendly
- Experiment Launcher compatible with operations
- Documentation and Support