# Presentation: 531436

## Software and Data Management Issues in Climate Modeling

- Robert Jacob

## Components of a Climate Model

- Global Atmosphere, Ocean and Sea Ice
- Treated as a geophysical fluid; each with their own numerical method for solving the “dynamics” (under time-varying external and internal forcing)
- Each contains a small (ocean) to large (atmosphere) collection of parameterizations to handle diabatic, adiabatic and sub-grid scale processes (radiation, convection, mixing, etc.)
- Land, Vegetation, Rivers
- Each a large collection of parameterizations (and maybe a 1-D diffusion model for heat)

## Climate Models vs Other Scientific Applications

- Collision of two black holes
- Supernova Explosion
- Car Crash
- Weather Forecast
- Protein Folding
- 	-----Simulate an event over a finite length of time
-             -----Final computed state is important

## Climate Models vs Other Scientific Applications

- Climate Models
- 	------Need hundreds of simulated years
- 	------Final computed state is irrelevant
- Why the difference?
- 	“Climate” is a statistical property:  Mean, variance of temperature, precipitation, etc.
- 	The Climate system has internal time scales of decades to centuries.  Need centuries for their mutual interaction to reach a dynamic equilibrium.
- 	Climate models are in some sense large multivariate, multidimensional time series generators which we hope produce time series that have similar properties to the real thing.

## Global Annual Average SST for 100 years (FOAM 1.5)

## Applications of State-of-the-art, high resolution Climate Models

- Traditionally, Climate models are built to do one problem:  Global Climate Change from current climate  (or global climate change from pre-industrial conditions through present)
- The *computational expense* of current climate models discourages making many coupled experiments.
- CSM2.0  4-5 years/day on 128 pes of IBM SP3
- 128x64x28(?) atmosphere, 328x320x40 (masked) ocean
- PCTM 5 years/day on 64 pes of IBM SP3
- 128x64x18 atmosphere, 384x288x32 (masked) ocean

## Fast Ocean Atmosphere Model v1.5A “mixed-resolution” climate model

- (Atmos.) PCCM3-UW:  R15 (40 lats, 48 lons, 18 levs) with CCM3.6 atmospheric physics. Spectral and Semi-Lagrangian (moisture) dynamics
- (Ocean) OM3:  128x128x24 z-coordinate ocean model. 2nd order finite differences, free surface, fully explicit timestepping.
- (Land) CCM2 nameless land model (4 layer soil model with 7 vegetation types)
- (SeaIce) CSIM 2.2.6 from CSM 1.4.  Leads, Semtner 3-layer, 0-layer, brine pockets, solar penetration.
- (River) UW Parallel River Transport Model

## FOAM Timings

- 800 MHZ PIII/ Scali Net

## Setting up a Climate Model ExperimentWhat needs to be changed?

- Initial and/or boundary conditions
- Initial values for T, U, V in ocean/atmos.  Land temperature, sea ice extent, etc.
- Boundary values:
- bathemetry
- topography
- land vegetation type
- albedo
- Parameter values
- CO2, CH4, Solar constant, etc.
- Source code configuration
- swap convection scheme, dynamical core, ocean model, etc.
- Time-varying CO2 instead of constant.

## Setting up a Climate Model ExperimentCurrent Obstacles

- Initial and/or boundary conditions
- No “reference” data set for things like paleoclimate.
- Different file formats
- Programs which created files aren’t documented/distributed (only doing one problem so why bother?)
- Parameter values
- Not all values are run-time configurable
- Source code configuration
- It’s Fortran
- Inflexible build system

## Setting up a Climate Model ExperimentSource code configuration example: FOAM

- Currently:  Nearly all source in a single directory with one makefile.  295 Fortran files, 17 C.  CVS control.
- In the near-future we want:
- A modern build system with at least one directory for each model and a Makefile in each directory called by a top-level Makefile
- Ability to build components *and physics packages* as “stand-alone” models for testing: ocean, atmosphere, radiation-scheme only.
- All of the above working under CVS or other source code control.

## Must be identical in repository!

- 	Makefile
- 	pccm3uwsrc/
- 		Makefile
- 		pccm3.F
- 		ccm3radsrc/
- 			Makefile
- 		ccm3BLsrc/
- 			Makefile
- 	om3src/
-       ...
- CCM3rad/
- 	Makefile
- 	ccm3radsrc/
- 	driver.F

- This can be done with FOAM’s dusty deck F77 with a carefully thought out CVS repository structure and CVS “modules” file.

- FOAM example cont’d

## Setting up a Climate Model ExperimentSource code configuration example: FOAM

- Some remaining problems:
- FOAM is a single-executable image; need to set make options, recompile for every change.
- New models from outside sources have to conform to FOAM build/CVS structure.
- ITR/AP: Flexible environments for grand-challenge climate simulation
- Use scripting language (python) to configure models “on the fly”.
- Can easily add runtime vis, diagnostics packages, new models into the data flow.
- Construct python interfaces to code with automatic wrapper builder SWIG.

## Setting up a Climate Model ExperimentWish-list Items

- Boundary condition design tool:
- Allow scientist to “design a planet” and simulate its climate
- continent placement, ocean bathemetry, surface topography, vegetation layout, etc....
- Consistency checking:  Do all land points have a vegetation type?
- Output at selected resolution
- Ingest reference data when available.
- Case Tracking (more later....)

## FOAM configured for two neo-proterozoic experiments

## Running a Climate Model ExperimentCurrent Obstacles

- A 500 year integration of FOAM would require about 15 days of continuous integration on 32-40 processors of a current MPP.
- Most compute centers have a 5-6 wallclock hour limit. (longest is at NCSA:  400 hours)
- Models must have restart ability to conduct a long integration
- “Queue banger” -- a shell script to monitor job progress and automatically submit the next job to start from the last restart file of the previous job.
- Realizable throughput highly dependent on queue wait time which in turn depends on other users, system uptime, etc.
- Output data broken up into several files.

## Running a Climate Model ExperimentFuture Developments

- Globus-enabled tools such as “Espresso” for handling long integrations with multiple re-submissions.
- Compute Centers?  Who needs them?!
- Single-investigator owned commodity clusters with fast switches such as Myrinet or ScaliNet are matching or beating Origin 2000 for FOAM1.5.
- Storage capacity and access becomes chief attraction.

## Analyzing a Climate Model ExperimentCurrent Obstacles

- Data sets can be large:
- FOAM 1.5:  	188 MB/ simulated year
- PCTM:		1.6 GB/simulated year
- CSM2.0:  		4 GB/simulated year
- Continuous time series broken up into several files.
- Data analysis/graphics packages have no concept of a time series spread out over several files.  Some can fake it but still need all data online at once.

- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 15:38 ha.F1_5.preindctl.00034.nc
- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 15:39 ha.F1_5.preindctl.00035.nc
- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 15:39 ha.F1_5.preindctl.00036.nc
- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 19:15 ha.F1_5.preindctl.00037.nc
- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 19:15 ha.F1_5.preindctl.00038.nc
- -rw-r-----      1 rob       ac       AR  common  41856236 Jan 12 19:15 ha.F1_5.preindctl.00039.nc
- -rw-r-----      1 rob       ac       DK  common  41856236 Jan 12 22:00 ha.F1_5.preindctl.00040.nc
- -rw-r-----      1 rob       ac       DK  common  41856236 Jan 12 22:00 ha.F1_5.preindctl.00041.nc
- -rw-r-----      1 rob       ac       DK  common  41856236 Jan 12 22:00 ha.F1_5.preindctl.00042.nc

## Analyzing a Climate Model ExperimentTypical Experience:

- Log on to remote compute center
- Stage data from tape
- Wait
- Move data from archive to center’s local disk.  Hope local disk is large enough.
- Concatenate or otherwise post-process only the variables you want into a single file  (NCAR’s NCO utilities very helpful)
- If your favorite graphics/analysis package isn’t available or network is to slow for remote X-windows, scp file to your local disk.
- Repeat above for any other variables you want to look at!

## Analyzing a Climate Model ExperimentWish List

- Grid-enabled data analysis/visualization package with publication quality output which can cope with:
- Time series over several files
- Files in remote locations
- Not all files online at once.
- Annotation system:  Tell at-a-glance which experiment you’re looking at and be able to retrieve compilation settings, run time parameter settings and boundary condition data used.

## Summary

- Mixed-resolution climate models have reached “useable” throughput speeds.
- Data bottlenecks and software management are biggest obstacles to scientific progress in climate.
- Climate scientists get by today with a chain of     ad hoc solutions.   Help!

## Who Can Help?

- Several SciDAC projects
- Scientific Annotation Middleware
- Scientific Data Management
- Earth System Grid
- Grid toolkits (Globus)
- TeraGrid project
- But....we need this now.  When will the middleware be done and who’s going to write the end-ware?