# Presentation: 054513

## Using The GMAO GEOS-5 AGCM

- Software Integration and Visualization Office
- NASA/GSFC

## About GEOS-5

- Developed by the NASA/GSFC Global Modeling and Assimilation Office (http://gmao.gsfc.nasa.gov)
- First model to be constructed “ground up” based upon Earth System Modeling Framework
- Designed for weather & climate research
- Makes use of finite-volume dynamical core (Lin, 2004)

**Notes:**
The Global Modeling and Assimilation Office (GMAO) has delivered the next generation global atmospheric model, GEOS-5. This system represents a new paradigm for coupled earth system modeling, in that GEOS5 is being developed entirely within  the Earth System Modeling Framework, and in that context is integrating model elements from different agency and community investments.

GEOS-5 is a major element of NASA's “Earth System Model,” a model where the components may be interchanged or augmented via ESMF and in collaboration with the GMAO. 

Under the MAP modeling environment (MAPME), community investigators may perform scientific experiments with the model directly  or modify/replace/add components easily within ESMF.

## Outline

- Model Overview
- Getting the code
- Directory Structure
- Browsing the code (Understand for Fortran)
- Setting your Environment
- Building the code
- Setting up an experiment
- The run.script
- Submitting/Monitoring Jobs at NCCS
- Diagnostics
- Post-Processing
- Verification
- Quick Start Guide
- Where to get help

## Let’s Get an interactive job in the Training Queue

- Interactive Batch
- qsub -I -l ncpus=4 -l walltime=8:00:00 -W group_list=k3002 -q training /usr/bin/tcsh

**Notes:**
The job can be run in Batch or Interactive modes on Explore...

	For batch we can qsub the run.script, watch for execution, and look for STDOUT/STDERR to return to our experiment directory.

	Alternatively, we can grab a set of interactive processors for testing changes and confirming execution...
		lets try to grab a small number now...

	For interactive jobs we see all STDOUT/STDERR returned to our terminal window.  During batch execution if you wish to observe your stdout you will have to proceed with the following ssh commands (until the NCCS moves the pbs spools to a GPFS file system that we can view from palm)

## GEOS-5 Overview from the GMAO

- The Goddard Earth Observing System Model, Version 5 (GEOS-5) is a system of models integrated using the Earth System Modeling Framework (ESMF).
- The system includes an atmospheric general circulation model (AGCM) developed for both weather and climate applications using the finite-volume dynamical core (Lin, 2004).
- Developments of GEOS-5 were guided by the need for realistic representation of tracer transports and stratospheric dynamics and the need for a realistic representation of moist processes for climate applications and for atmospheric data assimilation.
- The GEOS-5 AGCM is coupled to the Catchment Land Surface Model (LSM).
- The GEOS-5 atmospheric data assimilation system (DAS) couples the AGCM with the Grid-point Statistical Interpolation (GSI) analysis scheme being developed by the Environmental Modeling Center of the National Centers for Environmental Prediction (NCEP/EMC) and GMAO.
- The AGCM and the DAS have been run with 72 levels to 0.01 hPa and at horizontal resolutions from 2° to 0.25 °.

**Notes:**
The GMAO GEOS-5 is a Unified Atmospheric Model, using the finite-volume dynamical core developed in the DAO and physics packages developed by NSIPP, other scientists in the NASA Science Mission Directorate, and at other modeling centers. 

The GMAO has a strong land surface model development effort in the Catchment Land Surface Model with Randy Koster of the GMAO.

Assimilation tools are developed to optimize the use of the high resolution information from satellite observations and diagnose their impact on predictions of both weather and climate. 
Research-quality assimilated datasets, including trace gas, aerosol and climate products, ocean and land surface products, are generated for use by NASA instrument teams and for research analyses.

## GEOS-5 Overview from the GMAO

- The Aerochem component of GEOS-5 incorporates parameterized tropospheric chemistry, the stratospheric chemistry and GOCART aerosol modules from the Atmospheric Chemistry and Dynamics Branch at GSFC as well as a carbon transport module.
- The GEOS-5 Coupled Climate General Circulation Model (CGCM) couples the AGCM with the Poseidon V5 ocean model developed by Paul Schopf at George Mason University.
- The GMAO Ocean Data Assimilation System (ODAS) is fully integrated into GEOS-5 using ESMF couplers. Other chemistry and ocean models will be included over time.
- The GEOS-5 architecture supports running each of the component models in stand-alone mode as well as in various coupled configurations.
- The GEOS-5 system undergoes continual development to support the scientific investigations of GMAO and collaborators.
- Major new releases can be expected every 6-12 months, with minor upgrade releases for bug fixes as needed.

**Notes:**

While the code we will be working on is solely the AGCM (with a Data ocean model (OGCM)), the GMAO has developed a coupled climate model with the Poseidon OGCM developed by Paul Schopf of George Mason University, as well as the Ocean data assimilation system (ODAS).

All models/components can be run in stand-alone mode or in a variety of coupled configurations via ESMF.

Major releases from the GMAO can be expected every 6-12 months, with minor updates for bug fixes as needed.

## GEOS-5 GCM Documentation

- http://gmao.gsfc.nasa.gov
- _[http://gmao.gsfc.nasa.gov/systems/geos5/test.php](http://gmao.gsfc.nasa.gov/systems/geos5/test.php)_
- _[http://gmao.gsfc.nasa.gov/systems/geos5/index_arch.php](http://gmao.gsfc.nasa.gov/systems/geos5/index_arch.php)_
- http://map.nasa.gov
- _[http://map.nasa.gov/GMAO_GEOS5.html](http://map.nasa.gov/GMAO_GEOS5.html)_
- _[http://map.nasa.gov/GEOSGCMbrowser/index.html](http://map.nasa.gov/GEOSGCMbrowser/index.html)_
- http://map.nasa.gov/GEOSGCMbrowser/GEOSgcm_5_lvls.jpg

**Notes:**
There is ample documentation available via the GMAO and the MAP websites.  

The GMAO provides links to model documentation and references to scientific work performed with the model.

The MAP website provides an in-depth look at the source code through a browse-able interface as well has a detailed diagram of the code structure. 

## GEOS-5 GCM Structure

**Notes:**
This diagram is an depiction of ESMF-based architecture of the GEOS-5 atmospheric general circulation model. 

Each box is an ESMF component.  

Component and coupling interfaces are standardized to facilitate exchanges and extensions. 

The operations in each component or coupling transformation can be easily customized.

The coupled application (“CAP”) consists of an Atmospheric GCM and Ocean GCM (both with optional Data-only components), and a history component (for handling model diagnostics).  Exchange of information from AGCM to OGCM is handled by the Coupler component.

The AGCM is broken down into Dynamics and Column Physics components. 

The dynamics component consists of the finite-volume dynamical core (Lin, 2004) and a gravity wave drag component (from NCAR).

The column physics includes moist processes, the surface component including lake and ice components and the catchment land surface model, and a radiation component (with solar/IR components)

## Getting the Code

- Sourcemotel / CVS
- cd $HOME
- cvs -z3 -d:ext:*$USER*@sourcemotel.gsfc.nasa.gov:/cvsroot/esma co -r GEOSagcm-Eros-beta7p12 GEOSGCM
- Tarball from MAP website
- cd $HOME
- tar -zxvf GEOSagcm-Eros-beta7p12.tgz

**Notes:**
Slide directing to login to Palm.

The source code can be retrieved in two ways depending on your level of collaboration with the GMAO and the requirements of your team.

As a direct collaborator where you will be making changes to the code or replacing components you will be provided direct access to the CVS archive via Sourcemotel (the Goddard mechanism for managing source code and software development through the entire development life cycle, supported by SIVO.)

Sourcemotel provides groups with software management tools including Group Forums and Mailing lists, controled access to repositories via CVS, and additional tools including documentation management, issue tracking and task management.

To access the code via CVS:
	Need to be added to the ESMA project
	Need to set your access mode as setenv CVS_RSH ssh
	You can also define you cvsroot in the environemtn: setenv CVSROOT :ext:$USER@sourcemotel.gsfc.nasa.gov:/cvsroot/esma

Alternatively, groups participating solely as users of the code, who will not be requesting access to directly provide updates or add/replace components will have access to a tarball download from the MAP website.

## GEOSGCMModel Directory Structure

- src
- src/GNUmakefile
- src/Config
- src/Application
- src/Shared/GEOS_Shared
- src/Shared/GEOS_Base
- src/Shared/GEOS_Util
- src/Shared/GEOS_Util/post
- src/Shared/GEOS_Util/plots
- src/Shared/GMAO_pilgrim
- src/GEOShistory_GridComp
- src/GEOSgcm_GridComp
- src/GEOSgcm_GridComp/GEOSogcm_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp/FVdycore_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp/GEOSgwd_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSturbulence_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp/GEOSsolar_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp/GEOSirrad_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSmoist_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSchem_GridComp/GEOSpchem_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp
- src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp

**Notes:**
After acquiring the code via cvs/tarball you will see the GNUmakefile and the following directory structure.

Src/Config - contains all architecture specific compiler/mpi/build/model options

Src/Application - contains the main progam (GEOSgcm.F90) the Composite Gridded Component for the SuperDynamics forced by full physics.  This directory also contains the sample AGCM run scripts (run.script and others).

Src/Shared - contains the core utilities for the model including the base utilities and the utilities used to create the model ESMF gridded components (Shared/GEOS_Base/MAPL/MAPL_Generic.F90), the Parallel communication library (GMAO_pilgrim),  and post processing utilities (GEOS_Util)

src/GEOShistory_GridComp - contains the module for Initializing the GEOS History Lists for Diagnostic Output, including documentation on how the modify the diagnostic output streams.

src/GEOSgcm_GridComp - includes the core atmospheric components
src/GEOSgcm_GridComp/GEOSogcm_GridComp - the ocean model or data component.
src/GEOSgcm_GridComp/GEOSagcm_GridComp - the atmosphere component 
	consisting of ‘superdynamics’ including:
		the Fintie-Volume dynamical core
		the NCAR WACCM1b Gravity wave drag component)
	and the phsyics component including:
		the turbulance component [a combination of the first-order scheme of Louis---for stable PBLs and free atmospheric turbulence---with a modified version of the non-local-K scheme proposed by Lock for unstable and cloud-topped boundary layers.]
		the radiation component [the shortwave solar component based on M-D Chou SORAD routine http:\\climate.gsfc.nasa.gov\~chou\solar, and the longwave radiative transfer calculation based on M-D Chou's IRRAD routine as referenced in "A Thermal Infrared Radiation Parameterization for Atmospheric Studies" M.-D. Chou et al., NASA/TM-2001-104606, Vol. 19, 55 pp, 2003.]
		the moist processes component [including convection, large-scale condensation and precipitation and cloud parameters]
		the chemistry component [a proxy component for Aerochem that implements the specification or simple parameterization of the Aerochem species. It works on three types of species: chemical species (oxygen, nitrous oxide, CFC-11, CFC-12, CFC-22, methane, water vapor), diagnostic species (age-of-air), and aerosols (arbitrary).]
		the surface (Land) component [the dynamic vegitation (VegDyn) and the Catchment Land Surface Model of Koster et al. (2000)]

## Understand for Fortran

- What is it?
- An interactive development environment tool providing reverse engineering, automatic documentation, metrics and cross referencing of FORTRAN source code.
- Uses
- Browsing source code
- Understanding connections in the code
- Understanding variable declaration/usage
- Viewing use/use-by/invocation trees
- Generating in-depth reports
- References
- http://www.scitools.com/uf.html
- http://scitools.com/documents/manuals/pdf/understand_f.pdf

**Notes:**

This tool is useful for in-depth analysis of the source code, including graphical reverse engineering, fast code navigation, detailed cross referencing, interactive source browsing, automatic documentation via HTML output reports, various complexity and volume metrics

As a quick exercise let’s look at the GEOSGCM code in UFF...

Viewing a previously created GEOS-5 project...

	View the “Files Tree”
	Expand the “Application” tree
	locate the GEOGgcm.F90 file
		we can view the source code in the ‘Text Window’
		by right clicking the file name we can view the ‘Info Window’

	expanding the Subprograms tree in the info window we can see the main program
		right clicking allows us to display a graphical view of the ‘Use’ tree for the GEOS application
		this gives us an in-depth tree view of the entire module structure of the code (quite overwhelming large...)

	delving deeper into the code, we can locate the AGCM->Dynamics->Fvdycore_GridComponent->dynamics_vars
		clicking once brings this routine up in the Info Window, we can see all the info on this file including public subprograms,variables and types
		as well as all places where this module is used and the file metrics including line counts and references within the code
	exploring further, we can right-click the Fvdycore_wrapper (the main driver for the FV dycore) and view the ‘Invocation Tree’
		This brings up a graphical view of the call structure for the entire dycore... (including the C-D Grid core (cd_core), the tracer transport (trac2d) and the vertical remapping (te_map), expanding all of this throughout the call structure)

	Of course we can delve deeper into individual routines, variables, types etc...
		we can view the Fvdycore_wrapper source code, and right-click the FV_DYCORE_VARS type and view the state variables carried with this (u,v,pt,pe,pkz,tracers)...

## Creating your development branch

**	**_**Make a branch**_**: ****You have checked out a “tagged” version. You cannot commit changes to a “tagged” version, since that would make the tag meaningless.  Before committing your changes, you must create a new “branch” and then commit to the branch:****	****cvs tag –b Branch_Name  $files****	****where ****$files**** is the list of files that you have modified, and ****Branch_Name**** is a name of your choice.  The branch name should be somewhat descriptive, including things such as the tag you are branching from (eg., ****b_GEOSagcm-Eros-beta7p12****) or the name of a control experiment (eg., ****b_e0247****).****Once the branch has been created, you can “update” your files to that branch:**
**	****	****cvs upd -r Branch_Name  $files**
**	****This does not change the files you modified, only their cvs “status” in your sandbox.**
**	**_**Commit**_**: **
**	****To commit your changes, simply type:****	****	****	****cvs commit $files**
**	**_**Tag**_**:**
**	****Finally you are ready to tag ALL files (newly modified as well as untouched) to reflect your new experiment.  At the GEOSGCM/src level:**
**	****	****cvs tag Your_Tag_Name****	****	****	****	****	**

**Notes:**
After making modifications to the code, you’ll need to create a branch where you can commit your changes and tag updates...  This procedure is outlined here.

## Building on Palm/Explore at NCCS

**Set Module Environment (preferably within .cshrc or .login):**
**module purge  **
**module load intel-comp.8.1.034**
**module load mpt.1.12.0.nas**
**module load scsl.1.5.1.1**
**setenv BASEDIR /share/ESMA/baselibs/v2_2rp2.34meta**
**setenv LD_LIBRARY_PATH**** ${LD_LIBRARY_PATH}:/share/ESMA/baselibs/v2_2rp2.34meta/Linux/lib**
**setenv GEOSUTIL ~/GEOSGCM/src/Shared/GEOS_Util**
**Issue install command:**
**cd ~/GEOSGCM/src**
**gmake install**

**Notes:**

The module environment needs to be setup to build/execute the model.  

This environment setup can be performed in the .cshrc or the .login file, alternatively if you will be running other codes that require different module setups you can source an environment setup file for GEOS-5 (example in ~wputman/GEOS-5.0_env).
	-Note: if you would like GEOS-5 to run out-of-the-box you’ll need to include this in you .cshrc or .login file, otherwise we will need to add the appropriate setup in the make process and the run.script

BASEDIR is a link to baselibs which houses core libraries and third party software (netcdf, hdf ...)  to build and execute GEOS.  The current baselibs support points to a pre-release version, this should be updated if not by this talk, in the very near future to a stable released version in /share/ESMA/baselibs

GEOSUTIL points to the location of the post-processing utilities, there is a copy which comes from CVS or a copy may be placed in ~/GEOS_Util via direct CVS checkout (cvs -z3 -d:ext:userid@sourcemotel.gsfc.nasa.gov:/cvsroot/esma co GEOS_Util)

gmake install will build the executable this can take in excess of 2 hours to complete

## Building the Code

- Installation
*gmake install* builds the executable
- Dependencies
- Generated during the build process
*May* reduce recompilation times after changes
- Errors/Warnings
- Warnings
- /home1/wputman/GEOSagcm-Eros-beta7p12/GEOSGCM/Config/GMAO_base.mk:160: WARNING: No environment check performed.
- fortcom: Warning: mpi_isend.F90, line 12: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [REQUEST]
-  comm, request, ierror)
- -------^
- fortcom: Warning: mpi_recv.F90, line 11: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [STATUS]
-     subroutine mpi_recv( buf, count, type, pe, tag, comm, status, ierror)
- ----------------------------------------------------------^
- fortcom: Warning: mpi_irecv.F90, line 11: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [REQUEST]
-     subroutine mpi_irecv( buf, count, type, pe, tag, comm, request, ierror)
- -----------------------------------------------------------^
- fortcom: Warning: cloudnew.F90, line 100: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [CLDNCCN]
- 			CLDNCCN      , &
- --------------------------^
- Where is my executable?
- ~/GEOSGCM/src/Application/GEOSgcm.x

**Notes:**
Again, the build process may take in excess of 2 hours.  This is normal.  During the build process all dependencies within the code are generated which may reduce build times after you make changes to the code.

There are some typical warnings that do appear, these are typically warnings regarding the use of intent(out) variables for miscellaneous arguments.  These are harmless and do not effect the execution of the code.

The executable is created in the ~/GEOSGCM/src/Application directory where it can be linked or copied to the experiment directory you will create later.

Let’s go ahead and launch the make process...
	source you module setup file for GEOS if you have not included this in your .cshrc or .login file..
	let’s launch the build in the background redirecting the output to a log file so we can continue on with the experiment setup while the build is in progress.
		gmake install >& gmake.log &

## Setting up an Experiment

- Experiment Directory
**	****	****mkdir -p $HOME/geos5/e0247**
**	****	****cd $HOME/geos5/e0247**
**	****	****cp $HOME/GEOSGCM/src/Application/run.script .**
**	****	****edit run.script (change group ID from g0620 to ...[k3002])**
- Run Directory (Work/Scratch space)
- Work directory
**	****	****mkdir /nobackup/$USER/e0247**
**	****	****cd /nobackup/$USER/e0247**
**	****	****ln -s $HOME/GEOSGCM/src/Application/GEOSgcm.x .**
**	****	****cp /nobackup/wputman/GEOS-5.0_restarts/* .**
- Scratch directory (where the model runs...)
**	****	****/nobackup/$USER/e0247/scratch  ****(ONLY if $TMPDIR is not defined on the system)**
- Mass Storage
- Created by run.script
**	****	****/output/$USER/GEOS5.0/e0247**

**Notes:**

The experiment directory is created in your home directory, labeled by the experiment ID ( in this case e0247 )
	this directory houses the scripts for model execution and will contain the stdout and stderr from your batch jobs as you execute the model.

The run directory is created on /nobackup...
	this directory is labeled by experiment ID and will house the executable, associated restarts, the scratch space where the model is actually executed, and temporary housing for all model output prior to post-processing (conversion to pressure levels, time-averaging and plotting)

The structure of the scratch directories after execution and model output directories will be described later...

$RESTARTS needs to be better defined (currently taken from /nobackup/ltakacs/test6/*restart /nobackup/ltakacs/test6/*.rc I have a copy of these in /nobackup/wputman/GEOS-5.0_restarts  144x91 72L configuration)

## Getting to know the run.script

- Basic modifications
- processor count
- MPI layout
- resolution
- timestep
- begin/end dates
- Job segments (YYYYMMDD HHMMS)
- Iterations/counters
- diagnostics

**Notes:**

Run.script handles all aspects of model execution.

Let’s walk through the run.script...

This can be run out-of-the-box with a simple change of the group ID from  g0609 to your_gid (k3002)..

Most of the script is self-sufficient and requires no modification...
	The batch parameters are set at the top of the script for PBS on the Altix and BSUB on Halem...
	System Environment Variables are set depending on the architecture and host you are running on..
		this includes the boundary condition directory (BCSDIR currently located at /share/ltakacs/geos5/bcs)
		and the mass storage machine in this case for NCCS /output on Palm..
	All Experiment Directories are setup as we described earlier..

The Model Run Parameters is an area where we will be making most of our modifications..
	domain decomposition for MPI parallelism (NX, NY)
	AGCM model resolution IM, JM, LM
	dateline (DC = centered, DE = edge)
	OGCM model resolution
	timestep DT
	BEG_DATE - needs to be <= current date in restart
	END_DATE - needs to be > current date in restart
	JOB_SGMT - defines the length of integration for each model execution segment, (default is 10-days) YYYYMMDD HHMMSS (00000010 000000)
	EXPDSC - gives a description of the experiment attached to all restarts/output streams (usually providing the expid and the tag information)
	ANALYSIS - set to 0 since this is an AGCM run, not a Data Assimilation (DAS) run.

The CAP.rc resource file is created given the information provided in the model parameter setup.

The History List for Output is created annd stored in History.rc resource file.
	we can make changes here depending on what you like to output, and whether you like to use the pre-packaged output plotting routines from the GMAO.
	modifications here will be described in greater detail in slides to follow..

The AGCM Resource File is created GCM.rc to translate the parameters provided for the model to ingest...
	this includes information such as the time-interval to refresh the SST data, the timesteps to perform radiation updates, restart files and boundary condition datasets to use.

The fvcore_layout resource file is created to describe the domain, mpi decomposition, time-stepping and time-splitting.. for the FV dycore.

The Boundary Datasets are then linked to the run directory, and the restarts and executable are copied...

The run.script executes a number of job segments, checkpoints, archives and launches post-processing using the iterations and counter parameters
	default is to run 5 10-day iterations per run.script.
	after this, the run.script re-submits itself to the batch queue (if necessary) and exits

For this experiment we need to change the resolution, so lets do that and let’s update the timestep to be consistent with the resolution change.

We also need to change the default batch queue for the post-processing to general_small !!!!!

## Running Your Experiment

- PBS
- Batch
- Set appropriate group ID
- qsub run.script
- Interactive
- qsub -I -l ncpus=32 -l walltime=2:00:00 -W group_list=your_gid /usr/bin/tcsh
- Monitoring job progress
- 		qstat -s
- 		184932.palm     user_id  general e0247   28266  -- 120 398540 06:00 R 01:21
- 		   Job run at Tue Nov 21 at 17:12 on e1:ssinodes=120:ncpus=120:mem=4798464...
- 		ssh e1 “ls -l /pbs/spool/*”
- 		ssh e1 “tail -f /pbs/spool/184932.palm.OU”

**Notes:**
The job can be run in Batch or Interactive modes on Explore...

	For batch we can qsub the run.script, watch for execution, and look for STDOUT/STDERR to return to our experiment directory.

	Alternatively, we can grab a set of interactive processors for testing changes and confirming execution...
		lets try to grab a small number now...

	For interactive jobs we see all STDOUT/STDERR returned to our terminal window.  During batch execution if you wish to observe your stdout you will have to proceed with the following ssh commands (until the NCCS moves the pbs spools to a GPFS file system that we can view from palm)

## Diagnostics

- Where are they defined
- History.rc file created in run.script
- Example
-   list(1)%filename:   'geosgcm_prog',
-   list(1)%template:   '%y4%m2%d2_%h2%n2z',
-   list(1)%format:     'CFIO',
-   list(1)%frequency:  060000,
-   list(1)%fields:     'PHIS'           , 'SUPERDYNAMICS',
-                             'SLP'            , 'DYN'                        ,
-                             'U'                 , 'DYN'                        ,
-                             'V'                  , 'DYN'                        ,
-                             'T'                  , 'DYN'                        ,
-                             'PS'               , 'DYN'                        ,
-                             'DELP'           , 'DYN'                        ,
-                             'OMEGA'       , 'DYN'                        ,
-                             'QV'               , 'MOIST'                    ,
-                             'H2O_PROD' , 'CHEMISTRY'          ,
-                             'RH2'             , 'MOIST'                    , 'RH'  ,
-                             ::
- An experiment may have Multiple output streams
- Varying formats (binary [‘raw’] or HDF [‘CFIO’])
- Consistent vertical dimension
- 1 to LM for vertical centered variables
- 1 to LM+1 for edge variables
- surface or 2D variables
- The default output streams in run.script are required for the GEOS_Util post-processing plotting utility to execute cleanly.
- Post-processing of eta to pressure level conversion should still occur

**Notes:**
The hirstory.rc file is created in run.script at line 155 as we saw earlier and is used to Create a History List for out Output streams.

For each experiment can create multiple output streams (files) 
	list(1)%filename: ‘geosgcm_out1’
	list(2)%filename: ‘geosgcm_out2’
	list(3)%filename: ‘geosgcm_out3’

An example of a standard output stream is shown here.  This produces the prognostic variables in an HDF (CFIO) file format named as geosgcm_prog (this will be appended to the experiment id for identification).  The output frequency is every 6 hours and the fields will be written as instantaneous snapshots of the data.

The fields are listed with their corresponding gridded-component including in this case:
	surface geopotential from the superdynamics
	sea-level pressure, U,V winds, temperature, surface pressure, pressure thickness (DELP) and vertical pressure velocity (OMEGA) from dynamics
	specific and relative humidity from the moist physics component
	and an H20 production from the chemistry component

Each output stream must have a consistent vertical coordinate
	ie, we cannot have vertical centered variables mixed with edge variables
	we CAN have surface variables (or 2D variables) mixed with 3D variables (as is the case above for SLP, PS and PHIS)

The default output streams are required for the GEOS_Util post-processing plotting utilities to execute cleanly (eta2prs should occur naturally for all files).

## Diagnostics

- Options for Output Streams
- filename
- Character string defining the filename of a particular diagnostic output stream.
- template
- Character string defining the time stamping template following GraDS conventions.
- format
- Character string defining the file format (‘flat’ of ‘CFIO’) ‘flat (raw binary)’ is default, ‘CFIO’ is HDF.
- mode
- Character string equal to "instantaneous" or "time-averaged". Default = "instantaneous".
- descr
- Character string equal to the list description. Defaults to "expdsc".
- frequency
- Integer (HHMMSS) for the frequency of output.  Default = 060000.
- acc_interval
- Integer (HHMMSS) for the acculation interval (<= frequency) for time-averaged diagnostics.
- ref_date
- Integer (YYYYMMDD) reference date from which the frequency is based.
- ref_time
- Integer (HHMMSS) reference time from which the frequency is based. Default = 0.
- end_date
- Integer (YYYYMMDD) ending date to stop diagnostic output.  Default is disabled.
- end_time
- Integer (HHMMSS) ending time to stop diagnostic output. Default is disabled.
- duration
- Integer (HHMMSS) for the duration of each file.  Default = 240000 (one-day segments).
- fields
- Paired character strings for the diagnostic Name and its associated Gridded Component.
- resolution
- Optional resolution (IM JM) for the ouput stream. Default is the native resolution.
- levels
- Optional list of output levels (1 to LM for vertical centered, 1 to LM+1 for edge vars). Default is all levels.

**Notes:**
There are a large number of options for the output streams:

The most common options will be the file format, output mode,  and frequency.

Some useful additional options are the start/end dates/times for recording output streams and the ability to write the data at resolutions other than the native resolution.

These option can be seen in the ~/GEOSGCM/src/GEOShistory_GridComp/GEOS_HistoryGridComp.F90 history gridded component documentation.

## Run Directory Structure (after execution)

- /nobackup/$USER/$EXPID
- Scratch (if $TMPDIR does not exist)
- restarts
- listings
- Holding (daily eta and prs output streams)
- Output streams (monhtly averages):
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_edge
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_gwd
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_moist
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_prog
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_rad
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_surf
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_tend
- drwxr-xr-x   2 wputman k3002       10 2006-11-20 18:46 geosgcm_turb

**Notes:**
The model runs in the scratch directory, or $TMPDIR...

There is a directory to house all restarts at the end of each job segment.

The post-processing batch scripts are housed in a listings directory, and all stdout and stderr from these scripts are sent here.

The files from your output streams are held in the holding directory in separate directories for each stream

The top-level output stream directories will hold the monthly averaged files...

There is also a plots directory created by the post-processing to hold the monthly averaged plots (including means, comparisons to other re-analyses, Taylor diagrams...)

## Output Directory Structure

- /output/$USER/GEOS5.0/$EXPID
- restarts
- output streams
- Including eta and pressure level files
- monthly
- Plots
- All output streams are GrADS readable
- HDF files use sdfopen ...

**Notes:**
This is long-term mass storage of key files including, restarts/checkpoints, output streams (both eta and pressure level), and monthly means (by default for long integrations daily files are removed), and any plotting from the GEOS_Util package.

## Post-Processing

- GEOS_Util
- eta2prs
- Based on History.rc file
- Monthly means
- Archiving
- Plotting
- Some visualization capability is provided in the post-processing scripts
- Quickplot routine produces seasonal mean plots

**Notes:**
By default, all daily eta (model-level) output streams will be stored in the scratch (/nobackup) space.  The GEOS_Util utilities will post-process the eta files and produce eta2prs pressure files based on the History.rc list generated by run.script.

Post-processing by default will produce monthly-averaged files, and archive daily/monthly output streams to mass storage (/output at NCCS or lou at NAS).  All daily output streams will be removed from /nobackup leaving only monthly-averaged files behind for graphical processing.

The quickplot procedure included in the GEOS_Util package will produce seasonal mean plots for comparison with other GEOS runs or external re-analyses.

## Quick Start Guide at NCCS

- Create Source Directory
**	****	****cd $HOME**
**	****	****cvs -z3 -d:ext:*****$USER*****@sourcemotel.gsfc.nasa.gov:/cvsroot/esma co -r GEOSagcm-Eros-beta7p12 GEOSGCM**
- Set Module Environment (in .cshrc or .login)
**	****	****module purge  ****	****module load intel-comp.8.1.034****	****module load mpt.1.12.0.nas****	****module load scsl.1.5.1.1****	****setenv BASEDIR /share/ESMA/baselibs/v2_2rp2.34meta**
**	****	****setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/share/ESMA/baselibs/v2_2rp2.34meta/Linux/lib**
**	****	****setenv GEOSUTIL ~/GEOSGCM/src/Shared/GEOS_Util**
- Build
**	****	****cd GEOSGCM/src**
**	****	****gmake install**
- Experiment Directory (scripts)
**	****	****mkdir -p $HOME/geos5/e0247**
**	****	****cd $HOME/geos5/e0247**
**	****	****cp $HOME/GEOSGCM/src/Application/run.script .**
**	****	****edit run.script (change group ID from g0620 to [k3002], confirm resolution, date, execution intervals..)**
- Run directory (work/scratch space)
**	****	****mkdir /nobackup/$USER/e0247**
**	****	****cd /nobackup/$USER/e0247**
**	****	****ln -s $HOME/GEOSGCM/src/Application/GEOSgcm.x .**
**	****	****cp $RESTARTS .**
- Submit the job
**	****	****cd $HOME/geos5/e0247**
**	****	****qsub run.script**

**Notes:**
gmake install BOPT=g

For those like me interested in a one-stop shop for how to get the code built and running here is a quick start guide to GEOS-5.

Create the source directory via CVS checkout as we have displayed previously.

Set you module environments (either in a .cshrc or .login files, or create a GEOS-5 setup file to source prior to building/running)

Issue the build command in GEOSGCM/src (wait 1.5 or more hours)

Setup a run directory for your scripts in your home directory, the experiment name used here is consistent with that which is checked out from CVS.

Move the run.script to the run directory and modify the group ID.

Setup your experiment directory in /nobackup
	link or copy the executable to the experiment directory
	move restarts into the experiment directory (sample restarts included in /nobackup/wputman/GEOS-5_restarts)

Submit your job and wait in the queue and you’re up and running...

Now we will look at these steps in more detail. And you can work along with your copy of the code.

Other notes:
BASEDIR=/share/ESMA/baselibs/v2_2rp2.34meta or /home/trayanov/baselibs/v2_2rp2_nb2
Build time on Palm takes about 1.5 or more hours
$RESTARTS needs to be better defined (currently taken from /nobackup/ltakacs/test6/*restart /nobackup/ltakacs/test6/*.rc)

## Verification

- A 10-day sample case
- Typical STDOUT
- HDF eta or pressure level data
- GrADS readable
- A long integration
- Monthly means
- Plotting
- Horizontal contour plots
- Comparisons with NCEP/ECMWF Reanalyses
- Taylor Diagrams

**Notes:**
While there is no formal validation process available, a user is able to complete short runs and quickly view the output using GrADS.  The data may be HDF eta level or post-processed pressure level data.

After completing a long integration, seasonal data are used to complete an exhaustive series of plots including horizontal slices on pressure levels, comparisons with NCEP or ECMWF reanalyses, and complex Taylor diagrams to assist in evaluating the model performance.

## Sample plots

- Correlation

- Std Deviation

- Std Deviation

- RMS Error on Green Curves

**Notes:**
Taylor diagrams (Taylor, J. Geophys. Res., 106, 28033-28059, 2001; see also PCMDI Report 55) are designed to allow one to quickly assess how model changes affect model performance. They can also be helpful in determining how an individual model fairs with respect to others. The diagrams provide a statistical comparison between simulated and observed fields.  Three statistics are shown on a Taylor diagram: the correlation coefficient between the observed and simulated field (given by the cosine of the azimuthal angle), the centered root-mean-square (RMS) difference between the two fields (proportional to the distance from any point on the diagram to the point on the x-axis marked "observed''), and the standard deviation of the fields (proportional to the radial distance).  In some cases, the RMS errors and the standard deviations have been normalized by the observed standard deviation, yielding a non-dimensional version of the diagram with the observed standard deviation found at unit distance from the orgin.  Note that the statistics shown on the Taylor diagram are "centered'' statistics (i.e., the overall means of the fields have been removed), so any mean bias in the model will not be evident.  Also note that on some diagrams the "observed" point is labeled by the name of the observational data set (e.g., ERA15, NVAP, SOC, etc.); see table.

Model skill in simulating a particular field can be assessed using the three statistics defined above.  A skillful model will have a relatively high correlation, a small RMS error and a standard deviation approximately equal to the observed.  In short, the point on a Taylor diagram representing a relatively good model will lie relatively close to the "observed'' point.

Usually many points appear on a single Taylor diagram, either representing different models or representing, on normalized plots, different fields. The relative positions of the points provides a visual perspective concerning skll differences.  On some Taylor diagrams an arrow is drawn between two points representing different model version.  By convention the older model version statistics are indicated by the tail of the arrow, and the newer model statistics by the arrow head.  If the arrow is directed toward the "observed" point, the new model can be said to more accurately simulate the observed field.

## Sample plots

## Visualization Support

- All GEOS-5 output streams are GrADS readable
- There are some built-in utilities for generating monthly-mean plots in GEOS_Util using GrADS
- NCCS/NAS systems provide access to IDL for visualization
- SIVO and the Scientific Visualization Studio will provide support to collaborators with these tools as well as via WMS-based tools

**Notes:**
All model output streams will be grads readable either binary or HDF files.  There is extensive plotting capability built into the post-processing utility (GEOS_Util), however this is only available for looking at monthly mean or seasonal output.  

Both the NCCS and NAS provide access to IDL on their HPC systems, and the user may develop plotting for GEOS-5 output using this tool, SIVO and the SVS are available for assistance with visualization requests.

Collaborators will also be able to request support from SIVO/SVS for visualization via WMS-based tools.

## Where to Get Help

- MAP website
- GEOS-5 Software Page
- http://map.nasa.gov/GMAO_GEOS5.html
- GEOS-5 FAQs
- To be developed on MAP website...
- GMAO website
- Experiments *(GMAO internal access required)*
- http://gmao.gsfc.nasa.gov/systems/geos5/index_results.php
- Technical Documentation
- http://gmao.gsfc.nasa.gov/systems/geos5/index_arch.php
- Scientific Documentation
- http://gmao.gsfc.nasa.gov/systems/geos5/index_pubs.php

**Notes:**
The MAPME website will provide details on the GEOS-5 software including this tutorial, as well as other support material for GEOS-5 including a FAQs section to be added to the website.

The MAPME site also provides a browsable interface to the source code, along with other model documentation. This site is also the place to look for contact information to get direct support from SIVO for building/running the code or visualization.

The GMAO website provides a great deal of information regarding the model, including past experiments performed (internal access), and technical/scientific documentation.