# Presentation: 139307

## The Flexible Modeling System Runtime Environment (FRE)

- September 8, 2006

**National Oceanic and Atmospheric Administration****Geophysical Fluid Dynamics Laboratory****Princeton, NJ 08542****http://www.gfdl.noaa.gov**

**Amy Langenhorst, RSIS/GFDL****Phone: 609 - 452 - 6572  ****Fax: 609 - 987 - 5063     ****Email: Amy.Langenhorst@noaa.gov**

## Overview

- Why FRE? What is FRE?
- Mechanics of FRE
- Command line utilities
- Optional:
- Tour of the XML
- DEMO: regression testing

**Notes:**
our solution is based on XML
XML language, XML tag set

## Why FRE?

- Get the right source code files, versions
- Set the right parameters for compilation
- Get the right input, correct versions of datasets
- Manage long production runs, job control
- Manage the output, postprocessing, figures
- Perform integrity testing on several model configurations
- Be able to reproduce your actions next year

**Notes:**
cvs
compilation: which libraries, cppdefs
job control:multiople submissions, resubmission to the correct job queue

## Previous Environments

- Up to the individual, differed from group to group, or person to person
- Notes/documentation optional
- Secondary users must decipher and translate the runscript without missing any details
- Models not reproducible over time
- The more general and flexible the runscript, the more complex, and harder to maintain
- Extensive regression testing onerous

## Key Points About FRE

- A standard mechanism for all FMS models
- A comprehensive description of model configurations
- records details from code versions through compiling, running, and postprocessing
- records model components, grids, dynamics and physics choices, all input parameters and data
- XML is self-documenting
- Easy to automate running regression tests

## FRE Components

- XML Model Description File: contains a complete description of one or more experiments
**cvs**: tags and modules needed to obtain code
**compile**: compiler flags, mkmf template, etc.
**input**: namelists, diag_table, etc
**run**: simulation length, npes, etc
**postProcess**: time series and averages, figures
- Command line utilities read XML file
**fremake**, **frerun**, **frepp**, etc

## FRE Components

- CSH script templates
- compile scripts
- model runscripts
- postprocessing scripts
- Standards and conventions for file names and locations
- 'root' directory tree in /home
- 'archive' directory tree in /archive

## FRE Mechanics

- XML

- FRERUN

- CSH

- Model
- Output

- FREPP

- CSH

- PostProc
- Output

**Notes:**
user -> xml -> frerun -> csh -> direct output
                                         -> frepp -> csh -> pp output

## User Edits XML

- <experimentSuite>
- <setup>
-   <directory type="root">$HOME/fms</directory>
-   <directory type="archive">$ARCHIVE/fms</directory>
- </setup>
- <experiment name="cm2u">
-   <cvs>
-     <codeBase>fms_cm2</codeBase>
-     <modelConfig>jakarta</modelConfig>
-   </cvs>
-   <input>
-     <initCond file="/path/to/file"/>
-     <gridSpec file="/path/to/file"/>
-     <namelist file="/path/to/file"/>
-   ...

## User Calls frerun

**>** frerun -s cm2u
- your job (432243) has been submitted
**>** frerun cm2u
- TO SUBMIT: qsub /home/user/.../scripts/cm2u
**>**

## Model Output in Archive

- cm2u
- |-- ascii
- |   |-- ascii_out.cpio
- |   `-- stdout
- |-- history
- |   |-- 00010101.nc.cpio
- |   |-- 00020101.nc.cpio
- |   `-- 00030101.nc.cpio
- `-- restart
-     |-- 00010101.cpio
-     |-- 00010201.cpio
-     |-- 00010301.cpio
-     |-- 00010401.cpio
-     |-- 00010501.cpio
-     |-- 00010601.cpio
-     ...

## Postprocessing Output

- cm2u
- |-- ascii
- |-- history
- |-- restart
- |-- analysis
- `-- pp
-     |-- atmos
-     |   |-- ts
-     |   |   `-- monthly
-     |   |       `-- 5yr
-     |   |           |-- atmos.000101-000512.mon.nc.cpio
-     |   |           `-- atmos.000601-001012.mon.nc.cpio
-     |   `-- av
-     |       `-- monthly_5yr
-     |           |-- atmos.0001-0005.01.nc
-     |           |-- atmos.0001-0005.02.nc
-     ...           ...

## Auto-generated Figures

- A subset of the figures automatically generated for one of our publicly-released atmospheric models is available at
- _[	http://www.](http://www.gfdl.noaa.gov/fms/fre.xml)__[gfdl](http://www.gfdl.noaa.gov/fms/fre.xml)__[.](http://www.gfdl.noaa.gov/fms/fre.xml)__[noaa](http://www.gfdl.noaa.gov/fms/fre.xml)__[.gov/fms/pubrel/analysis/m45_am2p14/](http://www.gfdl.noaa.gov/fms/fre.xml)_
- Internally, users can browse experiments and their associated scripts, XML, data, and figures through our Model Development Database
- Perl CGI scripts on a MySQL server

## Experiment Database

## XML Editors

- VIM and Emacs
- Syntax highlighting, tag completion, folding
- _[SciTE](http://www.gfdl.noaa.gov/~vb/mkmf.html)_: free user-friendly GUI-based editor with folding, can easily add menu options to launch utilities which act on current file
- Other editors should syntax highlight
- Free Graphical XML Browsers, Editors
- Mozilla-based web browsers
- _[pollo](http://www.gfdl.noaa.gov/fms/fre/dev/fre.xml)_

## The FRE Utilities

***fre-utility [options] [experiments]***
**fremake**: check out code, create/submit compile scripts
**frerun**: create and submit run scripts
**frepp**: create and submit postprocessing scripts
**frelist**: list experiments in an xml file
**frestatus**: show status of batch compiles and runs
**frecheck**: compare regression test runs
**freppcheck**: report missing postprocessing files
**frepriority**: change batch queue information
**frescrub**: delete duplicate postprocessing files
**freversion**: use XSLT to change XML format

## The FRE Utilities

- Written in Perl
- Use XML::LibXML
- very good at searching xml with xpath queries

## fremake

**fremake -x xmlfile experimentname**
- performs cvs checkout if you don't already have code in $name/src
- generates a simple shell script which calls mkmf (the makefile-generator) and make.  You can qsub this script or run interactively.
**-p** option: prevent auto-generation of path_names file (list of source code files) for input to mkmf
- if you already have an executable, you do not need to call fremake

## frerun

**frerun -x xmlfile experimentname**
- generates a cshell runscript which you can qsub or run interactively
**frerun -r basic -x xmlfile experimentname**
**-r keyword** causes frerun to perform one or more short runs for reproducibility testing
- make sure you have **~/.rhosts** containing the name of your machine, otherwise frerun will not be able to write to /archive.

## frepp

- called at the end of each model simulation year by the model runscript
- creates and submits postprocessing scripts to create time series, time averages, and analysis plots
- can be called 'offline'
**frepp -t1983 -p4 -x xmlfile experimentname**
**-A** option: creates figures only, no data
**-d dir** specifies non-default history directory
**-c split** creates one job per component

## frelist

**frelist -x xmlfile **
- lists all the experiments in an xml file and notes whether they inherit from another experiment
**-v** option: also prints <description> tag contents for each experiment

## frestatus

**frestatus -x xmlfile experimentname**
- prints the status of batch compiles and runs for the experiment
- useful along with **qa -u user -n 45**
**-c** option: status for batch compiles only
**-r **option: status for batch runs only
**-p **option: status for production runs only

## frecheck

**frecheck -x xmlfile experimentname**
- verifies reproducibility by comparing the output of regression test runs with **resdiff**
- prints timing statistics for the regression runs, generates a scaling plot
**-l** option: just list all the output restart files
**-L** option: list restart files with migration status (whether files are online or offline)
**-n** option: perform no dmgets of offline files

## freppcheck

**freppcheck -x xmlfile experimentname**
- checks for missing postprocessing files
- will prompt you for the startYear and endYear that should be checked
**-T** option: check all timeSeries variables
- by default only the first variable is checked

## frepriority

**frepriority -x xmlfile  experimentname**
- manage job priority: queue allocations and job project
**-a queue=num** change number of remaining queue allocations
**-P proj** change job's project
**-s **option: suspend job
**-g** option: 'go' -- submit runscript

## frescrub

**frescrub -x xmlfile experimentname**
- prints report of duplicated archive files for an experiment
**-A** remove netcdf files when a cpio file exists
**-B** remove redundant, uncombined 'raw' history 	files
**-C** clean up $root by deleting excess old scripts
**-D** all of the above cleanup options
**-s** option: automatically qsub the script

## freversion

**freversion in.xml out.xml**
- If there is a change in the XML language, users can use this program to automatically convert their XML files to the new version.
- Uses xsltproc

## Coming Soon: fredb

**fredb -x xmlfile experimentname**
- to add or extract experiments easily to or from the experiment database

## RTS Demo

- RTS: Regression Test Suite
- RTS was also the original name for FRE, which was originally designed to automate large amounts of regression testing
- First, obtain an xml file:
**> cvs co -r khartoum rts**
**cvs checkout: Updating rts**
**U rts/rts.xml**

**Notes:**
add a link

## frelist

**> frelist**
**HSn45**
**n45shallow**
**am2p13_ipcc**
**am2p13**
**am2p13_bin INHERITS FROM am2p13**
**am2p13_stable INHERITS FROM am2p13**
**am2p13_lm3**

## frelist -v

**> frelist -v**
**HSn45**
**-------------------------------------------------**
**A hydrostatic, finite difference primative equation GCM with Held-Suarez physics.**
**n45shallow**
**-------------------------------------------------**
**A B-grid model of shallow incompressible fluid dynamics.**
**...**

## fremake

**> fremake -s am2p13 am2p13_stable**
**your job 121030 ("mk_am2p13") has been submitted**
**your job 121031 ("mk_am2p13_stable") has been submitted**
**> fremake am2p13 **
**TO SUBMIT:  qsub /home/lat/rts/FRE_workshop/rts/scripts/mk_am2p13**
**> qsub /home/lat/rts/FRE_workshop/rts/scripts/mk_am2p13**
**your job 121034 ("mk_am2p13") has been submitted**

## frestatus -c

**> frestatus -c am2p13 **
**>>> /home/lat/rts/FRE_workshop/rts/am2p13/exec/stdout **
**   ****<<< Wed Nov 17 10:57:56 2004**
**NOTE: make successful for am2p13**

## frerun -sr

**> frerun -sr suite am2p13**
**your job 121068 ("am2p13_1x0m8d_15pe") has been submitted**
**your job 121069 ("am2p13_4x0m2d_15pe") has been submitted**
**your job 121070 ("am2p13_1x0m8d_30pe") has been submitted**
**your job 121071 ("am2p13_1x0m8d_45pe") has been submitted**
**your job 121072 ("am2p13_1x0m8d_1pe") has been submitted**
**your job 121073 ("am2p13_1x0m8d_3pe") has been submitted**

## frestatus -r

**> frestatus -r am2p13**
**>>> /archive/lat/MS/FRE_workshop/am2p13/1x0m8d_15pe/ ascii/stdout <<< Tue Nov 16 22:37:28 2004**
**NOTE: Natural end-of-script for /home/lat/rts/FRE_workshop/scripts/am2p13_1x0m8d_15pe.**
**...**

## $root directory structure

**> ls**
**am2p13/  CVS/  rts.xml  scripts/**
**> ls am2p13**
**exec/  include/  src/**

## $archive directory structure

**> cd /archive/lat/MS/FRE_workshop/**
**> ls**
**am2p13/ **
**> cd am2p13**
**> ls**
**1x0m2d_15pe/  1x0m8d_1pe/   1x0m8d_3pe/  **
**1x0m8d_15pe/  1x0m8d_30pe/  1x0m8d_45pe/**
**> cd 1x0m2d_15pe**
**> ls**
**ascii/    history/  restart/**

## $archive directory structure

**lsc6> cd ascii/**
**lsc6> ls**
**ascii_out.cpio   reload_commands  stdout**
**lsc6> cpio -iv<ascii_out.cpio **
**19820101.diag_integral.out**
**19820101.dynam_integral.out**
**19820101.fms.out**
**19820101.logfile.out**
**19820101.xgrid.out**
**256 blocks**

## $archive directory structure

**lsc6> cd history/**
**lsc6> ls**
**19820101.nc.cpio  **
**lsc6> cd ../../restart/**
**lsc6> ls**
**19820103.cpio**

## frecheck

**lsc6> frecheck -n am2p13 am2p13_stable**
**/// am2p13**
**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\**
**NPES    RUNTIME (sec) for am2p13 (1x* runs)**
**---------------------------------------------------------------------**
**1      20311.648 0m8d                         **
**3       6916.245 0m8d                         **
**15      1266.961 0m8d                         **
**30       687.238 0m8d                         **
**45       519.492 0m8d                         **
**--------------------------------------------------------------**
**/archive/lat/MS/FRE_workshop/am2p13/1x0m8d_15pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13/1x0m8d_1pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13/1x0m8d_30pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13/1x0m8d_3pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13/1x0m8d_45pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13/4x0m2d_15pe/restart/19820109.cpio**
**/archive/lat/MS/FRE_workshop/am2p13_stable/1x0m8d_15pe/restart/19820109.cpio**
**/archive/fms/khartoum/am2p13/1x0m8d_15pe/restart/19820109.cpio**
**---------------------------------------------------------------------**

## frecheck (continued)

**////archive/lat/MS/FRE_workshop/am2p13_stable/1x0m8d_15pe/restart/19820109.cpio **
**\\\/archive/lat/MS/FRE_workshop/am2p13_stable/4x0m2d_15pe/restart/19820109.cpio**
**        ****Comparing atmos_coupled.res.nc...**
**        ****Comparing atmos_tracers.res.nc...**
**        ****Comparing bgrid_prog_var.res.nc...**
**        ****Comparing coupler.res...**
**        ****Comparing ice_model.res.nc...**
**	****	****Comparing radiative_gases.res.nc...**
**        ****Comparing soil.res.nc...**
**        ****Comparing strat_cloud.res.nc...**
**        ****Comparing vegetation.res.nc...**
**         ****PASSED: am2p13 am2p13_stable**
**         ****FAILED:**
** ****CRASH DETECTED:**
**NO RUNS TO TEST:**

## Other useful in-house utilities

**ls -1 *.cpio | resdiff**
- extracts each cpio file to a temporary directory and uses cmp to compare each restart file
**ls -1 *.cpio | histdiff**
- extracts each cpio file and uses Remik's utility nccmp to compare each netcdf file
**/home/rsz/util/netcdf/nccmp file1 file2**
- utility for comparing netcdf files
- several options for omitting metadata comparison, excluding certain variables, etc

## Where to go for more info

**http://www.gfdl.noaa.gov/fms/fre**
- FMS webpage -> **FRE Usage**
- An (alpha) version of FRE is available for public download at http://fms.gfdl.noaa.gov/
**oar.gfdl.fre@noaa.gov **mailing list
- Amy.Langenhorst@noaa.gov

**Notes:**
add a link

## Work towards a new version

- New XML structure (work in progress)
- http://www.gfdl.noaa.gov/fms/fre/dev/fre.xml
- New features:
- Explicit support for model development
- Support for generating multiple executables from a single source definition
- Use of libraries to reduce compilation overhead
- Greater flexibility in specifying model run units for both development and production (i.e. steps, seconds, minutes, hours, days, etc)
- Script output tailored to intended use
- Interpretation of dates so that datasets and other input can change with model date
- Support for ensembles
- Provide permanent record of production run information through integration with GFDL Data Curator
- Enhanced regression testing capabilities
- Enhanced post-processing capabilities; more control over time units of post-processing
- Support for FRE environment running on remote hosts
- Support for model implemented checkpoint / restart (this will require major  revisions to the FMS infrastructure)

**Notes:**
add a link