# Presentation: 136102

## GSI Compilation, Coding, and Updates

- Russ Treadon
- NOAA/NWS/NCEP/EMC

- GSI User Orientation
- 4-5 January 2005

- Room 209, WWB
- Camp Springs, MD

## General disclaimer

- What follows is a collection of my views and experiences from
- development within the EMC data assimilation team
- a growing collaboration with GMAO
- Open to (seeking) suggestions in the area of code management & support
- Expect growing pains
- Practical realities of workplace limits what can be done
- Be patient and understanding

## Overview

- GSI code
- Compilation
- Coding Structure/Standards
- GSI updates
- How to submit
- How to get

## GSI Code - Compilation

- Libraries (*info below based on NCEP AIX build*)
- NCEP libraries
- BUFR  libbufr_d_64.a
- Used to read observation files
- W3  libw3_d.a
- Used to manipulate date/time
- Splib  libsp_d.a
- Used in global mode to perform grid  spectral transforms
- Brightness temperature assimilation forward models
- CRTM libcrtm.a
- Contains code for radiative transfer model and jacobian
- IRSSE libirsse.a
- Contains code for IR sea-surface emissivity model

## GSI Code - Compilation

- Libraries (*based on NCEP AIX build*)
- NetCDF
- Currently using version 3.5.0
- May obtain from _[http://my.](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[unidata](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[.](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[ucar](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[.](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[edu](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[/content/software/](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[netcdf](http://my.unidata.ucar.edu/content/software/netcdf/index.html)__[/index.html](http://my.unidata.ucar.edu/content/software/netcdf/index.html)_
- WRF I/O
- Currently using
- $(WRFPATH)/frame/internal_header_util.o $(WRFPATH)/frame/pack_utils.o
- $(WRFPATH)/main/libwrflib.a
- $(WRFPATH)/external/io_int/libwrfio_int.a
- $(WRFPATH/external/io_netcdf/libwrfio_nf.a
- More information available at _[http://www.](http://www.wrf-model.org/)__[wrf](http://www.wrf-model.org/)__[-model.org/](http://www.wrf-model.org/)_

## GSI Code - Compilation

- Source code for all libraries *except* NetCDF and WRF I/O are available on _[GSI web site](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/)_

## GSI Code - Compilation

- Machine dependent configuration files
- Current configurations
- Makefile.conf.AIX
- Makefile.conf.AIX.omp
- Makefile.conf.IRIX
- Makefile.conf.IRIX64
- Makefile.conf.Linux
- Makefile.conf.Linux.IA64.ifort
- Makefile.conf.OSF1
- If your environment is not listed above, you need to create appropriate Makefile.conf.YOUR_MACHINE

_**NOTE:**_  	I have only
- 	worked with
- 	AIX compilation

## GSI Code - Compilation

- Simple make
- Make or check machine dependent configuration file for your machine
- As neeed
- Install GSI libraries
- Edit compiler & load options, library & include paths/names
- Type ./configure in source code directory
- Links Makefile  makefile
- Links Makefile.conf to appropriate configuration file
- Type “make” or “make –f makefile”
- Successful build creates executable named global_anl

## GSI Code – Structure

- Language
- Fortran 90/95 (and beyond)
- Minimize use of machine-specific Fortran extensions
- Routines with machine-specific code suffixed .F90  pass through pre-processor during compile
- Use Modules
- Why?
- Improves code organization
- Eases maintenance/development
- Keeps subroutine argument lists short

## GSI modules

- Quick alphabetical list of GSI modules
- berror:  variables/code related to background error
- constants:  fixed and derived constants
- gridmod:  variables/code related to grids
- guess_grids:  variables/arrays containing guess (and analysis updated) grids
- jfunc:  variables/code related to inner loop

## GSI modules

- Listing (continued)
- kinds:  type declarations
**PLEASE** *use these declarations*!
- mpimod:  variables/code related to MPI
- obsmod:  variables/code related to obs processing
- oneobmod:  variables/code related to single observation test

## GSI modules

- Listing (continued)
- ozinfo:  variables/code related to use of ozone data
- pcpinfo:  variables/code related to use of precipitation data
- qcmod:  variables/code related to observation quality control
- radinfo:  variables/code related to use of brightness temperatures

## GSI modules

- Listing (continued)
- regional_io:  driver for code related to regional GSI input/output
- satthin:  variables/code related to thinning of brightness temperature data
- specmod: variables/code related to grid  spectral transforms

## GSI Code – Structure

- Strive to maintain *logical* organization
- Use existing modules when possible
- If new functionality developed, consider organizing it within a new module
- Use meaningful routine and variable names
- Proper placement of functionality
- For example,
- Observational data reads done from calls to routines named read_*.f90 called from read_obs.f90
- Analysis equation setup is done from calls to routines named setup*.f90 from setuprhsall.f90
- Gradient calculations are made in routines named int*.f90 (int*_qc.f90) from intall.F90 (intall_qc.F90)

## GSI Code – Structure

- Strive to maintain c*onsistent* organization
- Make use of constants defined in constants.f90
- Make use of variable type definitions in kinds.f90
- Use implicit none in ALL routines
- Initialize variables and arrays
- Don’t assume compiler / machine does this
- Document all code (more on this later)

## GSI Code – Structure

- Efficiency is critical for NCEP operations
- Operational wall times
- Global:  15 ~ 20 minutes
- Regional:  7 ~ 10 minutes
- Make good use of parallel nature of code
- Read observations (regional guess) in parallel
- Subdivide analysis domain based on perceived computational load
- Spread analysis variables across various tasks when applying background error
- ...

## GSI Code – Structure

- Efficiency is critical for NCEP operations (continued)
- Code efficient loops
- Access data in stride (sequentially)
- Keep loop size manageable
- Limit subroutine calls from loops
- Consider limited amount of loop unrolling
- Maintain efficient input/output
- May benefit from use of MPI-IO
- Efficiency considerations may push certain tasks to post-processing job steps following analysis

## GSI Code – Structure

- Documentation
- Two accepted formats
- NCEP docblock
- e.g., gsimain.f90, gsisub.f90, glbsoi.f90, pcgsoi.f90
- ESMF ProTeX-compliant format
- e.g., gridmod.f90, wrsiga.f90, spruv.f90, mpimod.f90
- For more details read *ESMF Developer's Guide*  _[http://www.](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[esmf](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[.](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[ucar](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[.](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[edu](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[/](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[esmf](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[_docs/dev_guide.](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)__[pdf](http://www.esmf.ucar.edu/esmf_docs/dev_guide.pdf)_
- Also need extensive documentation within code
- Explain logical sections of code
- Provide enough detail so someone (*like me!*) not familiar with code can understand what is being done

## GSI Code – NCEP docblock

- !$$$  subprogram documentation block
- !                .      .    .                                       .
- ! subprogram:    sprp        gross check, stats, save info for inner loop
- !   prgmmr: weiyu yang       org:  np20               date: 1998-04-07
- !
- ! abstract: For surface pressure observations, this routine
- !              a) applies a gross check
- !              b) prepares arrays for inner loop minimization
- !                 (specifically, prepare interpolation weights and
- !                  innovations for use in int* and stp* routines)
- !              c) loads arrays with sums uses for o-g statistics
- !
- ! program history log:
- !   1998-04-07  weiyu yang
- !   1999-08-24  derber, j., treadon, r., yang, w., first frozen mpp version
- !   2004-06-17  treadon - update documentation
- !   2004-10-11  parrish - nonlinear qc
- !   2004-11-16  treadon - add huge_r_kind limit on obs error
- !   2004-11-22  derber - remove weight, add logical for boundary point
- !   2004-12-23  treadon - restructure code to exit early for o-a call
- !
- !   input argument list:
- !     pdata    - obs info at obs locations for given mpi task (geographic region)
- !     nn_p     - number of data elements per obs
- !     npsdta   - number of obs
- !     luse     - flag indicating if ob is used in penalty
- !     mype     - mpi task id
- !
- !   output argument list:
- !     awork    - array containing information for data counts and gross checks
- !     pwork    - array containing information about obs-ges statistics
- !
- ! attributes:
- !   language: f90
- !   machine:  ibm RS/6000 SP
- !
- !$$$

- Name, org, date info

- Brief description

- Extended description
- of routine function

- Please update program
- history log as you
- make changes

- Description of routine
- input and output

- Language, machine
- code developed with/on

## GSI Code – Updates

- Frequency
- As quickly as every month
- Update usually available 1st half of month
- Updates due COB last Friday of each month
- Reminder email sent out
- Monday prior to Friday due date
- Friday due date
- If you want your group point of contact (more about POC later) to receive update reminder emails
- send email to _[russ.treadon@noaa.gov](mailto:russ.treadon@noaa.gov)__ stating so_

## GSI Code – Updates

- Requirements
- Updates must be made in most recently released version
- Updates made to previously released versions WILL NOT BE ACCEPTED
- Developer must complete and submit change log
- Model change log on those posted on GSI web site  _[http://www.](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[emc](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[.](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[ncep](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[.](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[noaa](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[.](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[gov](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[/](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[gmb](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[/](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[treadon](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[/](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[gsi](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[/changes/](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[gsi](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[_changes.](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)__[htm](http://www.emc.ncep.noaa.gov/gmb/treadon/gsi/changes/gsi_changes.htm)_
- Failure to submit completed change log will delay/deny acceptance/merger of your changes in the GSI code
- Group POC sends change log and source code to _[russ.treadon@noaa.gov](mailto:russ.treadon@noaa.gov)_

## GSI Code – Updates

- Change log contents
- Author/date
- Who made the changes and when completed
- Reason for changes
- Why were these change made?
- Special notes related to changes
- Highlight critical changes here ... things other users should know before running your code
- For example,
- Changes in fixed file format
- Change in namelist variables that renders old scripts obsolete
- Changes in format of analysis output or diagnostic files

## GSI Code – Updates

- Change log contents
- Expected differences
- What differences should users expect when running your code?
- List of files
- Removed
- Added
- Modified
- For each of the above lists, provide a brief (one or two line) explanation
- Removed:  	Why removed?
- Added:  	What was added?
- Modified:  	What was modified?

## GSI Code – Updates

- Change log contents
- Makefile changes
- Includes changes to Makefile, Makefile.dependency, Makefile.conf
- For each of the above changes, provide a description of what changed
- Script changes
- Provide description of what changed
- Fix file changes
- Provide description of what changed

## GSI Code – Updates

- Change log contents
- Test results
- Code must be run using previous (control) and updated versions of the code
- For each version, must run
- Canned global canned
- Canned regional case (at least one configuration)
- For each run indicate
- Script run, machine run on
- For control and updated codes provide
- Maximum resident set size
- Wall clock time
- Initial penalty & gradient information
- Final penalty & gradient information

## GSI Code – Updates

- Things to note:
- If updates from various developers conflict,
- I’ll try to resolve the conflict myself
- Contact developers to resolve conflict
- Sending me  your change log and code *does not guarantee* that your changes will get into the next GSI release
- Changes may be returned for various reasons
- Inadequate documentation
- Inefficient code or generally poor coding
- Changes deemed neither necessary nor beneficial to overall GSI development
- Failure to follow GSI coding standards
**NOTE**:  These standards are likely to become more stringent in the future

## GSI Code – Updates

- Once updates are merged, a new version will be released
- If you want your group POC to receive update notification emails, have him/her register as outlined on the GSI web site

## GSI Code – Updates

- Releases available via
- NCEP CCS
- limited to those with CCS account
- GSI web site
- Method most external users access code
*SourceMotel *?
- Possible future entry point, not currently available
- Web-based entry into CVS repository

## Support

- GSI code made available on an “*AS-IS*” basis
- Code changes with each update  no commitment to maintain/support previous versions
- Limited personnel precludes extensive user support
- Burden for resolving machine specific questions rests on user (Please share what you learn as it may benefit others)
- Problems with older versions may be resolved in subsequent updates  try latest version before sending support email

## Support

- Suggested support model
- Each group identifies one or two local points of contact (POC)
- User questions
- Channel local user support questions through POCs to _[russ.treadon@noaa.gov](mailto:russ.treadon@noaa.gov)_
- Do not send support questions to others in EMC GSI group
- Response
- Support questions will be answered as priorities dictate
- As needed, I’ll contact EMC personnel to help answer questions
- Response will be sent to local POC
- Goals of this approach
- Train POCs to be local GSI “experts”
- Allow EMC scientists to focus on development, not user support

## Slide 31

## Additional slides

- The slides that follow were added as after-thoughts following the presentation
- Included with presentation
- For additional information
- Stimulate further discussion

## Current GSI code status

- GSI code is a ***developmental***, as-of-yet non-operational, system
- Code can (has) changed quickly, but recent trend is for smaller, isolated changes
- Reluctant to label any developmental version of the GSI code as an “official” release
- “official” releases  those versions implemented in NCEP operations

## GSI in NCEP operations

- Implementation requirements
- Improve NWP guidance
- Improvements quantified by various measures
- Objective:  forecast skill scores, O-G fits, ...
- Subjective:  forecaster / field evaluation
- Improvements must be robust
- Improving “A” should not degrade “B”
- Performance needs to be consistent over all cases
- Multiple daily runs expose code to numerous scenarios
- Analysis output drives / affects other systems
- Code runs within scheduled operational time window
- Minimally acceptable documentation

## GSI general goals

- GSI development goals
- Exceed NCEP implementation requirements
- Innovative approaches (solid science) to address various data assimilation issues
- Caveat:  creative development needs to be consistent with operational realities
- Readable, easily understood, largely machine independent code
- Healthy amount of documentation
- Modular, logical, consistent design
- make good use of Fortran 95/?? features
- Modules
- Structures
- ...

## Working together

- Coordination/integration of development across various groups will be a challenging, ongoing process
- Each group has different
- Resources:  people and computing environments
- Objective:  measures of “success”
- Perspectives & personalities
- Need to delineate between essential and extra
- Desire for coordinated, focused development without stifling creativity
- How are development decisions made?
- How to differentiate essential from extra?

## Code management

- Need to track, merge, test changes
- Develop/enforce standards
- What are the standards?
- Who has authority to enforce?
- Desire open access balance by need to control what winds up in updates/new versions?
- How are these decisions made?
- Different groups likely have different needs
- How do we handle (manage/coordinate/merge) these special needs
- NCEP operational branch + Research branch?
- Beyond GSI code, can/do we share utilities?
- Visualization, verification, monitoring tools

## Responsibilities

*Central* -vs.- *local* responsibilities
- Code sits in a central location for everyone to access
- Certain tasks need to be done centrally  which tasks?
- Need dedicated resources (people & hardware)
- What is expected from local developers?
- What obligations (understandings) exist between central repository “keeper” and local developers?
- Support
- When questions arise, who do you contact?
- Compilation/runtime errors:  machine problems
- code structure & functions:  code problems
- File issues:  data format problems
- What expectations should users have in terms of support
- Need dedicated support personnel