# Presentation: 006480

## XMclimate – A Local Tool for Climate Services

- CR Climate Services Workshop
- July 8-9, 2003
- Matthew J. Bunkers, SOO UNR

## Objectives

- What is XMclimate and what are some of its applications?
- Where do you get XMclimate, how do you install it (for both HP & Linux), and how do you run it?
- How are data ingested into XMclimate, and what maintenance is necessary for the program?

## Part I

- XMclimate Functionality

## XMclimate Developers

- Carl Dierking, WFO Juneau, AK
- Paul Shannon, WFO Juneau, AK

## XMclimate “Main” Interface

## The “Display Observed” Interface

-  This appears as a second window.

## Page 2

## The “Display Observed” Interface

## The “Display Normals” Interface

- These “Normals” can be entered manually, calculated using a utility in XMclimate, or imported via normals2000 (more on this later).

## The “Display Extremes” Interface

- The “Help”, “Print”, “Prev Month”, and “Prev Year” buttons are intuitive.

## The “Graph Observed” Sub-Interface(you need the XMgraph utility for this)

## Changing Stations

- Click

## Station Selection GUI

- This appears as yet another (i.e., third) window, which you can leave open.
- If you start XMclimate with -idonly, names will not appear.

## What are some applications?

- Somebody calls up and asks:
- “Have we ever been hotter than 105 °F in Rapid City during June or July?”

## Setup the “Search” Criteria

## 10 Records are Displayed

## What if we made this 2?

## You get a “2 DAY MEAN”

## What are some applications?

- Somebody calls up and asks:
- “Our minimum temperature has been less than -10 °F for four days in a row – isn’t that unusual for Rapid City?”

## Setup the “Search” Criteria

## 15 Records are Displayed

## What are some applications?

- Somebody calls up and asks:
- “What are the first, latest, and average dates of sub-freezing temperatures for Rapid City?”

## Setup the “Search” Criteria

## Several Statistics are Displayed

## Editing Existing Data

## Be careful – there is no undo!(But you can password protect the file)

## Rapid City WFO Web Apps

## Research Applications

- E.g., extracting maximum temperature and wind data for SDSM&T students

## Research Applications Cont.

- E.g., extracting temperature & precip data for a fire weather study

## Printing Records

- Script continued
- on next page.

## Printing Records Cont.

## In Summary

- XMclimate is very useful for many different types of climate requests—such as extreme events—from our users (e.g., Media, public, etc.)
- XMclimate is helpful for local forecasting (limits)
- You can maintain a database of 150+ sites of 1st Order and COOP stations (e.g., as does DSM)
- Scripts can be written to extract data for Web applications, and also for research

## Part II

- XMclimate Installation

## Getting Help for XMclimate

- First, there are several _html_ files included with XMclimate.  These are very useful in the setup process—refer to them (the *.Help files are for XMclimate usage only)
- Others who have used XMclimate extensitvely include Craig Cogil (DSM), Mike Gillispie (FSD), and myself (UNR)

## Acquiring & Installing XMclimate

- The HP version is on the AWIPS Local Applications Database (LAD)
- A Linux version is available from:
- 	_[ftp://inetsrv.arh.noaa.gov/pub/apps/xmclimate/](ftp://inetsrv.arh.noaa.gov/pub/apps/xmclimate/)_
- This is not on the LAD; there are also many other applications here, including the HP version of XMclimate

## Options from the LAD

## XMclimate v1.4.6 (for HP)

- _xmclimateSetup.man_ – basic setup file
- _xmclimateDocs.html.tar.Z_ – main program documentation
- _xmclimate020509.hpux10.exe.tar.Z_ – all program files (also contains above files, plus README.changes & README.climate)

## Unpacking/Installing

- Let’s assume you are installing in the /home/xmclimate directory; this is NFS mounted, which means you can access the data files from all workstations using one location
- % cd /home/xmclimate
- (get the appropriate file)
- % gunzip xmclimate020509.hpux10.exe.tar.Z
- % tar –xvf xmclimate020509.hpux10.exe.tar

## Unpacking/Installing Cont.

- % mkdir /home/xmclimate/html
- % mv *.html html (just for housekeeping)
- % mv *.jpg html
- % mkdir /home/xmclimate/data
- % mv O0CL* data
- file naming convention – O0CLxxxxx00000 where “xxxxx” is the station ID (e.g., O0CL_PAJN00000 for Juneau, AK); if the ID is less than 5 characters it is preceded with underscores; these are netCDF files

## Unpacking/Installing Cont.

- % mkdir /home/xmclimate/xmgraph
- % cd /home/xmclimate/xmgraph
- (get the appropriate file)
- % gunzip xmgraph010308.hpux.exe.tar.Z
- % tar –xvf xmgraph010308.hpux.exe.tar

## Running XMclimate

- % /home/xmclimate/xmclimate  -stn  PAJN \
- 	-ddir  /home/xmclimate/data \
- 	-udir  /home/xmclimate/DIR.utl \
- 	-hdir  /home/xmclimate \
- 	-gdir  /home/xmclimate/xmgraph \
- 	-bdir  /awips/dev/xmclimatedata \
- 	-idonly
- This is all one command line argument, but it can be run from a script (other options are also available).
- Put on the ~/.dt/localapps.dtwmrc menu; start with the most commonly used station (e.g., I use KRAP)

## In Summary

- XMclimate (& XMgraph) is easy to download and run
- Initially, no configuration necessary
- There are several supporting files (i.e., XMgraph, normals2000, xm_utils)
- You need to create your own data files – we’ll talk about that next

## Part III

- XMclimate Maintenance

## Creating a New File

- First, an “empty” data file needs to be created
- You need to gather basic station information:
- station name & ID, 6-digit COOP ID, and beginning date (and optionally – latitude, longitude, elevation, etc.)
- Once you create a file, you can’t change the beginning date (you need to create new file)!

## Two Ways to Create New File

- 1st:  Use the climateConvert script in the /home /xmclimate/DIR.utl directory (command line)
- The “toolsDir” variable (near the end of the file) needs to be modified so you can set the correct path to the utility directory
- only one modification needed to this script
- 2nd:  Use the XMclimate GUI (under the File  “Create New Station...” menu option)
- This option is preferred/recommended

## “climateConvert” Script Mod.

## “Create New Station...” GUI

- Sioux Falls

- WFO Sioux Falls, SD

- 397667

- NWS

- SD

- Central

- Minnehaha

- Sioux Falls

- 2359

- 1428

- 43.35

- 96.45

- 1932

- 2003

- KFSD

- Add whatever you want here.

## Data Sources

- Data can be acquired in several ways:
- NCDC COOP Summary of the Day CD-ROMs
- I don’t use this option anymore
- _[http://cdo.ncdc.noaa.gov/plclimprod/plsql/poemain.poe](http://cdo.ncdc.noaa.gov/plclimprod/plsql/poemain.poe)_
- We’ll use this instead
- _[ftp.ncdc.noaa.gov](ftp://ftp.ncdc.noaa.gov/)_
- Used for monthly updates (data about 5 months old)
- Daily RR3 (COOPs) and DSMs (ASOS)
- Used for near real-time collection
- Manually (necessary for some things, such as QC)

## Downloading Web Data

## Downloading Web Data Cont.

## Downloading Web Data Cont.

## Downloading Web Data Cont.

## Downloading Web Data Cont.

## Downloading Web Data Cont.

## Ingesting Initial Data

- After creating a new file and obtaining data, you will need to populate the file; the initial data will likely come from the Web or CD-ROM (we’ll use the Web)
- Make sure you have edited:
- 	/home/xmclimate/DIR.utl/climateConvert
- 	to put in the correct “toolsDir” path, then run the script

## “climateConvert” output

## The Next Step

- Now you have created a new file and populated it with data from the web
- This file may need to be edited for pre-1948 data, as well as for the last 5 months—if desired
- The next step is to setup an automated collection of data, followed by updating extremes and creating “normals”

## The NCDC ftp Server & Ingesting Monthly Data “Automatically”

- A script can be written to automate the collection of monthly data files from NCDC
- This allows you to collect data for all stations in one step, with the latest files typically five months old
- /home/xmclimate/NCDC/get_NCDC.csh
- We run this on the 1st of each month at UNR (see next page)

## Sample Script for NCDC Data

## Ingesting RR3 (COOP) Data

- Edit the script:
-  	/home/xmclimate/DIR.utl/shef2climate.pl
- Change “perl5” to “perl” in the first line
- Update the paths and configuration settings at the beginning of the file
- Make a few fixes (see next page)

## Edit shef2climate.pl

## Add RR3 to Triggers

- FSDRR3RAP  /home/xmclimate/DIR.utl/shef2climate.pl  FSDRR3RAP
- After you localize your triggers, the COOP data files will update everytime an RR3RAP arrives (works with corrections, but doesn’t always work with multiple stations in the same RR3)
- If station doesn’t exist, no big deal

## Ingesting DSM (ASOS) Data

- Edit the script:
-  	/home/xmclimate/DIR.utl/climDSMload.pl
- Change “perl5” to “perl” in the first line
- Update the paths and configuration settings at the beginning of the file (see next page)

## Edit climDSMload.pl

## Add DSM to Triggers

- CYSDSMGCC  /home/xmclimate/DIR.utl/climDSMload.pl  CYSDSMGCC
- FSDDSMRAP  /home/xmclimate/DIR.utl/climDSMload.pl  FSDDSMRAP
- Etcetera...
- After you localize your triggers, the ASOS data files will update everyday (although I have found ~1% missing data)
- This will get you wind data too!

## Manual Editing

- We (UNR) perform manual editing and quality control at the beginning of each month for our “primary” climate sites
- Some files are password protected, thus reducing the chance someone will modify the data inadvertently—doesn’t work on current month
- This can be done with the Edit  “Chg Password” menu
- If you forget password, do an ncdump (HP) on the pertinent data file and grep for “pass”

## Manual Edit GUI

- You must tab through.

## Updating Extremes

- Extremes should be updated each time you import new data into the xmclimate database for a given station
- This can be accomplished in a script, which we do monthly at UNR (see next page)
- This can also be done with the Tools  “Chk Extremes” option with the main XMclimate interface

## Updating Extremes Script

## Creating Normals

- Normals need to be created for each new station, if the period of record is long enough
- can be done with normals2000.pl script, can be done manually, or can be done with XMclimate utility

## In Summary

- Blank files need to be created for each new XMclimate station you desire
- Data can be obtained from the NCDC Web, NCDC ftp, NCDC CD-ROM, RR3’s, and DSM’s
- Some editing of utility scripts is necessary prior to ingest of climate data
- Some maintenance and configuration is necessary to keep the database (i.e., observed, extremes, & normals) current and of high quality

## Potential Problems

- Some older data, especially prior to 1948, require manual input
- Erroneous data in NCDC files (on both web and CDROM); XMclimate works good for QC’ing
- Make sure you have a backup of your data files (can do this in scripts)!
- climDiff is useful for comparing files, especially after NCDC web data are downloaded—so you don’t overwrite good data with bad data

## Potential Problems Cont.

- Can have permission problems with data files (e.g., fxa vs. awipsusr, etc.)
- Several edits are required to the scripts in the /home/xmclimate/DIR.utl directory, especially with respect to the paths, perl5  perl, and configuration settings
- Normals created with XMclimate do not necessarily agree with NCDC normals

## Potential Problems Cont.

- May unwantingly overwrite program files when untarring an upgrade to XMclimate
- Suggest untarring in a separate directory, and then comparing the file creation dates, as well as the file contents (using the diff/sdiff commands)
- Missing data when searching for top 10’s
- If you are missing one day in a given month, a search for monthly extremes will omit that month
- A fix may be forthcoming for this

## Potential Problems Cont.

- NCDC website may only be free for noaa.gov sites (not sure about .edu sites)
- Not everything with the HP version is functional on Linux yet (e.g., ksh scripts)
- There are a few typos in the *.html files

## Final Comments

- XMclimate is a very powerful climate data interrogation tool
- Forecasters and HMTs alike can use this program to help answer a wide array of questions from local citizens, media, and other users
- Other features are available with this program, you just need to read the *.xtml files

## References

- Petrescu, E., 1998:  Creation and maintenance of a comprehensive climate database. NOAA Tech. Memo., NWS WR-255, 28 pp.

## Acknowledgements

- Jeff Chapman, Craig Cogil, Mike Gillispie, and Preston Leftwich provided information used in this presentation