National Centers for Environmental Prediction

W/NP11, NSC, Room 307

Camp Springs, MD

November 20, 2006

MEMORANDUM FOR: NCEP Programmers

FROM: John Ward

Chief, Production Management Branch

NCEP Central Operations

SUBJECT: Policy for Operational Implementations on the IBM Central
Computer

System

This memorandum outlines the policy for operational implementations on
the IBM SP. Please carefully review and follow the standards outlined
here. Any JIFs requesting implementation on the IBM Central Computer
System (CCS) that don\'t adhere to this policy will be rejected. There
will be no exceptions. If we determine that parts of this policy should
be changed, we will notify programmers of the change.

When we receive a JIF requesting an implementation on the IBM CCS, and
these steps have been followed, we will copy the contents of your
directory to our test directory. No work will be done in your directory.
We require a minimum of two weeks lead time to prepare and test changes,
thus reducing the potential for errors and providing sufficient time to
advertise the change. In general, requests for change to operational
codes received by noon on Thursday will be scheduled for implementation
no earlier than the second Monday after the JIF is received. **Do not
submit** **JIF requests until the following steps are completed.**

> 1\) All source codes must be placed in a directory that can be
> accessed by NCO\'s implementation staff. (Do not put files in HPSS).
> All of the source code (main program and subroutines) required to
> execute your program **must** be included in this directory as
> separate files (no concatenated files). Please include **only** the
> source code required to execute the program. (We will not selectively
> copy routines from your directory.) We will not include or link source
> or executables from private libraries. To ensure that the latest
> version of the source code is used, programmers should modify the
> operational source codes which they have copied from /nwprod.
>
> 2\) Your directory must also contain the makefile(s) needed to create
> executable modules. If different compiler options are needed for each
> subroutine, then it is the programmer\'s responsibility to include
> these options in the makefile. See Attachment I for information
> regarding makefiles.
>
> 3\) All programs that do any I/O operations **must** adhere to the
> standard unit number conventions listed in Attachment II.
>
> 4\) The location of the script, necessary to execute the code, must
> also be listed on the JIF if it is being changed. To ensure that the
> latest version of the script is used, programmers should modify the
> operational scripts which they have copied from /nwprod.
>
> 5\) Your directory must also contain any new or modified parameters or
> fixed fields needed by the code.
>
> 6\) DOCBLOCKs must be present in all main programs. If DOCBLOCKS are
> missing, the JIF will be returned to you.

7\) When preparing operational scripts, the following standard must be
followed. See

Attachment III for examples and general information which will be
helpful in developing

production scripts.

1\. Use POSIX Shell (/bin/sh).

2\. Obtain the NCEP production dates, using the Y2K compliant dates in

/com/date or by using the setpdy.sh production utility located in

/nwprod/util/ush.

> 3\. Utilize standard production environmental variables. Utilize
> standard
>
> production file and directory naming conventions.
>
> 4\. Each block of moves or copies from the scratch directory to /com,
> /nwges or /pcom must be wrapped with logic testing for the presence of
> the variable SENDCOM.
>
> 5\. Each block of dbnet alerts must be wrapped with logic testing for
> the presence of the variable SENDDBN.

6\. Each execution of a C or FORTRAN code must be wrapped with the use

> of production utilities prep_step, startmsg and err_chk. The standard
> error
>
> should be redirected to a file named errfile in the current working
> directory.
>
> The standard output of each execution should be appended to \$pgmout
>
> (standard production variable).

7\. In each system of scripts, the top level script sets the standard
production

environmental variables and resides in /nwprod/jobs with the standard

naming convention of JXXXX.sms.prod. The top level J-job script called
the

main driver script which resides in /nwprod/scripts with the standard
naming

convention of exXXX.sh.sms. Any needed sub-scripts to the main driver

script will be located in /nwprod/ush.

8.  Production utilizes a centralized cleanup and creation of
    directories in

> /com and /nwges. Production scripts should not remove or create
> production directories at the /com/\$NET/\$envir/\$RUN.\$PDY level.

**ATTACHMENT I**

Makefiles provide the rules to the make utility. The following makefile
shows the preferred format, but programmers are free to create (and
test) their own.

Makefile example, where progname=the name of your code:

SHELL= /bin/sh

\#

\#

SRCS = progname.f subroutine.f subroutine2.f

OBJS = progname.o subroutine.o subroutine2.o

\# Tunable parameters

\#

\# FC Name of the Fortran compiling system to use

\# LDFLAGS Options of the loader

\# FFLAGS Options of the compiler

\# LIBS List of libraries

\# CMD Name of the executable

\#

FC = Xlf_r

LDFLAGS =

LIBS = -L/nwprod/lib -lw3_4

CMD = progname

FFLAGS = -O3

\# Lines from here down should not need to be changes. They are

\# the actual rules which make uses to build CMD.

\#

all: \$(CMD)

\$CMD): \$(OBJS)

\$(FC) \$(LDFLAGS) -o \$(@) \$(OBJS) \$(LIBS)

clean:

-rm -f \$(OBJS)

clobber: clean

-rm -f \$(CMD) \$(CMD).prof

void: clobber

-rm -f \$(SRCS) makefile

**ATTACHMENT II**

1\) Units 1 through 4, 7 through 10, and 50 are reserved for future use.

2\) Use units 5, 11-49 for all INPUT files; i.e., all files containing
data created prior to the execution of the program.

3\) Use units 6, 51-79 for all OUTPUT files; i.e., all files containing
data for subsequent programs to use.

4\) Use units 80-94 for all WORK files; i.e., all files that are written
and read in the same program but have no further use.

**Note:Units 95-99 can now be used as work files.**

Except for work files, the same unit number should NEVER be used for
both input and output by the same program.

**Note:**

Users should associate filenames to unit numbers in the shell script
prior to program execution. On the IBM CCS, users should us the
environmental variable **XLFUNIT\_#**.

Example:

export XLFUNIT_16=\"inputfilename\"

export XLFUNIT_60=\"outputfilename\"

**ATTACHMENT III**

**1.1 Overview**

**1.2 Directory Structure**

Table 1-1 shows an overview of the directory structure necessary to run
production.

*Table 1-1 Production Directories*

  ----------------------------------- -----------------------------------
  **Directory**                       **Description**

  /nwprod                             Production Applications

  /nwtest                             Test Applications

  /nwpara                             Parallel Applications

  /nwbkup                             Backup of Production Applications

  /nwges                              Model Spin-Up Data

  /com                                Data and Application Output

  /dcom                               Incoming Data

  /pcom                               Outgoing Data
  ----------------------------------- -----------------------------------

**1.2.1 Structure of Application Directories**

Table 1-2 shows an overview of the application directories. The
directory names are the sub-directories within /nwprod, /nwtest and
/nwpara.

*Table 1-2 Application*

  ----------------------- -----------------------------------------------
  **Directory**           **Description**

  jobs                    Wrapper Scripts (J-Jobs)

  scripts                 Main Driver Scripts (ex-scripts)

  fix                     Static Input Data

  parm                    Static Input Data

  exec                    Executables

  sorc                    Source Code

  util                    Utilities spanning multiple applications;
                          contains scripts, fix, parm, exec and sorc
                          subdirectories.
  ----------------------- -----------------------------------------------

**1.2.2 Structure of /nwges Directory**

Several of the weather forecast models running in production produce
output to be used later as input for subsequent model runs. This select
set of critical output data used to begin model runs is often referred
to as model guess fields. The model guess fields are stored in /nwges.
Table 1-3 shows the directory stucture of /nwges.

*Table 1-3 /nwges Directory Structure*

  ----------------------------------- -----------------------------------
  **Directory**                       **Description**

  prod/*model_name*.*YYYYMMDD*        Production Spin-up data for model

  test/*model_name*.*YYYYMMDD*        Test Spin-up data for model

  para/*model_name*.*YYYYMMDD*        Parallel Spin-up data for model
  ----------------------------------- -----------------------------------

**1.2.3 Structure of /com Directory**

The /com directory contains output data, stdout and stderr from
production jobs. The default resident time for data in /com is ten days.
Table 1-4 shows the directory structure of /com.

*Table 1-4 /com Directory Structure*

  ----------------------------------------- -----------------------------------
  **Directory**                             **Description**

  *model_name*/prod/*net_name*.*YYYYMMDD*   Production Model Output for a day

  *model_name*/test/*net_name*.*YYYYMMDD*   Test Model Output for a day

  *model_name*/para/*net_name*.*YYYYMMDD*   Parallel Model Output for a day

  output/prod/*YYYYMMDD*                    Job stdout/stderr for a day

  output/test/*YYYYMMDD*                    Job stdout/stderr for a day

  output/para/*YYYYMMDD*                    Job stdout/stderr for a day

  logs                                      Log Files
  ----------------------------------------- -----------------------------------

**1.3 Standard Environmental Variables**

Inside of the production scripts there are environmental variables
reserved for production use. A majority of the production utilities rely
on the use of these standard variable. These variables are set inside of
the production wrapper scripts. Unsetting or programing around these
variables inside of driver and supporting scripts may result in an
undesired job outcome making it difficult to troubleshoot. Table 1-5
shows the list of the standard environmental variables used in
production.

*Table 1-5 Standard Environmental Variables*

  ----------------------------------- -----------------------------------
  **Variable Name**                   **Description**

  PDY                                 Today's Date formatted YYYYMMDD

  PDYm1-7                             Date 1-7 days ago formatted
                                      YYYYMMDD

  PDYp1-7                             Date 1-7 days ahead formatted
                                      YYYYMMDD

  DATA                                Temporary Scratch Directory

  jlogfile                            Logfile of start and end time of
                                      all jobs

  outid                               Job id appearing in jlogfile

  jobid                               Jobid appearing in scratch
                                      directory name

  pgmout                              Name of stdout file for all
                                      programs in a job

  cycle                               Model Cycle time formatted tHHz

  cyc                                 Model Cycle time formatted HH

  SENDCOM                             Enable/Disable file copying to /com

  SENDDBN                             Enable/Disable DBNet Alerts

  SENDSMS                             Enable/Disable SMS hooks

  NET                                 Model Name

  RUN                                 Type of Model Run

  pcom                                Directory for copies to /pcom

  COMIN                               /com directory for data input

  COMOUT                              /com directory for data output

  GESdir                              /nwges directory for read and write

  utilities                           Directory containing utility
                                      scripts

  utilexec                            Directory containing utility
                                      executables

  EXEC*model_name*                    Directory containing model
                                      executables

  FIX*model_name*                     Directory containing model fix
                                      files

  PARM*model_name*                    Directory containing model
                                      parameter files

  USH*model_name*                     Directory containing supporting
                                      model scripts

  SMSBIN                              Directory containing SMS
                                      executables
  ----------------------------------- -----------------------------------

**1.4 Standard File Naming Conventions**

In production the file name should represent the name of the model run,
the cycle of the model run, the type of data the file contains and the
forecast hour the data represents. Filenames should not contain the date
as the directory in which it resides already represents the date.
Filenames should not contain uppercase characters.

Example:

gfs.t\${cyc}z.pgrbf\${fhr} where cyc is the cycle and fhr is the
forecast hour.

**2.1 Basic Production Utilities**

There are several utilities available in production to help you
incorporate the basic job functionality required to meet operational
standards. This section is intended to introduce you to the basic
utilities used by most production jobs.

**2.1.1 Date Utilities**

***finddate.sh***

Given a date, ***finddate.sh*** will return date a specified number of
days before or after the provided date. finddate.sh will also provide a
sequence of dates leading to the specified number of days before or
after the provided date.

Example 2-1*.*

*Example 2-1 Script Using finddate.sh*

#!/bin/sh

utilscript=/nwprod/util/ush

today=20020101

\# Single Date Example

ten_days_ago=\`sh \$utilscript/finddate.sh \$today d-10\`

ten_days_ahead=\`sh \$utilscript/finddate.sh \$today d+10\`

\# Sequence Example

last_four_days=\`sh \$utilscript/finddate.sh \$today s-4\`

next_four_days=\`sh \$utilscript/finddate.sh \$today s+4\`

echo "Today's Date is \$today"

echo

echo "The date ten days ago was \$ten_days_ago"

echo "The date in tens days will be \$ten_days_ahead"

echo

echo "The last four days where \$last_four_days"

echo "The next four days are \$next_four_days"

*Example 2-1 Output*

Today\'s Date is 20020101

The date ten days ago was 20011222

The date in tens days will be 20020111

The last four days where 20011231 20011230 20011229 20011228

The next four days are 20020102 20020103 20020104 20020105

***setpdy.sh***

***setpdy.sh*** is a shell script to help you set the variables PDYm1-7,
PDY and PDYp1-7. This utility will output a file PDY in the current
working directory which can be sourced in the parent script to set the
PDY variables. ***setpdy.sh*** expects to environmental variable cycle
to be set when executed. The default centered date is the current days
date. If the environmental variable PDY is set when executed, the
centered date will be the value of PDY.

This utility script uses date files in /com/date set by production jobs
/prod00/ncepibm00/j100_00 and /prod12/ncepibm12/j100_12 run at 2330 UTC
and 1130 UTC respectively. At 2330 UTC the date files for cycles 00-11
UTC are set ahead to the next day. At 1130 UTC the date files for cycles
12-23 UTC are set ahead to the next day. Therefore, if you were to set
cycle to t12z and run setpdy.sh between 2230 and 1130 UTC, you would get
a PDY file centered on the previous days date. This is because the 12
UTC cycle has not started. This has been done by design to allow 12 UTC
production jobs to be run late into the 00 UTC cycle.

Example2-2.

*Example2-2 Script Using setpdy.sh*

#!/bin/sh

export utilscript=/nwprod/util/ush

\# If PDY is not set, the dates would be centered based off the current
cycle date.

\# Try running with PDY not set to see what happens.

export PDY=20020101

export cycle=t12z

\$utilscript/setpdy.sh

. PDY

*Example2-2 Contents of File PDY*

export PDYm7=20011225

export PDYm6=20011226

export PDYm5=20011227

export PDYm4=20011228

export PDYm3=20011229

export PDYm2=20011230

export PDYm1=20011231

export PDY=20020101

export PDYp1=20020102

export PDYp2=20020103

export PDYp3=20020104

export PDYp4=20020105

export PDYp5=20020106

export PDYp6=20020107

export PDYp7=20020108

**2.1.2 Logging and Error Checking Utilities**

***setup.sh***

To properly execute a program inside of a production script you must use
runtime compiler options to pass the program its unit assignments, log
its start and stop time, check its return code and execute appropriate
SMS hooks respective to the return code. This all sounds daunting but
***setup.sh*** will assist you in meeting these standards by gathering
the needed utilities into your scratch area. After running this script,
the utilities prep_step, err_chk, err_exit, postmsg and startmsg will be
available for use. These five utilities are described below.

***prep_step***

In production you must use the runtime compiler options and variables to
pass a FORTRAN program its unit assignments. For the IBM SP, the
environmental variable XLFUNIT\_*numberi* is used to pass unit
assignments to the program. Since there may be multiple FORTRAN programs
running inside of a job, these variables must be reset before each
program execution. Running prep_step before each program execution will
set the variable XLFRTEOPTS to enable the use of the variable
XLFUNIT\_*number*, and will unset all XLFUNIT\_*number* variables
currently set in the environment.

***postmsg***

***postmsg*** simply writes a message to a log file. The first argument
is the log file name and the second argument is the message. You should
use the log file named /com/logs/jlogfile when using ***postmsg*** in a
production job.

***err_chk***

The script ***err_chk*** is used to check for a non-zero return code of
a program execution and run a series of commands based on this return
code. If a program executes with a return code of zero the end time is
logged and job execution continues. If a non-zero return code is found
stdout/stderr are written to the job output log, the time of the error
is logged, an abort flag is sent back to SMS and the job is cancelled.
The return code is passed into ***err_chk*** by setting the
environmental variable err.

***err_exit***

The script performs the same tasks as a non-zero return code passed to
err_chk

***startmsg***

***startmsg*** simply posts the start time of the program to be executed
to a log file. The name of the log file is set through the standard
environmental variable called jlogfile.

Example2-3.

***Example2-3*** shows you how to create a simple job script to execute
a FORTRAN program using the utilities described above. A majority of the
environmental variables set are standard variables used by these
production utilities as listed in *Table 1-5.* To run this example
script as a batch job yon must use the llsubmit command on the IBM SP.
This example will create a job output file, jlogfile and subdirectory in
your current working directory. The file called jlogfile is a log of the
start and end times of the job and FORTRAN executable. Inside of the
subdirectory will be all the utilities discussed above plus the input
and output files created by the script and executable. Try to run this
script as listed and review the output files and subdirectory. Then, try
running the script with the input file to the FORTRAN program missing
and review the output.

*Example2-3 Script Using Utilities from setup.sh*

\# @ job_name = j242_12

\# @ output = ./WW0242_12.o\$(jobid)

\# @ error = ./WW0242_12.o\$(jobid)

\# @ initialdir = ./

\# @ class = 1

\# @ min_processors = 1

\# @ max_processors = 1

\# @ notification = never

\# @ shell = /bin/sh

\# @ queue

export outid=\"example2‑3\"

export jobid=\"\${outid}.o\$\$\"

export pgmout=\"OUTPUT.\$\$\"

\#

\# Set cycle time for setpdy.sh to figure out the date

\#

export cyc=12

export cycle=t\${cyc}z

\#

\# Set Name of stdout/stderr file for executables

\#

export pgmout=\"OUTPUT.\$\$\"

\#

\# Set Temporary Scratch Directory

\#

export DATA=\`pwd\`/\${jobid}

mkdir \${DATA}

\#

\# Run inside of Scratch Directory

\#

cd \${DATA}

\#

\# Disable SMS Hooks

\#

export SENDSMS=NO

\#

\# File to log job information

\#

export jlogfile=../jlogfile

\#

\# Run setup.sh to copy utilities into scratch directory

\#

export utilscript=/nwprod/util/ush

sh \$utilscript/setup.sh

msg=\"Beginning of Example Job Sucessfully\"

postmsg \"\$jlogfile\" \"\$msg\"

\#

\# Set the Date Variables

\#

sh \$utilscript/setpdy.sh

. PDY

\#

\# Try running with the line below commented out to change the

\# outcome of the script

\#

echo \$PDY \> date_file_in

\#

\# Set Standard Variable pgm for use by utilities then

\# source prep_step to set XLFRTEOPTS and unset XLFUNIT vars

\#

export pgm=example2‑3

. prep_step

export XLFUNIT_20=\"date_file_in\"

export XLFUNIT_51=\"date_file_out\"

\#

\# Log start of program execution

\#

startmsg

../example2‑3 \>\> \$pgmout 2\>errfile

\#

\# Check the return code of example2‑3

\#

export err=\$?;err_chk

msg=\"Got to End of Example Job Sucessfully\"

postmsg \"\$jlogfile\" \"\$msg\"

exit

*Example2-3 Source of FORTRAN code example2-3*

PROGRAM EXAMPLE2_3

C\$\$\$ MAIN PROGRAM DOCUMENTATION BLOCK

C . . . .

C MAIN PROGRAM: EXAMPLE2_3

C PRGMMR: MICHAUD ORG: NP11 DATE: 2002‑01‑24

C

C ABSTRACT: PROGRAM READS ONE LINE OF FILE AND WRITES IT BACK OUT

C

C PROGRAM HISTORY LOG:

C 02‑01‑24 D.L.MICHAUD

C

C USAGE:

C INPUT FILES:

C 21 ‑ INPUT FILE

C

C OUTPUT FILES:

C 51 ‑ OUTPUT FILE

C

C EXIT STATES:

C

C REMARKS:

C

C ATTRIBUTES:

C LANGUAGE: FORTRAN 90

C MACHINE: IBM SP

C

C\$\$\$

CHARACTER \* 8 PDY

INTEGER IOS

READ(20,IOSTAT=IOS,FMT=\'(A8)\')PDY

IF (IOS.NE.0) THEN

CALL ERREXIT(IOS)

END IF

WRITE(51,FMT=\'(A8)\')PDY

END

*Example2-3 Makefile for example2-3*

FC = xlf

CMD = example2‑3

LIBS = /nwprod/w3lib90/w3lib_4

LDFLAGS =

PROFLIB =

FFLAGS =

OBJS = example2‑3.o

all: \$(CMD)

\$(CMD): \$(OBJS)

\$(FC) \$(LDFLAGS) ‑o \$(@) \$(OBJS) \$(LIBS)

rm ‑f \$(OBJS)

clean:

rm ‑rf \$(OBJS)

*Example2-3 Sample Output of jlogfile with Successful and Unsuccessful
Runs*

01/24 22:29:21Z example2‑3.o21036‑Beginning of Example Job Successfully

01/24 22:29:22Z example2‑3.o21036‑example2‑3 started

01/24 22:29:22Z example2‑3.o21036‑example2‑3 completed normally

01/24 22:29:22Z example2‑3.o21036‑Got to End of Example Job Successfully

01/24 22:30:21Z example2‑3.o24088‑Beginning of Example Job Successfully

01/24 22:30:22Z example2‑3.o24088‑example2‑3 started

01/24 22:30:22Z example2‑3.o24088‑ERROR PROGRAM example2‑3 RETURN CODE
255

01/24 22:30:22Z example2‑3.o24088‑ FAILED example2‑3.o24088 ‑ ABNORMAL
EXIT