**Research to Operations Guidelines for the Climate Forecast System**

These "Research to Operations (R2O) Guidelines" establish the path to
implementation of changes in the NCEP operational climate model suite.
They establish the roles and responsibilities of the "home research
institution" and NCEP (EMC, CPC and Climate Test Bed) in this process.
Significant resources for the home research institution and the NCEP are
required to carry out a smooth transition. The R2O Guidelines, which
describe the implementation path of changes to the NCEP operational
climate model suite, consist of the following seven steps:

> **1. Model development and refinement**
>
> **2. Preliminary assessment**
>
> **3. Calibration**
>
> **4. Interface with operations**
>
> **5. Final skill assessment**
>
> **6. Parallel tests**
>
> **7. Approval**

While most of the steps are identical to the way NCEP implements changes
in other model suites, the calibration step is unique to the climate
model suite. Some of the changes may not need to go through all of the
steps and, therefore, are negotiable on a case-by-case basis. The three
NCEP center directors (NCO, EMC and CPC) must approve changes to the
procedures. The NCEP director makes the final decision to approve an
implementation. Details of each step are described below.

1.  **Model development and refinement**

The NCEP CFS is a fully coupled system for atmosphere-ocean-land
surface-sea ice. It is expected that any CFS improvements will come from
any or all of these components or from provision of additional,
independent information to supplement operationally available CFS
products. Upgrades to CFS components must be tested with the full CFS
and compared with the operational system to demonstrate any changes to
the system's climatology (using a 30- to 50-year coupled run with CMIP,
Coupled Model Intercomparison Project) protocols, and improved
simulation of the seasonal-to-interannual (El Niño/Southern Oscillation)
and intraseasonal (Madden-Julian Oscillation) signals via a subset set
of 1-year hindcasts (usually initialized in April and November). The
development and refinement of any upgrades are to be performed by the
model host institution, using its personnel and computing resources,
except if the development is supported by the NOAA Climate Test Bed
(CTB). In the latter case, some computing resources may be available for
CMIP runs and a hindcast subset through the CTB allocation on the NOAA
R&D computer. If testing on the retrospective forecast subset indicates
sufficient improvement compared to the operational CFS, a full hindcast
set (initialized for all 12 calendar months using procedures compatible
with NCEP's operations) must be completed before implementation can be
considered. At the present level of computing resources, NCEP and the
CTB cannot support execution of a full, pre-implementation hindcast set.

Upgrades to CFS components may also be tested in the NCEP Global
Forecast System (GFS), which covers the 1-15 day forecast range, and a
future Monthly Forecast System (MFS, 16-45 day range). Both the GFS and
MFS may become fully coupled systems in 2009-2012. Testing system
components in both the weather and climate applications is a major
strategic item at NCEP, and fosters a unified approach to the
weather-climate problem.

The second strategy for improvements to NCEP's climate forecasts is
through provision of additional, independent information to supplement
operationally available CFS products. This may be done by a Multi-Model
Ensemble (MME) approach. Testing any new climate model system must
follow the same strategy as for CFS upgrades outlined above. As with CFS
component upgrades, NCEP and the CTB cannot support execution of a full,
pre-implementation hindcast set with the present level of computing
resources.

Evaluations of all climate (and weather) system upgrades and MME
configurations will be done with standard techniques, documented and
available at NCEP. More details are given in the sections below. EMC
scientists will participate in a consulting role for projects involving
model changes, while CPC scientists will participate in projects
involving evaluations and product development for all upgrades and MME
systems.

**2 Preliminary assessment**

Developers of CFS upgrades and MME candidates should provide the subset
of hindcast results (including raw monthly-mean fields) to NCEP and work
with CPC and EMC to assess whether there are forecast improvements
relative to existing operational dynamical and statistical predictions.
Iterations back to Step 1 may be necessary if the results do not provide
improvements. Statistically significant improvements would lead to an

implementation plan and completion of the full, pre-implementation
hindcast set. A team of scientists from the external institution, CPC
and EMC will work together to make the evaluation. A plan for transition
to operations at NCEP will be written by the team and approved by NCEP
management.

**3 Calibration**

Candidates for operational implementation must provide a full (9-12
month) hindcast

set for the period 1982-present, with 20 or more ensemble members per
month, together with the code and scripts used to generate the
hindcasts. Calibration includes bias correction and statistical
quantification of skill, using cross-validation, for U. S. temperature
and precipitation prediction using standard NCEP procedures available
publicly.

The resources needed to complete such runs are very large. For major
projects such as multi-model ensembles, the host model institution will
provide the necessary resources. Additional resource needs at NCEP for
operational execution will have to be proposed and planned for well in
advance of the implementation process.

Developers will collaborate with CPC and EMC to perform the skill
assessment and cross-validation for these forecasts. The key metric for
success is the ability of proposed upgrade to provide independent,
additional information to the products that are already operationally
available for U. S. temperature and precipitation, including
statistical-empirical methods.

**4 Interface with operations**

As the calibration and assessment work is completed, the provider will
work with

EMC and NCO to make all programs in the model comply with NCO standards
and to make all model output comply with WMO and NCO standard formats.
EMC and NCO will assess the additional computer resources needed to add
the system changes into the operational job stream. For new MME systems,
the provider may need to work with NCEP to obtain additional resources
needed for the operational runs of new systems.

**5 Final skill assessment**

As the calibration runs are completed and the interface with operational
standards is finished, CPC and EMC will work with the provider to
complete the final assessment to determine if the change is ready for
implementation. Assuming production resources are available for the
additional runs, the parallel tests will begin.

**6 Parallel tests**

EMC staff will work with the code provider and NCO to bring all running
scripts up

to NCO production standards. NCO staff will run the new model in the
parallel

production environment and make routine near-real-time forecasts. CPC,
EMC and the

provider will verify that the parallel prediction products are the same
as the provider

intended. CPC forecasters will begin to evaluate the products in
preparation for

operational ISI forecasts.

NCEP will provide the computing resources for this step and EMC and CPC
will participate in all aspects of the final assessment with the
developer.

**7 Approval**

NCO Production Management Branch staff will brief the director of NCEP
on the

technical evaluation of the new model. EMC Global Climate and Weather
Modeling

Branch staff will brief the director of NCEP on the scientific
evaluation of the new

model, and the CPC Prediction Branch staff will brief the director of
NCEP on the

operational evaluation of the forecasts from the new model. Four
categories will be

considered and checked off before the briefing takes place: 1)
Forecast/analysis benefits;

2\) Efficiency Assessment (especially important for NCO and CPC); 3)
Compatibility

with NOAA/NCEP IT assets; 4) Sustainability. Upon approval by the NCEP
director, the

new model will be scheduled for implemen