**Research to Operations Steps**

The CTB "Transition to Operations Guidelines" establish the path to
implementation of changes in the NCEP operational climate model suite.
They establish the roles and responsibilities of the "home institution"
and the CTB in this process. The Guidelines are also found on the CTB
website
([www.cpc.ncep.noaa.gov/products/ctb](http://www.cpc.ncep.noaa.gov/ctb)).

The Path to implementation of changes in the NCEP operational climate
model suite consists of the following seven steps:

1.  **Model development and refinement**

2.  **Preliminary assessment**

3.  **Calibration**

4.  **Interface with operations**

5.  **Final skill assessment**

6.  **Parallel tests**

7.  **Approval**

While most of the steps are identical to the way EMC implements a
majority of the changes in other suites of models, the calibration step
is unique to the climate model suite. Some of the changes may not need
to go through all of the steps. The three NCEP center directors (NCO,
EMC and CPC) will have to approve changes to the procedures. In the end,
the NCEP director will make the final decision to approve a change.
Details of each step are described below.

**1 Model development and refinement**

The key issue is additional, independent information to supplement
operationally available products. The development and refinement are to
be performed at model host institutions. For funded transition projects
with a commitment of computer resources from the Climate Test Bed (CTB),
the CTB computer resource will be used.

For 1-tier systems: Make 30- to 50-year CMIP (Coupled Model
Intercomparison Project) runs, for comparison with the NCEP CFS, to
demonstrate improved simulation of the S-to-I ENSO (El Niño/Southern
Oscillation) and MJO (Madden-Julian Oscillation) signals, and a set of
1-year retrospective forecasts for January, April, July, and October for
1981-present (with a 30-member ensemble) to demonstrate skill in
addition to the CFS benchmark.

For tier-2 systems: Make a set of 1-year retrospective forecasts for
January, April, July, and October for 1981-present (with an ensemble of
30 or more members) using the predicted SST from operational dynamical
model retrospective forecasts from NCEP, other operational centers (when
such forecasts become routinely available to NCEP), or one of the CPC
statistical SST forecasts to demonstrate skill in addition to the CFS
benchmark.

For modifications to the CFS model, the general guideline is that the
changes have to first be implemented in the NCEP Global Forecast System
(GFS). This means that the model changes will be evaluated in both the
climate and the weather arena. EMC has a well-established evaluation
criterion on model changes so a close working relationship with the EMC
staff is important. Evaluation tools under development by CPC and its
partners on the CTB will be used to decide on the added value of each
new model.

RESOURCES: The development and refinement are to be performed at the
model host institution. AO-driven projects may seek computing resources
from the NOAA "Red" R&D computer. Retrospective forecasts are very
expensive and are currently not part of the resources allocated to
either NCEP or the CTB, so developers need to work with NCEP to
ascertain the necessary resources.

NCEP participation : EMC scientists should participate in projects
involving model changes while CPC scientists should participate in
projects involving evaluations and product development from the start of
the project.

**2 Preliminary assessment**

Participants on the CTB should provide the complete retrospective
forecast results (including raw monthly-mean fields) to NCEP and work
with CPC and EMC to assess whether additional, independent information
from the retrospective forecasts exists in relation to existing
operational dynamical and statistical predictions. Iterations back to
Step One may be necessary if the results do not provide the additional,
independent information.

Statistically significant improvements obtained by the above steps would
lead to an initial plan for implementation. A team of scientists from
the external agencies, CPC and EMC will work together to make the
decision to go forward or iterate back to step 1.

RESOURCES: CTB AOs will provide limited resources for preliminary
assessment activities. These projects will be competitive and peer
reviewed. The CTB will provide computer resources and system and science
support for funded projects. A plan for transition to operational status
at NCEP will be written and approved by NCEP and NWS management. The
plan must include personnel and infrastructure (desktop computing,
travel, training, etc) resources for operational maintenance and
enhancement as well as computing (cpu, disk, dissemination, archive).

NCEP participation : The team of scientists will report to the directors
of NCO, EMC, CPC and CTB on the results of the preliminary assessment
and seek their approval to initiate the implementation. NCO Production
Management Branch will initiate an implementation plan and track the
progress of the plan upon the approval of the directors.

3.  **Calibration**

Participants on the CTB should provide the complete 1 year retrospective
forecasts for the period 1981-present for all calendar months (with an
ensemble of 30 or more members).

RESOURCES : The resources needed to complete such runs are very large.
For major projects such as multi-model ensembles and regional climate
models, the host model institution will provide the necessary resources.
Additional resource needs at NCEP will have to be proposed and planned
for well in advance of the calibration step.

NCEP participation : Collaborate with CPC and EMC to continue the skill
assessment for these forecasts and to perform cross-validation. The key
consideration will be the usefulness of the forecasts in providing
independent, additional information to the products that are already
operationally available. This includes whether a new product improves on
the multi-model ensemble in US-temperature and precipitation prediction
and whether it is at least as good as statistical/empirical methods.

4.  **Interface with operations**

As the calibration and assessment work is completed, the provider will
work with EMC and NCO to make all programs in the model conform to NCO
standards and to make all model output conform to WMO and NCO standard
formats.

RESOURCES : CTB AOs will provide limited resources for developing the
interface with operations. These projects will be competitive and peer
reviewed. The CTB will provide computer resources and system and science
support for funded projects.

NCEP participation : EMC and NCO will assess the additional computer
resources needed to put the runs into the operational job stream. The
model provider will work with NCEP to obtain additional resources needed
for the operational runs of such new systems.

5.  **Final skill assessment**

As the calibration runs are completed and the interface with operational
standards is finished, CPC and EMC will work with the provider to
complete the final assessment to determine if the change is ready for
implementation. Assuming production resources are available for the
additional runs, the parallel tests will begin.

RESOURCES : NCEP will provide the resources for this step.

NCEP participation : EMC and CPC will participate in all aspects of the
final assessment.

6.  **Parallel tests**

EMC staff will work with the code provider and NCO to bring all running
scripts up to the NCO production standards. NCO staff will run the new
model in the parallel production environment and make routine
near-real-time forecasts. CPC, EMC and the provider will verify that the
parallel prediction products are the same as the provider intended. CPC
forecasters will begin to evaluate the products in preparation for
operational seasonal forecasts.

RESOURCES : NCEP will provide the resources for this step.

NCEP participation : EMC and CPC will participate in all aspects of the
final assessment.

7.  **Approval**

NCO Production Management Branch staff will brief the director of NCEP
on the technical evaluation of the new model. EMC Global Climate and
Weather Modeling Branch staff will brief the director of NCEP on the
scientific evaluation of the new model, and the CPC Prediction Branch
staff will brief the director of NCEP on the operational evaluation of
the forecasts from the new model. Four categories will be considered and
checked off before the briefing takes place : 1) Forecast/analysis
benefits; 2) Efficiency Assessment (especially important for NCO and
CPC); 3) Compatibility with NOAA/NCEP IT assets; 4) Sustainability. Upon
approval by the NCEP director, the new model will be scheduled for
implementation in the NCEP production suite.