**NOAA Climate Test Bed Interim Report to the CTB Science Advisory Board
(FY07)**

![](media/image1.wmf){width="5.831944444444445in"
height="4.361805555555556in"}

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

**March 2007**

**Executive Summary**

A NOAA Climate Test Bed (CTB) Interim Science Meeting was held February
23, 2007 at the NOAA Science Center in Camp Springs, MD. The meeting was
intended to bring together the CTB management and staff and the SAB
Chair to review CTB progress on previous SAB recommendations and to
begin planning for the 3^rd^ CTB SAB Meeting late this summer. A related
purpose of the meeting was to begin discussion on science issues and
alternative solutions for which the CTB will request independent expert
advice from the SAB.

The purpose of this report is to provide the SAB with some details on
the outcome of the CTB Interim Science Meeting, including the science
issues and alternatives and tentative plans for the 3rd CTB SAB meeting.
The CTB will request a formal written response from the SAB on these
issues following the 3^rd^ CTB SAB meeting. The CTB has benefited
greatly from previous SAB advice and thanks you in advance for your
future efforts.

For reference, the presentations made at the February 23^rd^ Interim
Science Meeting are found on the CTB Webpage:

http://www.cpc.ncep.noaa.gov/products/ctb/ctb-meetings.shtml

Relevant background materials, including the CTB Multi-Model Ensemble
Strategy "White paper", and the latest draft of the CTB Science Plan and
Implementation Strategy are also found on the CTB Webpage. Tentative
plans for the 3^rd^ CTB SAB meeting are found in Appendix A of this
report.

1.  **Introduction**

NCEP's Climate Forecast System (CFS) became operational in August 2004.
This combined weather and climate modeling into a single \"global
modeling\" effort in which model improvements can be applied to both the
GFS and CFS concurrently.  NCEP also supported the Climate Test Bed
through the Climate Program Office and reallocated human and computing
resources onto the CTB. These efforts have really paid off for both
operations and research. For example, since the establishment of the CTB
there has been a string of very good seasonal forecasts, which have
raised skill levels well above the target level (Fig. 1).

**US Seasonal Temperature Skill**

![](media/image2.wmf){width="4.125in" height="2.5076388888888888in"}

> Fig. 1. US seasonal temperature forecast skill. Actual skill (blue
> line) and skill goal (red squares). Points used to compute the Heidke
> skill score come from regions that were not assigned the equal
> chances, "EC", category.

These skillful forecasts can be attributed, at least in part, to early
CTB efforts to consolidate the CFS with existing statistical forecast
tools used in CPC forecast operations. Ongoing efforts to accelerate
improvements in the CFS and related tools may improve these forecasts
further and maintain new skill levels above current GPRA goals.

Through the CTB transition efforts and related interactions with the
larger research community, NCEP is poised to make significant
advancements with the next upgrade of the CFS and work this advancement
within a Multi Model Ensemble (MME) framework consistent with recent
advice of the SAB and the broader climate research community. 

In response to these recent successes, and consistent with previous SAB
advice, the CTB has adopted the following strategic priorities:

-   Accelerate improvements in the NCEP CFS;

```{=html}
<!-- -->
```
-   Partner with the community in the MME enterprise;

```{=html}
<!-- -->
```
-   Accelerate development and delivery of new and improved climate
    forecast products for a diverse user community

Details of the CTB strategy and the related science priorities are
discussed in the CTB Science Plan & Implementation Strategy and in an
MME "White Paper" on the CTB web page at:

http://www.cpc.ncep.noaa.gov/products/ctb/ctb-documentation.shtml

While the CTB has made considerable progress, the CTB can only make this
strategy work with full NOAA Climate Program Office (CPO) support for
MME, CFS, and related climate forecast product development. In section 2
we discuss recent CTB progress on these priorities, identify specific
gaps / limitations, provide alternative solutions, and pose some
specific questions for SAB advice. These topics will be the focus of the
3^rd^ CTB SAB meeting. Tentative plans for the 3^rd^ CTB SAB Meeting are
found in Appendix A. The agenda from this Interim Science meeting is
found in Appendix B.

1.  **Science Priorities, Gaps, and Alternative Solutions**

**2.1. Next implementation of CFS**

The NCEP CFS became operational in August 2004. Currently, there are two
fully-coupled (ocean-atmosphere-land) 9-month forecasts made every day.
The present CFS operational system is frozen.

Development work is underway at EMC to improve the CFS and the proposed
timeline for the next CFS implementation is as follows:

Oct 2006: Testing of the sigma-p version of the GSI scheme for GFS.

Jun 2007: Testing of the ESMF version of the CFS coupler and MOM4.

Aug 2007: Pilot studies for fully coupled CFS reanalysis with new GFS,
GODAS & GLDAS.

Jan 2008: Production and Evaluation of CFS Reanalysis for 1979 to 2007
(29 years)

Jul 2008: Prepare CFS Retrospective Forecasts (2 initial months: October
and April)

Jan 2009: Complete CFS Retrospective Forecasts (remaining 10 months)

Nov 2009: Compute calibration statistics for CFS daily, monthly and
seasonal forecasts.

Prepare CFS Reanalysis & Retrospective Fcst data for public
dissemination.

Jan 2010: Operational implementation of the next CFS monthly & seasonal
forecast suite.

> The three main components of the next CFS implementation are:

-   Fully coupled Ocean-Atmosphere-Land-Cryosphere Model

```{=html}
<!-- -->
```
-   CFS Reanalysis (1979-2007)

```{=html}
<!-- -->
```
-   CFS Retrospective Forecasts (1981-2007)

The CFS upgrade will have improved physics, improved horizontal and
vertical resolution, improved land modeling (through cross cutting
efforts involving NCEP, OHD and the broader research community),
improved MOM4 ocean model (in collaboration with GFDL), improved
assimilation of ocean, land, ice and atmospheric data, and improved ice
modeling in the polar regions. 

Though we have made extensive progress towards the next operational
implementation of CFS, current budget plans are insufficient to sustain
the CFS upgrade schedule given above. Some specific gaps are

-   The coupled CFS reanalysis must be done with existing resources.
    Human resources are needed to assimilate radiance data from old
    satellites (60 of them), and to develop the coupler and 3DVAR system
    for the new ocean model, etc;

-   Computing resources for the CFS reanalysis have not been identified;

-   Computing resources for the CFS re-forecasts depend on the
    availability of the next operational computer in 2008;

-   Human resources to generate, diagnose and disseminate the reanalysis
    and re-forecast datasets are very limited.

A proposed solution to the gaps noted above is for the CTB CFS Testing
and Assessment Team to join forces with the EMC CFS Development Team in
2008-2009 to accelerate the CFS reanalysis and re-forecasting efforts
for the next implementation of CFS. In order to carry out this proposal,
roughly half of the CTB computing resources would be needed during the
2008-2009 period. This poses a considerable risk as the CTB anticipates
a considerable increase in the number of AO driven competitive projects
(and associated requests for computer resources) during FY08 (see
section 2.3). It also comes at a time when MME activities are rapidly
spinning up, with their own resource needs (both computing and human).
Because the CTB computer resources are limited, inevitable conflicts
will arise, and the CTB must work with the community and the SAB to find
an equitable solution.

Because weather and climate modeling are combined into a single \"global
modeling\" effort at NCEP, EMC expects that many modifications to CFS
will first be implemented in GFS. Therefore, the CTB funded external
researchers must collaborate with NCEP developers of both GFS and CFS to
implement their improvements in future implementations of the CFS. This
might involve running the models both in coupled and uncoupled mode, for
both short-term predictions and long-term simulations.

Some specific questions for the SAB are:

-   Should the CTB use its computing resources to help NCEP meet its
    goal for the timing of the next implementation of CFS?

```{=html}
<!-- -->
```
-   Should the CTB staff on the CFS Testing and Assessment Team join
    forces with the EMC CFS Development Team to accelerate CFS
    reanalysis and re-forecasting efforts required for the next
    implementation of CFS?

```{=html}
<!-- -->
```
-   What is the appropriate balance between AO-driven external projects
    and internal activities (such as CFS reanalysis and re-forecasting)?
    Should this balance be determined by whether a project contributes
    to the next implementation of CFS as opposed to the implementation
    that follows the next one?

-   What procedures should the CTB use to ensure that its computing
    facility is equitably shared between external (competitive) and
    internal projects?

-   With reanalysis and re-forecasts, we will be able to determine
    expected improvements to the skill of objective, model-based
    forecasts. How do we measure the economic and social benefit of
    these improvements?

> **2.2. Multi-Model Ensemble Prediction System**
>
> NCEP, through the CTB, is working with the climate community to
> develop a strategy for a MME Prediction System at NCEP with CFS
> included. Steps toward an operational MME Prediction System at NCEP
> include :

-   Next implementation of CFS;

-   Preliminary skill evaluation of International MME (IMME) and
    National MME (NMME) members;

-   Assembly of full re-forecast data sets from IMME and NMME
    contributors;

-   Consolidation of the IMME and NMME members with CFS included into a
    single operational MME Prediction System;

-   Assessment and communication of uncertainty for MME prediction.

Considerable progress on the preliminary skill evaluation of the IMME
was reported at the Interim Science meeting (see
TPT-MME-Saha-23feb07.ppt) on the CTB Webpage:

http://www.cpc.ncep.noaa.gov/products/ctb/ctb-meetings.shtml

In particular, two studies were conducted using the CFS and European
DEMETER data to evaluate the following questions:

-   How extensive (long) should hindcasts be?

```{=html}
<!-- -->
```
-   Do the NCEP CFS forecasts add skill to the European DEMETER-3
    forecasts to produce a viable International Multi Model Ensemble
    (IMME)?

For question 1 it was shown that CFS improves from extensive hindcasts
(21 years was noticeably better than 8) and that it has the most skill
of the 4 models considered for the IMME. For question 2 it was shown
that NCEP CFS contributes to the skill of an IMME (relative to DEM3) for
equal weights. This was especially the case when the probabilistic Brier
Score was used, and for precipitation. In comparison to ECMWF, METFR and
UKMO, the CFS as an individual model did well in deterministic scoring
(Anomaly Correlation) for precipitation and very well in probability
scoring (Briar Score) for both precipitation and surface temperature
over both the US and Europe.

The results of this study were sent to the ECMWF. The Director showed
interest, but wanted his own scientists to carry out a similar
evaluation. Dr. Doblas-Reyes (ECMWF) has downloaded the CFS
retrospective data from the CFS server and is in the process of
evaluating the IMME, but using the latest EUROSIP data (instead of the
DEMETER data).

Progress on the NMME was also reported, though it has been relatively
slow. Hindcast data have been obtained from GFDL for 4 initial months
(Apr, May, Oct and Nov). These data are being postprocessed and
transferred to NCEP for evaluation and inclusion in an NMME with the
CFS. Though NCAR is not ready to start their hindcasts, COLA has done a
few hindcasts with the NCAR model which show promise. A full hindcast
dataset is required for evaluation of an NMME with the CFS. NASA is not
ready to start their hindcasts.

The MME enterprise carries considerable risk for NCEP. Some key gaps
are:

-   Lack of a National Strategy on MME, including insufficient (US)
    computer resources for generating hindcast data sets;

-   Lack of a concept of operations, including MOUs with partners
    (International and US) for exchanges of operational models and real
    time forecast data;

-   The Europeans may well decide to keep their MME European;

-   NCAR and NASA have their own time lines for model development;

-   CTB has resource limitations for skill assessments, post-processing,
    calibration, consolidation, and preparation for operations;

-   Operational requirements dictate that there must be additional skill
    in monthly and seasonal 2-meter temperature and precipitation over
    the US as the basis of any MME. What happens if none of the NMME
    models add additional skill to the CFS in these measures?
    Concentrate on IMME?

Based on the results to date, the CTB MME Team proposes that CTB
continue working on both the IMME and the NMME. In addition to the IMME
activities reported above, the CTB has partnered with other countries to
develop the IMME, including BMRC, Australia and BCC, China. However,
many of the gaps and limitations noted above are the result of much
bigger issues that cannot be resolved by the CTB.

Some specific questions for the SAB are:

-   Has the CTB developed an effective MME implementation strategy?

-   Should CTB proceed with both IMME and NMME, and in what balance?

-   How can CTB influence development of a National Strategy for MME,
    and climate model development?

-   Does the CTB have the appropriate balance between MME and CFS
    activities?

> **2.3 Climate Forecast Products and Services**

The CTB plays a unique and critical role in NOAA Climate Services. The
CTB collaborates with the R&D community (e.g., ARCs, IRI) to accelerate
the transition of new and improved **science** driven climate forecast
and analysis products into operations with emphasis on the CPC specific
product range (6-10 day, week 2, monthly, seasonal). The CTB also
collaborates with the applications research community (e.g., RISAs,
SARP) for requirements on new and improved **demand** driven climate
products and services.

Three important examples that illustrate how the CTB is fulfilling its
role are given below. First, the CTB has established a program to
enhance interactions (partnerships, cooperative research, planning,
exchange visits) between CPC and the climate applications science
community. To date the effort has been primarily focused on the
cultivation of relationships with the RISAs in order to develop a deeper
understanding of decision contexts and needs for climate forecast
products, as well as effective means for supporting access to and use of
climate forecast information (Table 1). The basic premise of these
interactions is that the development of user-centric climate forecast
products would benefit from partnerships of the technology driven
efforts of CPC and the stakeholder-driven efforts of interest to the
RISAs. However, these communities have different priorities: CPCs
technology-driven projects are often limited in their ability to
determine specific user needs; stakeholder-driven projects are often
limited in their ability to fully implement sophisticated engineered
software projects required for successful transition to sustainable
operations. The challenge for the CTB is to bridge this gap between
communities in the development and delivery of user-centric climate
forecast products.

**Table 1. CTB/CPC - RISA Interactions**

**[RISA]{.underline} [CTB/CPC Liaison]{.underline} [RISA
Contact]{.underline} [RISA Visitor to CPC]{.underline}**

SECC Muthuvel Chelliah James Jones Guillermo Baigorria / Shrikant Jagtap
(May 07)

PEAC/PaCIS Luke He Eileen Shea

Alaska Jon Gottschalck James Partain

CAP Kingtse Mo Dan Cayan

CLIMAS Ed O'lenic Holly Hartmann

WWA Michelle L'Heureux Andrea Ray

CIG Douglas LeComte Dennis Lettenmaier

The CTB program summarized by Table 1 includes exchange visits in which
CPC/CTB staff make short visits to the RISAs to provide customized
overviews of CPC products. The RISAs in turn send longer term visitors
(up to several months) to CPC to develop customized products off of the
NCEP database and the CPC product suite. These interactions have already
led to a number of important new products that would not exist without
the CTB, including a concerted effort to convert CPCs monitoring and
forecast product suite to GIS format.

Second, the CTB has been working with the NOAA CPO to expand its
Competitive Grants Program through Annual Announcements of Opportunity
(AO) with priorities that accelerate improvements in NCEP climate
forecast models, products and applications. During FY06 the CTB funded 3
competitive projects aimed at accelerating improvements in the NCEP CFS
(section 2.1). During FY07 the CTB funded 2 competitive projects to
enhance CPCs suite of climate forecast products. These 5 projects are
listed in Table 2. During FY08 the CTB anticipates a somewhat larger AO,
with science priorities that include CFS improvements, Multi-Model
Ensembles, and drought / NIDIS. The CTB FY08 Federal Funding Opportunity
and Information Sheet is in preparation.

**Table 2.** List of current CTB competitive transition projects.

**FY06**

Using initial tendency errors to reduce systematic errors

(PI: Tim Delsole, COLA; Co-PI: Hualu Pan, EMC)

Neural network emulations of model physics for CFS

(PI: Michael Fox-Rabinovitz, U. MD; Co-PI: Kransapolsky, Suru Saha, EMC)

Ocean component of the NCEP ENSO CFS

(PI: Michael McPhaden, PMEL; Co-PI: Yan Xue, EMC)

**FY07**

Probabilistic forecasts of extreme events and weather hazards in the US

(PI: Charles Jones (UCSB); NCEP Co-PI: Jon Gottschalck, CPC)

System-wide advancement of user-centric climate forecast products

(PI: Holly Hartmann (UAZ); NCEP Co-PI: Ed O'Lenic (CPC))

The CTB is also providing resources for and actively participating in a
number of CPO Climate Prediction Program for the Americas (CPPA) funded
projects to support NIDIS:

![](media/image3.wmf){width="5.882638888888889in"
height="3.6770833333333335in"}

Finally, CPC is in the midst of a gradual transition to a more objective
approach to the climate forecast problem that relies on objective
statistical and dynamical forecast tools, consolidation techniques, and
objective verification. Though the forecaster still plays an important
role in the forecast process, it is also true that this transition has
allowed CPC to redirect more of its resources towards the development
and delivery of regionally and sectorally specific climate forecast
products and services. During the past year the CTB has been working
with RISA participants to help develop a list of user-centric products
that CPC could provide, a so-called RISA "wish list". The original "wish
list" was compiled by Kelly Redmond and sent from the NOAA RISA program
participants to both the SAB and the CTB in June 2006. Shortly after the
2^nd^ CTB SAB meeting, CTB staff provided an initial response to the
"wish list" outlining actions that were deemed relatively painless,
moderately difficult, and requiring careful thought, and pledging to
move forward on several items. The CTB has moved forward on those items
and will discuss progress with the applications research community
during the 5^th^ Climate Prediction Applications Science (CPAS)
Workshop, March 20-23, 2007, in Seattle, WA. The CTB is sending 7 CPC
participants to this meeting (Table 1) together with CTB management. In
addition to the Workshop, additional "face-to-face" meetings involving
the CPC personnel, CTB management, and RISA participants will be held to
discuss progress on and additional refinements of the "wish list".

Some specific questions for the SAB are:

-   Are CTB efforts to link CPC with the research applications community
    developing appropriately?

-   What criteria should the CTB use in developing the list of new
    user-centric products to be developed and transitioned to
    operations?

-   How should the CTB work with CPC forecast operations to enhance the
    development and delivery of climate forecast products and services
    (e.g. balance between objective and forecaster-enhanced forecast
    products)?

-   What should the CTB do to influence the research agenda of the
    applications research community in the longer term?

-   How should CTB measure success (e.g. number of new and improved
    products, economic impact, social utility, etc.)?

-   How do we decide on the proper balance of efforts between
    improvements to the skill of models and new climate products?

-   Should CTB develop a suite of web-based interactive tools that allow
    users to develop their own decision-oriented products?

-   Should CTB develop a real-time attribution system as part of an
    enhanced overall effort to explain forecasts to users?

-   Should CTB work with THORPEX to develop a long-term strategy aimed
    at improving extended range (6-10 day, week-2) forecasts by
    enhancing the climate-weather connection?

**Appendix A. Tentative Plans for the 3^rd^ CTB SAB Meeting**

Proposed dates for the 3^rd^ CTB Science Advisory Board (SAB) meeting
are August 28-29, 2007 at a hotel in the Washington DC area. These dates
are subject to change based on the schedules of the SAB members. The
proposed meeting objectives, expected outcome, and a strawman agenda are
given below. The SAB is invited to revise the tentative meeting plans
and to communicate revisions to the CTB Director.

The proposed meeting objectives for the 3^rd^ CTB SAB Meeting are: (i)
to review CTB progress on science priorities and SAB recommendations;
and (ii) to obtain independent expert advice on short-term (FY08-FY09)
and long-term (next 5 years and beyond) science priorities and
implementation strategy. The Expected Outcome of the 3^rd^ CTB SAB
Meeting is a SAB written report with expert advice and recommendations
that address the specific questions raised in this report as well as
other pertinent issues raised at the 2^rd^ CTB SAB meeting. Invitations
will be extended to the SAB (10), OB (7), CST (13), External PIs (5),
and CTB Staff (40). A venue that accommodates at least 50 people will be
chosen.

The science presentations during the 3^rd^ CTB SAB meeting will be
focused on the priorities, gaps, alternative solutions and questions
raised in section 2 of this report. We propose that roughly 1 day be
devoted to the science presentations, with roughly ½ day for a SAB
Executive Session and another ½ day for SAB closed meetings with CTB
Team Chairs, the CTB Oversight Board, and Exit Interviews. A strawman
agenda is given on the following page:

> **Proposed Agenda for 3^nd^ CTB Science Advisory Board Meeting**
>
> **Hilton Silver Spring**
>
> **Silver Spring, MD**
>
> **August 28-29, 2007**

**[Tuesday, August 28, 2007]{.underline}**

8:00 a.m. SAB Executive Session / Continental Breakfast

9:00 a.m. Welcome -- Louis Uccellini

9:15 a.m. Introductions, Logistics -- Melvyn Gelman

9:30 a.m. Update on the CTB and Meeting Objectives -- Wayne Higgins

9:45 a.m. Break

10:00 a.m. MME Prediction System -- Suranjana Saha and MME Team

*(How extensive (long) should hindcasts be?; International Multi Model
Ensemble*

*(IMME); National Multi Model Ensemble (NMME); Consolidation ; etc.)*

11:30 a.m. MME Discussion

12:00 p.m. Lunch

1:00 p.m. CFS Testing and Assessment -- Jae Schemm, Shrinivas Moorthi
and CFS Team

*(Next Implementation of CFS, CFS Reanalysis and Re-forecasting, Outcome
of*

*FY07 Experiments, PI Presentations, etc)*

2:30 p.m. CFS Discussion

3:00 p.m. Break

3:15 p.m. Climate Forecast Products -- Ed O'lenic and Climate Products
Team

> *(CTB/CPC Interactions with RISAs, New developments in CPC Forecast*
>
> *Operations; Drought Monitoring and Prediction / NIDIS)*

4:30 p.m. Climate Forecast Products Discussion

5:00 p.m. General Discussion of Science Presentations

6:00 p.m. Adjourn

**[Wednesday, August 29, 2007]{.underline}**

8:00 a.m. SAB Executive Session / Continental Breakfast

9:00 a.m. CTB Grants Program / Proposed O2R Concept -- Ken Mooney

9:30 a.m Shared Computing Facility -- Hualu Pan

9:45 a.m. Tracking CTB Tasks -- Mel Gelman

10:00 a.m. Break

10:15 a.m. Future of CTB Teams --Siegfried Schubert and Hualu Pan

10:30 a.m. General Discussion of Administrative Presentations

11:00 a.m. Discussion with TPT Chairs (Saha, Moorthi, Schemm, O'lenic)
and CST

Chair (Schubert) but without CTB Management (closed) -- Tony Busalacchi

11:30 a.m. Discussion with Oversight Board (Koblinsky, Uccellini, Kumar,
et al.) -- Tony

Busalacchi

12:00 p.m. SAB Executive Session (closed) -- Working lunch (provided) to
draft

responses to CTB questions

2:00 p.m. Exit Interview with CTB Management and Oversight Board

3:00 p.m. End of Meeting

**Appendix B. Agenda for CTB Interim Science Meeting**

> **Agenda for CTB Interim Science Meeting**
>
> **NOAA Science Center, Room 802**
>
> **Camp Springs, MD**
>
> **February 23, 2007**

1:00 p.m. CTB Issues -- Wayne Higgins

1:15 p.m. CTB Progress and Tracking CTB Tasks -- Mel Gelman

1:30 p.m CFS Issues / Limitations / Risks -- Hualu Pan

1:45 p.m. Future Role of the CST and TPTs -- Siegfried Schubert

2:00 p.m. Transition Project Team Reports

Multi-Model Ensembles -- Suranajana Saha

> CFS Testing and Assessment -- Shrinivas Moorthi and Jae Schemm
>
> Climate Forecast Products *--* Ed O'lenic

3:00 p.m. Discussion on planning for 3rd CTB SAB Meeting -- Wayne
Higgins

3:30 p.m. Adjourn