**Proposal to:**

**National Oceanic and Atmospheric Administration**

**Climate and Global Change Program**

**PROJECT AREA: Climate Test Bed**

**TITLE: Consolidation of Multi Method Seasonal Forecasts at CPC**

**Climate Prediction Center & Environmental Modeling Center**

**NCEP/NWS/NOAA**

**5200 Auth Road, Camp Springs, MD 20746**

**CPC Director:**

**\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_**

James D. Laver, Director

Climate Prediction Center

**Principal Investigator:**

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Huug van den Dool

Climate Prediction Center

Tel: (301) 763-8000 x7570

Submitted: April, 20, 2005

Funding Period: Fall 2005 - Fall 2006.

Total funding requested: \$160K

TABLE OF CONTENTS

ABSTRACT

0\. Preamble

1.  INTRODUCTION

2.  Ridge Regression

3.  Gaussian Distribution Kernel

4.  Practicalities

5.  Links to Other Research Activities

6.  Timetable& Workplan

7.  Readiness

8.  BUDGET

9.  CURRENT AND PENDING SUPPORT

10. VITAE

> Abstract
>
> We propose a method for consolidation of seasonal forecasts. The
> method takes into account that we have very short records (\~25 years)
> and there will be (or could be) many methods that are highly
> correlated. While we propose two specific methods in great detail,
> Ridge Regression and Gaussian Distribution Kernel, we have not written
> anything in stone and expect amendments when executing the proposal.
>
> Implementation for operational forecasts is envisioned, keeping the
> practicalities at CPC in mind.
>
> Preamble
>
> Since this is only a one-year proposal we see this activity only as a
> start. Involvement of other parties, especially IRI, in a new
> multi-year proposal to be submitted after completion of the first year
> should be considered.

**1. Introduction**

Consolidation of multiple forecasts is necessary for a number of
reasons. One can think of consolidation as the process of making the
best possible official forecast out of a number of different forecast
tools. This is a laudable goal for any weather service. While one may
entertain the thought of doing consolidation subjectively, one very good
reason for *[objective consolidation]{.underline}* is that the supply of
forecasts has become so large that no human (forecaster or user) is able
to absorb the information in the available time, weigh their relative
credibility, do justice to each component, and formulate the official
forecast. And the problem is only getting worse. This is probably true
for both short and long range forecasts, but here we address the latter.
The idea of formal consolidation is at least as old as Thompson (1977),
but development has been slow. Recent efforts to establish the Climate
Test Bed facility have identified tool consolidation as a top priority
for accelerating improvements in NOAA operational climate forecasts.

We consider it self evident that a) one needs hindcasts in order to make
an optimal consolidation and b) the real time forecasts need to be
consistent in all respects with the hindcasts. But note that this
practice has hardly been developed for dynamical models. CDC has
undertaken a reforecast project (Hamill et al 2004) for the week2
forecast -- this may be a prototype, an atmosphere only reforecast. The
new coupled ocean atmosphere model at NCEP (Saha et al 2005), named
Climate Forecast System (CFS), has a hindcast amounting to 3500 years of
integration. In the case of Hamill et al. (2004) the purpose was
calibration of the model in its own right, not consolidation. The CFS
also needs to be combined with other methods currently used at CPC.
Other literature or practices are often employing AMIP runs for the
hindcast of dynamical models (Peng et al 2002), but this may not
necessarily apply in the real time setting. For statistical methods the
idea of hindcasts comes more natural and is fairly well developed

Consolidation is difficult for many small reasons, but two large reasons
in particular. The first is a stunning lack of data vis-à-vis the number
of coefficients that need fitting. Think of consolidation as CON =
a\*A + b\*B +c\*C....., where the capital letters refer to forecast
tools, and the lower case coefficients need to be determined from
hindcasts. Seasonal forecast tools like the CFS (Saha et al 2005/06)
have only a 23 year record of hindcasts. This may sound like a lot, but
in terms of seasonal means there is very little independent material to
tune coefficients on. While some empirical methods like CCA (Barnston
1994) may have 50 (or potentially 100) years, the data set still falls
short by orders of magnitude if there are many methods/models to be
combined. This is especially so when A, B, C and all others are highly
correlated. The 2^nd^ major difficulty is low frequency climate change.
The period 1981-2003 was much warmer over the US than the 60's and 70's.
This major source of forecast skill (when verified against an 'old'
normal) is accounted for in real time operations at CPC by the Optimal
Climate Normals (OCN) method (Huang et al 1996). Dealing with this
aspect retroactively on very limited data may be next to impossible.

By the nature of the consolidation methodology used one may see a stark
distinction between two groups of consolidation activities, namely those
working with point forecasts (it will be 66.0 F tomorrow) and those
working with pdf forecasts (next winter a 50% chance of the upper
tercile). The distinction is not as absolute as it may seem, because in
applying linear regression to N point forecasts, one automatically
obtains the root-mean-square error (rmse), which, to first
approximation, serves as a standard deviation around the point forecast,
i.e. linear regression results in a consolidated pdf. Whether the spread
of forecasts, on a case-by-case basis, can be used to improve upon the
vanilla pdf remains an open question, but we note that Hamill et al.
(2004) have concluded in the negative.

We have to be practical, having in mind CPC's operational setting, and
knowing, more or less, which type of models/methods, in principle, could
participate. We propose two consolidation methods for further
development. The first is linear Ridge Regression, having the method of
data assimilation (Gandin 1965) in mind as a practical solution in a
situation of enormous redundancy. We use this method because we
recognize there is not enough data to tune all coefficients and to
resolve co-linearity among forecast methods. We thus hit the covariance
matrix hard with some strong medicine that allows stable albeit flawed
solution. The 2^nd^ method is the Gaussian Density Kernel method, which
seeks to combine pdfs assigned to each method (?). In the GDK method
there is ample opportunity for the spread in the ensemble to help shape
the pdf on a case-by-case basis.

In general, consolidation could be combined (or folded in) with
calibration. However, we approach consolidation separately from
calibration. That is, we consider it the task of the originators of the
forecast to correct for systematic errors in the mean or higher order
moments, if they wish.

Consolidation is not new at NCEP. Leaving a long history of subjective
consolidation aside, CPC actually has had a primitive consolidation of a
few of its seasonal forecast tools running from 1995 onward (Fig.1). But
it has proved difficult to maintain this tool in real time over the
years nor can it be adjusted to ingest more tools. Van den Dool and
Rukhovets(1994) designed a consolidation for the various members (at the
time unequal members in terms of resolution, age) of the global model
for support of the 6-10 day forecast. Ever since Unger et al(1996), CPC
has presented the official NWS SST forecast (Nino3.4 only) as a
consolidation of in-house tools (CCA, CA, MRK, coupled model) -- the
exact method has evolved over time and has led to the idea of GDK
presented here. In a certain sense 'ensemble' CCA (Mo, 2003) is also a
consolidation.

Outside NCEP 'multi-model ensembles' appears to be even more popular. At
the IRI a Bayesian method and a canonical variate method (Barnston et al
2003) are used to consolidate a large number of model forecasts from
various centers around the world. The DEMETER project in Europe has been
used to make a strong case in favor of multi-model ensembles (Hagedorn
et al 2005).

In sections 2 (3) we describe the Ridge Regression (Gaussian
Distribution Kernel). Although we present the details of these methods
we do not want to give the impression that the methods are cast in
stone. It is likely that changes will be made as we try out the methods,
and gather input from others. In section 4 we present other
practicalities, which may also be subject to change.

Although we are optimistic one should also be realistic. Consolidation
should be better than the single best participating tool, but for all
the work to be done to determine optimal weights etc, the following
should be kept in mind.

\(1\) The Consolidation will not be much better that the best individual
tool if there is little independent information provided by the other
tools

\(2\) The Consolidation will fail (in real time) if any of the hindcast
data sets gives a flawed impression of skill or if we cannot execute
real time forecasts 100% consistent with the hindcasts.

We do not have control over all of these factors.

**2. Ridge Regression**

For US T&P we propose to implement a version of Ridge Regression as the
consolidation method. It is assumed that this will function even with an
'overload' of participating methods and short data sets -- and this is
the key consideration. The solution will be kept sane by pulling it
slightly in the direction of a simpleminded approach based on the skill
of each method. Since skill ought to be positive, negative weights
should never be assigned to any method. Space dependence of weights may
be possible to some degree. 'Regime' dependence will be possible to some
degree. Missing methods in real time can be easily accommodated.

Definitions.

Let **A**, **B** and **C** be three forecast methods with a hindcast
history 1981-2003. **A** etc is shorthand for **A** (year, initial
month, lead, space) or **A** (year, target month, lead, space).
Stratification by month is customary, so **A** (y, l, s) suffices in the
notation below, where y is 1981 to 2003, lead=1, 6(13), space (s) could
be gridpoints NH (for example) or 102 super Climate Divisions in US. The
matching observations are **O** (y, l, s). The inner product is defined
by:

**AB** = Σ **A** ( y, l, s) \* **B** (y, l, s) (1)

where summation is over time y and (some or all of) space s. For
simplicity we work with just three methods, but the derivation can be
given for N methods.

# Simple consolidation

In general we look for: CON(solidation) = a\***A** + b\***B** + c\***C**
(2)

In a s

imple-minded solution a, b and c are proportional to the skill of
methods **A**, **B** and **C**, i.e. proportional to **AO**, **BO** and
**CO** (covariances), multiplied by 1/**AA** etc. In that case Σ a+b+c
probably needs an additional constraint like a+b+c=1. a, b and c could
be function of s, lead, initial (target) month. a, b and c should always
be positive because we do not admit methods with negative skill (over
the hindcast data set). As seen in section 3 alternatives to the weights
a, b and c could be R/(1-R), where R is the correlation of say A and O.

# Full solution

While a simple-minded solution may be practical we actually seek the
full optimal solution, taking into account both skill by methods and
'co-linearity' among methods:

Matrix \* vector = vector

\|**AA** **AB** **AC**\| \|a\| \|**AO**\|

\|**BA** **BB** **BC**\| \* \|b\| = \|**BO**\| (3)

\|**CA** **CB** **CC**\| \|c\| \|**CO**\|

If co-linearity were zero, note a = **AO**/**AA** etc, the simple-minded
solution. Also note there is no constraint on Σ a+b+c . The full
solution takes co-linearity of methods into account. I.e. if A and B
always give the same information they have to share the weight -- they
do not both get a high weight. The measure of co-linearity is given by
the strength of the off-diagonal elements, relative to the main diagonal
(AA, BB and CC).

## Addressing Serious Problems

Co-linearity is essential for the full solution, but problems arise when
the co-linearity is too large, or when there is not enough data to
estimate the co-linearity accurately. In either case the solution
(a,b,c) may be unstable. In consolidation of seasonal forecasts there is
FAR too little data to determine a,b,c...z, given the number of
participating methods (quickly increasing).

Ridge regression is an amendment to the full solution to address this
problem. One can stabilize the solution by adding small positive
constants to the main diagonal of the matrix. This changes (3) to (4).

Matrix \* vector = vector

\|**AA**+ε2

**AB** **AC** \| \|a\| \|**AO**\|

\|**BA** **BB**+ε2

**BC** \| \* \|b\| = \|**BO**\| (4)

\|**CA** **CB** **CC**+ε2

\| \|c\| \|**CO**\|

Even very small ε2 can stabilize the worst possible matrices, see
example in Appendix for a demo. Adding ε2 to the main diagonal plays
down the role of co-linearity ever so slightly, and drives the solution
very slightly in the direction (but not exactly so) of a simple-minded
solution. A 2nd layer of amplitude adjustment may be needed.

## About Ridging

The oldest reference on ridging in the English literature is
Tikhonov(1977) in translation, but this method may have been known since
1950 in Russia. The basic idea is to find a reasonable solution where
there are more unknowns than equations. Nominally we have 3 equations
and three unknowns in (3), but when co-linearity is too large there may
be effectively fewer than three equations. While (3) minimize the rms
difference between O and CON on a given data set, the ridge regression
minimizes simultaneously Σ a\*a+b\*b+c\*c. As can be seen from the
example in the appendix, ridging does this very effectively. The
situation we encounter in consolidation is similar to data assimilation
(Gandin 1965), where redundance (co-linearity) among observations to be
assimilated is large. In the data assimilation context ε2 relates to the
(assumed) error in the observations. One could even embrace the
situation as follows.

-) truncate forecast(obs) in EOF space

-) Now determine AA from filtered data....

-) Add ε2 which is related to variance of unresolved EOFs

-) Controlled use of noise: off-diagonal elements unchanged.

-) solve the system.

In this context ε2 has a real meaning, namely the variance of the
unresolved EOFs. We are not sure whether the EOF truncation is
necessary, and if it is whether either EOF's of the fields of variables
(say US T) or eigenvectors of the matrix in (3) are the best way to go.
We may have to work on adjusted ridging where in the limit of infinite
noise the simple-minded solution for a, b, c emerges.

Given our experience with ridging (Van den Dool et al 2003), we feel we
have a basis for assuming that the proposed consolidation will function
even with an 'overload' of participating methods and short data sets --
and this is the key consideration. The weights generated by ridging will
be handed off to the new procedure of making the probability forecast,
described in the section 3. Meanwhile, since the (ridge) regression also
comes with an rms error and a correlation (a frequentist' a-priori
skill) we could use an older recipe (Unger 1998, personal communication)
described in Fig.2 to make a probability forecast in three classes. For
the same a-priori skill the probability anomalies vary greatly as a
function of the point forecast anomaly.

Other than consolidation of drastically different methods, ridge
regression can also be used on the very tough problem of assigning
weights to the 15 members (per month) of the CFS ensemble. Being
different in age (as in lagged average forecasting some 20 years ago)
the older members should have lower weight, but the way weights decrease
backward in time is entirely unknown. Here too the forecasts correlate
very highly among themselves.

**3. Gaussian Distribution Kernel method**

An important part of any climate forecast is the ability to express
probabilistic information about the variable being predicted. A
probabilistic prediction is difficult because it requires statistical
information about the forecast system in order to provide a well
calibrated product. In the case of climate prediction, the forecast
system consists of many different tools, each of which uses different
prediction method. Some tools are based on statistical methods such as
regression, or canonical correlation analysis, while others are based on
physical models of the ocean and atmosphere. The physical modeling
approach has recently emphasized the ensemble forecasting approach, in
which multiple forecasts are produced for the same target period. The
problem at hand is how to process the information from the many tools
into a single probabilistic forecast.

We plan to use a consolidation method based on the relationships between
explained and unexplained variance of the forecast system. Forecasts
from each tool are weighted roughly according to the skill of the system
and to compensate for possible redundant information from any particular
tool we use the weights of Ridge Regression as described in section 2.
Statistics from the weighted ensemble are then analyzed to produce a
forecast probability distribution function based on the kernel smoothing
method (Silverman, 1986). Kernel smoothing is similar to a histogram
except that a distribution is placed around the histogram elements
rather than to assume a discrete value. When applied to an ensemble
forecast, the probability distribution of a forecast element, F, is
obtained by placing a distribution around each member\'s forecast for a
particular variable at a given location. The final PDF is obtained from
the sum of all individual \"kernel\" distributions, normalized to unity.
In this case, a Gaussian distribution is used as the kernel distribution
function, with standard deviation S~z~. (See Fig. 3.)

Before the Gaussian kernel density method can be applied to a prediction
problem, several important issues must be addressed. Namely, the width
of the kernel, S~z~, and how to correct forecast bias and account for
forecast skill. While there are several techniques documented in
meteorological literature that provide kernel smoothing parameters
Roulston and Smith (2003), Anderson(2001), none are as simple as the one
proposed here. A method that accounts for both the predictability of the
ensemble forecasting system, and the spread of the members is used here
to produce the kernel statistics.

Linear regression is a well known statistical technique that relates
predictor variables to their subsequent observations. A linear
regression equation derived from the ensemble means and their associated
observation is a measure of skill. Regression theory states that the
explained variance, σ~F~^2^ is equal to the climatological variance of
the predicted variable, σ~C~^2^ , multiplied by the square of the
correlation coefficient between the forecast (in this case the ensemble
mean) and the observation, R~F~^2^ ; σ~F~^2^ = σ~C~^2^ R~F~^2^ .

(See list of definitions at the end of section 3.)

The unexplained variance, also know as the error variance, therefore, is
the residual.

> S~F~^2^ = σ~C~^2^ (1 - R~F~^2^ )

An ensemble forecast attempts to account for part of the error variance
through the ensemble spread. Thus the error variance is divided into two
components: a predicted component, as determined by the ensemble spread,
expressed as a variance, E^2^ , and the residual, which is apportioned
equally about each ensemble members, since we don\'t know in advance
which member will be correct. If it is assumed that the errors
unaccounted for by the ensemble spread are normally distributed, then
the residual error will form a Gaussian distribution around each
ensemble member with variance, S~z~^2^ .

The following equations govern the relationship between the various
variance components of the system.

S~F~^2^ = S~Fm~^2^ + E^2^ (5)

S~Fm~^2^ = S~z~^2^ + E^2^ (6)

Regression theory can be used to provide a candidate for optimum
ensemble spread. The error variance for regression for the ensemble
mean, and individual ensemble forecasts respectively are:

> S~F~^2^ = σ~C~^2^ (1 - R~F~^2^ )

S~Fm~^2^ = σ~C~^2^ (1 - R~Fm~^2^ )

Substituting these into Eq. (5) yields:

> E^2^ = σ~C~^2^ (R~Fm~^2^ - R~F~^2^ ). (7)

Which implies from Eq. (6) that;

> S~z~^2^ = σ~C~^2^ (1 - 2R~Fm~^2^ - R~F~^2^ ) . (8)

Thus the statistical parameters of the ensemble system are specified by
the historical performance of the individual forecasts and their
ensemble mean. Note that occasionally some adjustments must be made to
keep S~z~^2^ positive.

The core of the GDK method is Eq (7) and (8).

The Gaussian kernel forecast is then produced by a four step process.
First the forecast is standardized by subtracting its mean and dividing
by the standard deviation;

> Z~i~ = (F~i~-F)/ σ~F~

This standardized forecast is then adjusted according to the
predictability of the system:

> Z\'~i~ = R~z~ Z~i~

The adjusted and standardized forecast is then restored by applying the
climatological mean and standard deviation of the forecast variable:

F\'~i~ = Z\'~i~ σ~C~ + C

These three steps are identical to Linear regression except that the
effective correlation,

R~z,~ is used rather than R~Fm~ . R~z~ is computed from the following
relationship:

R~z~^2^= ( 1-( S~z~^2^/ σ~C~^2^) ).

R~z~ ≤ R~Fm~ and is the expected skill of the system when all ensembles
agree.

Finally the ensemble spread is adjusted to its optimum value by applying
a correction proportional to the ratio of the optimum spread and the
actual spread over a historical set of forecasts.

F\"~i~ = ( F\'~i~ - F~m~)(E\'/E) + F~m~ .

Ensembles from any given model are assumed to be equal skill. Forecasts
from different models are weighted according to skill and co-linearity,
and entered into the ensemble system prior to the computation of Kernel
statistics. Kernel statistics are then computed according to each tools'
weighted contribution.

SYMBOLS AND DEFINITIONS:

F = Forecast , O = Observation, C = Climatology, F = Mean forecast over
many cases.

σ = Standard deviation of the forecasts, or observations. Standard
deviations are computed with annual climatological mean removed, and are
based on a substantial portion of the hindcast record.

S = Standard deviation of forecast errors, (F-O) , S is also based on a
substantial portion of the hindcast record.

R = The correlation coefficient between F and O.

E = A measure of the ensemble spread over n individual forecasts for a
given initial time.

E^2^ = Σ(F~i~ -F~m~)^2^/n

Subscripts on F:

Unsubscripted or with subscript i = individual ensemble member, m=
ensemble mean

Subscripts on S, σ, or R:

F = individual ensemble forecast, Fm = Ensemble mean, z = Gaussian
kernel.

**References to section 2 and 3:**

Anderson, J. L., 2001: An Ensemble Adjustment Kalman Filter for Data
Assimilation.

*Mon. Wea. Rev.,* 129, 2884-2903.

Barnston, A.G., 1994: Linear Statistical Short-Term Climate Predictive
Skill in the

Northern Hemisphere. Journal of Climate: Vol. 7, No. 10, pp.1513--1564.

Barnston, A. G., Simon J. Mason, Lisa Goddard, David G. Dewitt and
Stephen E. Zebiak.

> 2003: Multimodel Ensembling in Seasonal Climate Forecasting at IRI.
> Bulletin of the American Meteorological Society: Vol. 84, No. 12, pp.
> 1783--1796.

Gandin, L. S. (1965), Objective Analysis of Meteorological Fields,
translated by Isr.

Prog. Sci. Transl., 242 pp., Gidrometeoizdat, Leningrad, Russia.

Hamill, T. M.,Jeffrey S. Whitaker and Xue Wei. 2004: Ensemble
Reforecasting:

> Improving Medium-Range Forecast Skill Using Retrospective Forecasts.
> Monthly Weather Review: Vol. 132, No. 6, pp. 1434--1447.

Hagedorn, R., F. J. Doblas-Reyes and T. N. Palmer, 2005: The rationale
behind the

> success of multi-model ensembles in seasonal forecasting. Part I:
> Basic concepts. Tellus, 50, p (to appear).

Higgins, R. W., H.-K. Kim and D. Unger. 2004: Long-Lead Seasonal
Temperature and

> Precipitation Prediction Using Tropical Pacific SST Consolidation
> Forecasts**.** *Journal of Climate*: Vol. 17, No. 17, pp. 3398--3414

Huang, J., Huug M. van den Dool and Anthony G. Barnston. 1996: Long-Lead
Seasonal

> Temperature Prediction Using Optimal Climate Normals. Journal of
> Climate: Vol. 9, No. 4, pp. 809--817.

Kharin, V., V., and Francis W. Zwiers. 2002: Climate Predictions with
Multimodel

Ensembles. Journal of Climate: Vol. 15, No. 7, pp.793--799.

Mo. K.C., 2003: Ensemble Canonical Correlation Prediction of Surface

Temperature over the United States. Journal of Climate: Vol. 16, 11, pp.

1665--1683.

Peng, P., A. Kumar, H. van den Dool, and A. G. Barnston (2002), An
analysis of

> multimodel ensemble predictions for seasonal climate anomalies, J.
> Geophys. Res., 107(D23), 4710, doi:10.1029/2002JD002712.

Roulston, M. S. and L. A. Smith, 2003: Combining dynamical and
statistical ensembles.

*Tellus,* 55A. 16-30.

Saha, S. Nadiga, C. Thiaw, J. Wang, W. Wang, Q. Zhang, H. M. van den
Dool, H.-L.

> Pan, S. Moorthi, D. Behringer, D. Stokes,G. White, S. Lord, W.
> Ebisuzaki, P. Peng, P. Xie, 2005: The NCEP Climate Forecast System.
> Submitted to the Journal of Climate.

Silverman, B. W., 1986: Density Estimation for Statistics and Data
Analysis. Chapman

and Hall, 175pp.

Unger, D., Anthony Barnston, Huug van den Dool and Vern Kousky, 1996
onward.

> Consolidated Forecasts of Tropical Pacific SST in Nino 3.4 Using Two
> Dynamical Models and Two Statistical Models. Experimental Long Lead
> Forecast.Bulletin. 1996 onward

Tikhonov, A., and V.Y. Arsenin. Solutions of Ill-Posed Problems.
Winston,

Washington, 1977.

Thompson, Philip Duncan, 1977: How to Improve Accuracy by Combining
Independent

Forecasts Monthly Weather Review 1977 105: 228-229

Van den Dool, H. M., and L. Rukhovets. 1994: On the Weights for an
Ensemble-

Averaged 6--10-Day Forecast. Weather and Forecasting: Vol. 9, pp.
457--465.

Van den Dool, H., J. Huang, and Y. Fan (2003), Performance and analysis
of the

> constructed analogue method applied to U.S. soil moisture over
> 1981--2001, J. Geophys. Res.,108(D16), 8617, doi:10.1029/2002JD003114.

**4. Practicalities**

1)  We restrict ourselves to hindcasts initiated in 1981-2003 (this will
    increase), because the CFS is available for 1981 forward.

2)  Hindcasts of tools are collected. Each tool first gets a workout to
    make sure the details (grid, units, etc) are correct. A basic
    judgement has to be made whether or not we TRUST the hindcast data
    set. More tools can and will be added over time.

3)  An 11 year running mean (lf) is removed from the observations as
    well as the forecasts, leaving only the higher frequencies (hf). The
    lf is set aside until the end. The consolidation is done on the hf
    part. Take Higgins et al (2004) as a protoype example as to how the
    lf/hf separation can be done.

4)  Reduce co-linearity by removing doubles (via (A,B) buddy checks),
    assigning 'equal' members of ensemble equal weights a-priori,
    removing lf, and possibly pooling data in a) space, and maybe
    months b) m-1,m,m+1, and leads c) l-1, l, and l+1.

5)  Cross Validation, to be worked out (a major issue)

6)  Normal period for verification

7)  Dealing with methods that have already been CV-ed in their own right
    and suffer degeneracy

8)  The lf part (observed) is put back into the point forecast

9)  

**5. Links to other activities**

It goes without saying that our work will be done in close collaboration
with EMC. Within NCEP consolidation is an important near-term focus for
the CTB. In addition, we will work with the IRI, as they face the same
problem. At the very least we could learn from each other about methods
of consolidation. The practicalities at IRI and CPC, however, are very
different. We should also maintain communication with CDC on
consolidation and with DEMETER on multi-model ensembles. We have
downloaded their forecasts and can use the DEMETER data set for
experimentation, and to see where CFS stands relative to the European
efforts.

**6. Timetable and workplan**

Yr 1. Collect as many hindcast data sets of as many in-house methods as
we can collect at short notice. Subject each hindcast data set to basic
sanity check, and verify the units, grid, periods, range of leads etc
etc. Experiment with the ridge regression method to assign weights to
methods. Do this for any combination of M methods out of N (M\<N) so as
to make sure that using all N at the same time produces reasonable
answers. Preliminary test of the GDK method.

Beyond the first year we envision the following:

Yr 2. Collect more hindcast data sets. Formulate a protocol for any new
methods to participate. Finalize the ridge regression method and
generate weights. Prepare article about consolidation as per ridge
regression+GDK. Organize a workshop (paid for by??) for interested
parties (IRI, Europeans) focused on tool consolidation.

Yr3. Guide implementation, make adjustments. Implement a procedure that
can adjust to change (at least in terms of participating methods).
Prepare final paper on results. Hold the workshop and prepare
recommendations for the future.

**7. Readiness**

We claim CPC is ready for the proposed work because it is clearly in
CPC's interest, and we have people with the right talent and interest to
guide this work. In summer 2005 CPC will have a framework for a
consolidated forecast in place so any results of this proposal should
flow into operations without further difficulty.

8\. **BUDGET (\$K)**

Title of proposal: Consolidation of multi-method Seasonal Forecasts at
CPC

Principal Investigators: H. van den Dool

Year 1st

**Salaries and Overhead\***

Huug van den Dool (1 mo/yr) N/C

David Unger (1 mo/yr) N/C

Technician (0.5 mo/yr) N/C

1 or 2 Contract Support Scientists

145.0

> \-\-\-\-\-\-\-\-\-\-\-\-\--

Sub-totals 145.0

**Equipment**

Computer & 6.0

Hard Disk

**Travel** 5.0

**Publications** 4.0

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

TOTALS 160.0

\* Salaries include an overhead and benefits factor of 1.75 for
contractor support. Note the large CPC/NCEP contribution in personnel
plus partial computer costs at NC to CDEP/OGP.

**Explanation of the budget items**

**Salaries and Overhead**

H. van den Dool, with help of David Unger, will lead this project as the
PI, and is responsible for the overall coordination and management of
all the work related to the project. No cost to the project.

One or two contractors will work full time on this project mainly in
program coding, model output processing, verification and analysis.

Technician, will partially provide technical support with no cost to the
project.

**Equipment:** Support is being sought for purchasing a stand-alone PC
or workstation and hard disk during the lifetime of the proposal to
handle model data sets and analysis.

**Computer time:** Computer time and partial file-server storage will be
supported by CPC/NCEP with no cost to the project.

**Travel:** Support is being sought for one or two domestic scientific
and technical meetings per year.

**Publications:** Partial support is being sought to cover page charges
for about 1 or 2 publications per year

**Note.** Since this only a one-year proposal it may be hard to hire
anyone (unless OGP promises funding for more years). Quite possibly we
need to hire someone in-house at NCEP who needs to be carried forward.

#### 10 VITAE

[Name:]{.underline} **HUGO MARINUS VAN DEN DOOL**

[Work Affiliation]{.underline}: Climate Prediction Center

NCEP/NWS/NOAA

W/NP51, WWB, Rm. 604

5200 Auth Road

Camp Springs, MD 20746

tel (301) 763-8000 X 7570

fax (301) 763-8395

e-mail **huug.vandendool@noaa.gov**

[Education:]{.underline}

1975: Ph. D., Physics and Mathematics (Specialization in Dynamic
Meteorology), University of Utrecht, The Netherlands.

Ph D Advisors: E.O. Holopainen (Helsinki, Finland) & F. H. Schmidt
(Utrecht, The Netherlands)

1971: Doctoraal examen (Master), Physics and Mathematics, University of
Utrecht, The Netherlands.

1968: Kandidaatsexamen (Bachelor), Physics and Mathematics, University
of Utrecht, The Netherlands.

[Employment History:]{.underline}

1999-: Principal Scientist, Climate Prediction Center and Adjunct
Professor University of Maryland

1990-1999: Chief of the Prediction Branch of the Climate Prediction
Center (CPC/NCEP/NWS/NOAA), Washington DC.

1986-1992: Associate Research Scientist, Department of Meteorology,
University of Maryland

1984-1985: Research Associate, Department of Meteorology, University of
Maryland

1983: Visiting scientist, Scripps Institution of Oceanography, UC, San
Diego.

1981-1982: Adjunct Professor, Department of Physics, University of
Leuven, Belgium

1975-1985: Research Scientist, Climate and General Circulation Research
Group,

Royal Netherlands Meteorological Institute, De Bilt, The Netherlands.
Chairman of the Group from 1979-1981. (Mostly on leave from 1982-1985)

1971-1975: Research Associate, Institute of Meteorology and
Oceanography,

University of Utrecht, The Netherlands

1969-1970: Teacher Physics Atheneum Gouda

Publications PI since 2000:

Van den Dool, H.M., S. Saha and A. Johansson, 2000: Empirical Orthogonal

Teleconnections. [J. Climate]{.underline}, 13, 1421-1435.

van den Dool, H., 2000: Multiple entries in : Glossary of Meteorology,
2^nd^ edition published by the AMS.

I. Velicogna, J. Wahr and H. van den Dool, 2001: Can surface pressure be
used to remove atmospheric contributions from GRACE data with sufficient
accuracy to recover hydrological signals? J. Geophys. Res. Vol. 106 ,
No. B8 , p. 16,415

R. Kistler, E. Kalnay, W. Collins, S. Saha, G. White, J. Woollen,
M.Chelliah, W. Ebisuzaki, M. Kanamitsu, V. Kousky, H. van den Dool, R.
Jenne, M. Fiorino, 2001: The NCEP--NCAR 50--Year Reanalysis: Monthly
Means CD--ROM and Documentation. Bulletin of the American Meteorological
Society: Vol. 82, No. 2, pp. 247--268.

P. Peng, A. Kumar, Huug van den Dool, and Anthony G. Barnston, 2002: An
analysis of multimodel ensemble predictions for seasonal climate
anomalies. J. Geophys. Res., 107 (D23), 10.1029/2002JD002712.

Koster, R. D., M. J. Suarez, R. W. Higgins, and H. M. Van den Dool,
Observational evidence that soil moisture variations affect
precipitation, Geophys. Res. Lett., 30(5),
1241,doi:10.1029/2002GL016571, 2003.

Van den Dool, H. M., Jin Huang and Yun Fan, 2003: Performance and
Analysis of the Constructed Analogue Method Applied to US Soil Moisture
over 1981-2001. J. Geophys. Res., 108(D16), 8617,
doi:10.1029/2002JD003114,2003.

Fan, Y., and H. van den Dool (2004), Climate Prediction Center global
monthly soil moisture data set at 0.5° resolution for 1948 to present,
J. Geophys. Res., 109, D10102, doi:10.1029/2003JD004345.

van den Dool, H. M., Peitao Peng, Ake Johansson, Muthuvel Chelliah and

> Amir Shabbar. 2004: Seasonal_to_Decadal Predictability and Prediction
> of North American Climate \_ The Atlantic Influence. White Paper #7.
> Proceedings of CLIVAR workshop on Atlantic Variability, University of
> Reading, UK, April, 19-22, 2004. Clivar Publication Series no.81, WRCP
> Informal Report No. 11/2004, p225-236. (The same, in abbreviated form,
> also appeared in CLIVAR-exchanges, Volume 9 No.3 September 2004, p17 &
> 23-25.)

H. M. van den Dool, 2004: Climate Prediction at CPC. Proceedings of
CLIVAR workshop on Atlantic Variability, University of Reading, UK,
April, 19-22, 2004. Clivar Publication Series no.81, WRCP Informal
Report No. 11/2004, p44-45.

Waliser, D. , Klaus Weickmann, Randy Dole, Siegfried Schubert, Oscar
Alves, Charles Jones, Matthew Newman, Hua-Lu Pan, Andres Roubicek,
Suranjana Saha, Cathy Smith, Huug van den Dool, Frederic Vitart, Matthew
Wheeler, Jeffrey Whitaker, 2005: The experimental MJO .
Bull.Amer.Met.Soc, 86,

H. M. van den Dool, Peitao Peng, Ake Johansson, Muthuvel Chelliah and

> Amir Shabbar. 2005: Seasonal_to_Decadal Predictability and Prediction
> of North American Climate \_ The Atlantic Influence. J. Climate, 18, .
> (Special 'Atlantic issue'.)

S. Saha, S. Nadiga, C. Thiaw, J. Wang, W. Wang, Q. Zhang, H. M. van den

Dool, H.-L. Pan, S. Moorthi, D. Behringer, D. Stokes,G. White, S. Lord,
W. Ebisuzaki, P. Peng, P. Xie, 2005: The NCEP Climate Forecast System.
Submitted to the Journal of Climate

Yun Fan, Huug Van den Dool, Ken Mitchell, Dag Lohmann, 2005: 1948-1998
US Hydrological Reanalysis by the Noah Land Assimilation System. Under
Review J. Climate.

Yang, S., M. Cai, H. van den Dool and V. Kousky, 2005: Dynamical
Implications

of the Shape of Atmospheric Eddies. Submitted to JAS.

**Appendix.**

We have created a technically unsolvable problem with *[66]{.underline}*
coefficients**.** We made 30 equations (not shown), and the remaining 33
are just linear combinations of the first 30. The simple-minded solution
would feature positive weights for all 66 coefficients. The first Table
below is the solution. Given are 66 coefficients. On the far right are
three control quantities, the sum of the weights, the sum of the square
of the weights and the sume of the absolute values of the weights.
Clearly the solution makes little sense.

 \
1 -2.38 11 .27 21 -.58 31 .69 41 -1.00 51 -.80 61 -1.65 14.83Σw\
2 -4.31 12 2.09 22 4.04 32 5.14 42 -5.01 52-11.79 62 -2.89 1250.21Σw\*w\
3 .73 13 6.60 23 7.15 33 .58 43 3.58 53 2.27 63 6.97 213.27Σ\|w\|\
4 -1.53 14 -3.40 24 -2.06 34 .39 44 -3.46 54 4.72 64 3.01\
5 2.80 15 4.03 25 -4.35 35 -1.37 45 1.47 55 5.17 65-12.07\
6 1.93 16 -1.39 26 -2.40 36 .77 46 1.27 56 1.55 66 -2.40\
7 .69 17 -.72 27 2.56 37 3.95 47 -4.41 57 14.55\
8 3.82 18 -.19 28 -.16 38 .12 48 -.77 58 -9.84\
9 .48 19 -2.00 29 .59 39 -2.59 49 2.25 59 7.10\
10 3.96 20 -5.52 30 -3.52 40 3.48 50 -4.67 60 3.28\
 \
After adding =0.000000000000005 to the main diagonal of the covariance
matrix (not shown) the solution is already very reasonable:\
 \
1 .13 11 .06 21 .12 31 .03 41 .01 51 .04 61 .04 2.45 Σw\
2 .12 12 .10 22 .07 32 .00 42 .02 52 -.01 62 .07 .23Σw\*w\
3 .19 13 .06 23 -.07 33 .01 43 .04 53 .00 63 .01 3.04Σ\|w\|\
4 -.03 14 .04 24 -.01 34 -.04 44 .01 54 -.01 64 .04\
5 -.04 15 .09 25 .02 35 .04 45 .05 55 .07 65 .04\
6 .06 16 .11 26 .06 36 .04 46 .05 56 .03 66 .05\
7 .00 17 .06 27 .09 37 -.03 47 .05 57 .05\
8 .10 18 .10 28 .00 38 .07 48 .04 58 .03\
9 .06 19 .04 29 .03 39 .00 49 .02 59 .04\
10 .01 20 -.02 30 .02 40 -.04 50 -.01 60 .04

Suddenly the sum of the square of the weights goes down from 1250.21 to
0.23. Indeed ridging is enormously powerful in stabilizing the solution
of linear system. If one cannot live with negative weights (a lot) more
ridging needs to be applied.

Fig.1 A real life example of a consolidation map used at CPC for the
AMJ2005 forecast. Only two methods are combined here, CCA and OCN. The
forecasts are expressed as anomalies in units of standard deviation
times 10. Positives are in red, negatives in blue. The a-priori skill
estimate is conveyed by the size of the numeral. In areas where the
skill, in terms of correlation, is less than 0.3 no forecast is given at
all -- in this case the location of the
![](media/image1.png){width="4.2034722222222225in"
height="4.375in"}station is shown by the plus sign, where the color
indicates the sign of the anomaly.

Fig.2 shows the probability shift (contours), relative to 100\*1/3rd, in
the above normal class as a function of a-priori correlation (R ,
y-axis) and the standardized forecast of the predictand (F, x-axis). The
prob.shifts increase with both F and R. The R is based on a sample of
30, using a Gaussian model to handle its uncertainty. The same diagram
is automatically usuable for probability shift for the B class. For the
N class we have another diagram. Notice the asymmetry for large values
of the predictand -- the extreme class cannot give up more than
33.1/3^rd^ while it can gain 66.2/3^rd^ at the expense of both
![](media/image2.jpeg){width="4.647916666666666in"
height="5.916666666666667in"}the other extreme and N. Source Unger 1998

Figure 3. Diagram of Gaussian kernel density method to form a
probability distribution function from individual ensemble forecasts.
Four ensemble members are used in this example to produce a
consolidation forecast distribution. E represents the spread, F~m~ is
the ensemble mean, and S~z~ is the standard deviation of the Gaussian
kernel distribution. The x-axis represents some forecast variable, such
as air temperature in Degrees F, and the y-axis is probability density.
S~z~ is the same for all 4 kernels but the area underneath each kernel
varies according to the weight assigned to the member.

![](media/image3.png){width="6.25in" height="4.375694444444444in"}