Journal of the Meteorological Society of Japan, Vol. 90A, pp. 87--100, 2012.

87

DOI:10.2151/jmsj.2012-A04

Relationship of the Reproducibility of Multiple Variables among Global Climate Models

Kazuaki NISHII, Takafumi MIYASAKA1, Hisashi NAKAMURA2
Department of Earth and Planetary Science, The University of Tokyo, Tokyo, Japan

Yu KOSAKA
International Paciﬁc Research Center, University of Hawaii, Honolulu, Hawaii, USA

Satoru YOKOI, Yukari N. TAKAYABU
Atmosphere and Ocean Research Institute, The University of Tokyo, Kashiwa, Japan

Hirokazu ENDO
Meteorological Research Institute, Tsukuba, Japan

Hiroki ICHIKAWA
Graduate School of Environmental Studies, Nagoya University, Nagoya, Japan

Tomoshige INOUE
Graduate School of Life and Environmental Sciences, University of Tsukuba, Tsukuba, Japan

Kazuhiro OSHIMA
Faculty of Environmental Earth Science, Hokkaido University, Sapporo, Japan

Naoki SATO
Tokyo Gakugei University, Tokyo, Japan
Japan Agency for Marine-Earth Science and Technology, Yokosuka, Japan

andYoko TSUSHIMAMet O‰ce Hadley Centre, FitzRoy Road, Exeter EX1 3PB, UK(Manuscript received 18 March 2011, in ﬁnal form 29 August 2011)
Corresponding author and present a‰liation: Kazuaki Nishii, Research Center for Advanced Scienceand Technology, The University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8904, Japan.
E-mail: nishii@atmos.rcast.u-tokyo.ac.jp
1 Present a‰liation: Research Center for Advanced
Science and Technology, The University of Tokyo.
2 Present a‰liation: Research Center for Advanced
Science and Technology, The University of Tokyo.
6 2012, Meteorological Society of Japan

Journal of the Meteorological Society of Japan

88

Vol. 90A

Abstract
Numerous e¤orts have been made for evaluating the performance of global climate models with such expectation that those models with higher reproducibility of the current climate should provide more reliable projectionsof climate changes into the future. Attempts have been made to deﬁne a single general metric through which theoverall performance of a global climate model can be assessed. On the basis of general metrics deﬁned throughseveral techniques of multivariate analysis, the present study compares global climate models from a viewpointof their reproducibility of climatological-mean ﬁelds of multiple variables. The analyses indicate that a reproducibility of a particular variable is not necessarily independent of that of others, which may bring redundant
information into a general metric. The model reproducibility in upper and mid-tropospheric temperature andlower-tropospheric humidity, for example, tends to be anti-correlated with that in upper and mid-tropospherichumidity. It is argued that attention has to be paid to this kind of trade-o¤ relationships among some variables
and resultant redundancy in synthesizing multiple metrics. A possibility is suggested that an arbitrary selection ofvariables can yield some redundant information of variables. The redundancy is, however, found to exert no serious inﬂuence on the quality of a general metric as long as it is based on the su‰cient number of variables. In our
attempt to evaluate the climate models by introducing general performance metrics with reduced redundancy ofvariables, the overall model ranking is found rather insensitive to the speciﬁc deﬁnition of the metric.

1. Introduction
Quantitative projections of future climatechanges depend more or less on numerical climate
models. A multi-model ensemble (MME) is known
to outperform individual models in reproducing thecurrent climatic state owing to a tendency for theirbiases to cancel each other (e.g., Knutti et al. 2010).
The MME future projection has therefore been believed to be more reliable than the corresponding
projection based on a single model, as exempliﬁed
in the Intergovernmental Panel on Climate Change(IPCC) Fourth Assessment Report (AR4; Solomon
et al. 2007). In AR4 a simple algebraic average ofthe outputs from more than 20 global climate models that participated in the World Climate Research
Programme’s (WCRP’s) Coupled Model Intercomparison Project Phase 3 (CMIP3; Meehl et al. 2007)is used as the best guess for the future projection.
The cancellation of model biases is, however, notnecessarily perfect. For example, a group of theCMIP3 models in which a particular parameterization scheme is commonly adopted, say, for cumulus
convection may su¤er from a common bias, suggesting that model biases are not necessarily distributed randomly. Even if model biases were distributed randomly, the number of available models
would be unlikely su‰cient for their perfect cancellation (e.g., Knutti et al. 2010). In fact, the e¤ective
number (or degrees of freedom: DOFs) of theCMIP3 models has been estimated to be only between ﬁve and ten (Jun et al. 2008a, 2008b; Knutti
et al. 2010; Pennell and Reichler 2010). In otherwords, the amount of information provided as an
ensemble of those models may be less than what

would be expected under the assumption that all
the models were mutually independent1.
Spatial similarity of biases in such a model variable as climatological-mean surface air temperature(SAT) is often used as a measure of independency
among the models. In addition to the insu‰cient effective number of models as discussed above, thee¤ective number of these measures may also be limited. In fact, Yokoi et al. (2011) have demonstrated
that a performance metric for a given variable(hereafter referred to as ‘‘variable metric’’), whichquantiﬁes the similarity of its model-simulated distribution to its observational counterpart, may becorrelated with other variable metrics under theconstraint, for example, of thermal wind balancethat relates circulation and thermal ﬁelds.
E¤orts have been made to deﬁne a single general
performance metric (hereafter referred to as ‘‘general metric’’) into which various aspects of model
performance are incorporated (Gleckler et al.
2008; Reichler and Kim 2008). This general metriccan be used for determining weights for individualmodels to synthesize their outputs for deﬁning an
optimal MME (e.g., Murphy et al. 2004). Usually
in deﬁning a general metric, reproducibility of various variables is estimated separately on the basis ofvariable metrics before summed up, but what variables to be chosen is rather arbitrary. In fact,
Knutti et al. (2010) pointed out ‘‘there is virtually3

Annan and Hargreaves (2010) showed that in a paradigm of statistically indistinguishable ensemble,
CMIP3 models are well distributed in a sense that observations can be considered as a member of theCMIP3 ensemble.

February 2012

K. NISHII et al.

an inﬁnite number of metrics that can be deﬁned’’.
Furthermore, Yokoi et al. (2011) argued that a general performance metric might be marred if seriously biased variables are incorporated into it. Furthermore, adding a new variable metric to a general
metric may not necessarily lead to an e¤ective increase in the information included in the metric, if
the new variable is linked closely to any of the variables that have already been incorporated into themetric. In this case, the addition will introduce
some redundant information, or even some bias, tothe new general metric. In any case, ‘‘we currently
have no basis for assigning unequal weights forany variables’’ (Sexton and Murphy 2003) in deﬁning a general metric.
This study is motivated by Gleckler et al. (2008),
who argued ‘‘it might be fruitful to explore a wide
range of metrics, rather than striving for a single
index of overall skill, and then to use some objective method to reduce redundant information (e.g.,SVD)’’. We examine linkages among variable metrics for the CMIP3 models by applying several
techniques of multivariate analysis. We identify
positively-correlated variable metrics in particularvariable groups and other metrics showing tradeo¤ reproducibility of variables. We then propose
several deﬁnitions for general metrics in our attempt to reduce redundancy.
The metrics deﬁned in the following sections are
based only on the climatological-mean state. It
should be pointed out that they do not necessarily
capture every aspect of the performance of a climate model, since its reproducibility of the meanstate and that of natural variability around it do
not necessarily correlate positively (Gleckler et al.
2008; Santer et al. 2009). Another possible defectof our metrics arises from their rather straightforward deﬁnition. It has been pointed out that mostof such straightforward metrics as area-mean biases
and root-mean-square errors for the present day climate do not necessarily be applicable well to future
projections (Whetton et al. 2007; Abe et al. 2009;
Girogi and Coppola 2010; Knutti et al. 2010). Recently, e¤orts have been devoted to ﬁnding metrics
that can connect current climate reproducibility
reasonably to future projection (Hall and Qu 2006;
Boe et al. 2009; Shiogama et al. 2011), where these
metrics are expected to reduce uncertainty in future
projections based on ensembles of climate models.
In addition, a new paradigm of a statistically indistinguishable ensemble has been proposed (Annan
and Hargreaves 2010), which di¤ers from the par-89

ticular paradigm we adopt here that ensemble
members are assumed to be distributed around thetrue climate. Despite the defects mentioned above,
we nevertheless use our metrics because our main
goal is to explore inter-variable relationships ofmultiple metrics.
2. Data and analysis methods
2.1 Climate models and observed data
The multi-model dataset of the 20th Century Climate in Coupled Models (20C3M) experiment in
CMIP3 (Meehl et al. 2007) is utilized in this study.
In Table 1, the 22 variables used for our analysis
are listed with their abbreviations for reference.
For each of the variables, model output data from24 climate models are compared with observational
data whose source and available periods are also
listed in Table 1. Most of the variables are obtained
from the Japanese 25-year reanalysis (JRA-25) of
the global atmosphere (Onogi et al. 2007). We
have veriﬁed that the usage of the European Centre
Medium-Range Weather Forecast 40-yr Reanalysis(ERA40) data set (Uppala et al. 2005) in place ofJRA-25 yields no substantial changes in the resultspresented below. We deﬁne a variable metric forthe i-th model (i ¼ 1; . . . ; I ) and the j-th variable( j ¼ 1; . . . ; J) as
vﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
u
12 X
L
1u
1 X
Cij ¼ tð1Þ
wl ðmijkl  oijkl Þ 2 ;sj 12W k lwhere sj denotes standard deviation of the observed interannual variability of the j-th variable,
wl a local area weighting factor at the l-th grid
point (l ¼ 1; . . . ; L), W ¼ Swl , and mijkl and oijkl
are the simulated and observed climatologicalmeans of the j-th variable for the k-th calendar
month (k ¼ 1; . . . ; 12), respectively. Sj Cij2 =J isequivalent to the Climate Prediction Index (CPI;
Murphy et al. 2004) for the i-th model. Since available periods for observed OLR and SWTOA are
too short for a robust estimation of their interannual variances (Table 1), the estimation was basedon the JRA25 data. A shortcoming of such metrics
as ours that include mean square errors is that theycannot incorporate the signs of model errors. This
may artiﬁcially reduce the e¤ective variable number
estimated in our analysis.
The inter-model variance in C is not necessarily
comparable in magnitude among the variables. For
example, standard deviations are large in upper andmid-tropospheric temperature and speciﬁc humidity

Journal of the Meteorological Society of Japan

90

Vol. 90A

Table 1. List of used variables and reference dataset. JRA25 is for Japan Re-Analysis (Onogi et al. 2007). HadSST2 isfor the Second Hadley Centre Sea Surface Temperature dataset (Rayner et al. 2006). ISCCP is for the International
Satellite Cloud Climatology Project (Rossow and Schi¤er 1999). ERBE is for Earth Radiation Budget Experiment(Barkstrom et al. 1989). CMAP is for the CPC Merged Analysis of Precipitation (Xie and Arkin 1997).
Variable

Description

Reference

PeriodSLP

Sea level pressure

JRA25

1979–1999

U200

200-hPa zonal wind

JRA25

1979–1999

U850

850-hPa zonal wind

JRA25

1979–1999

V200

200-hPa meridional wind

JRA25

1979–1999

V850

850-hPa meridional wind

JRA25

1979–1999

T300

300-hPa air temperature

JRA25

1979–1999

T500

500-hPa air temperature

JRA25

1979–1999

T600

600-hPa air temperature

JRA25

1979–1999

T700

700-hPa air temperature

JRA25

1979–1999

T850

850-hPa air temperature

JRA25

1979–1999

Q300

300-hPa Speciﬁc humidity

JRA25

1979–1999

Q600

600-hPa Speciﬁc humidity

JRA25

1979–1999

Q700

700-hPa Speciﬁc humidity

JRA25

1979–1999

Q850

850-hPa Speciﬁc humidity

JRA25

1979–1999

LH

Surface latent heat ﬂux

JRA25

1979–1999

SH

Surface sensible heat ﬂux

JRA25

1979–1999

SAT

Surface (2 m) air temperature

JRA25

1979–1999

SST

Sea surface temperature

HadSST2

1979–1999

CloudC

Cloud cover

ISSCP-D2

1984–1999

OLR

Outgoing longwave radiation

ERBE

Feb. 1985–Feb. 1990

SWTOA

Reﬂected shortwave radiation

ERBE

Feb. 1985–Feb. 1990

Prec

Total precipitation

CMAP

1979–1999

ﬁelds (Fig. 1). In Section 3, variances in C havebeen standardized with inter-model standard deviations, to explore relationships among variable
metrics. However, no standardization has been applied to C in Section 4, where we discuss general
performance metrics that have to be related to themodel reproducibility of variables and therefore
their inter-model variances must be explicitly incorporated.
2.2 Multivariate analysis techniques
In this subsection we brieﬂy introduce three multivariate analysis techniques applied to C in thepresent study. One of them is a cluster analysis. As
in Yokoi et al. (2011), we apply a cluster analysis toa set of variable metrics, to identify several groupsof variable metrics that exhibit similar behaviors.
We adopt so-called Ward (1967) method, which isbased on the Euclidian distance between any pairof clusters in the phase space.
Unlike the cluster analysis, a principal component analysis (PCA), or an empirical orthogonal
function (EOF) analysis, seeks for basis vectors thatcan be regarded as new ‘‘variable’’ metrics each ofwhich can represent behaviors of multiple variable
metrics. Before performing a PCA the RMS biases

February 2012

K. NISHII et al.

91

Fig. 2. Comparison among basis vectors,
represented by arrows a and b, obtainedby (a) PCA and (b) NMF, in a phase spaceof a hypothetical two-variable coordinate
system. Ovals in each panel denote distributions of the points that represent errors(biases) of the individual models.
Fig. 1. Inter-model standard deviations of
Cij , deﬁned in (1). See text for details.

of individual variables within the model ensemble
have been subtracted from the CPI matrix C deﬁned in (1):
(
)
1
X
1
C 0 ¼ fCij0 g ¼ Cij 
Cij :
ð2Þ
I i
The resultant matrix C 0 can be decomposed in PCA
into a pair of orthogonal matrices:
C 0 ¼ U 0 V 0T ;or

Cij0 ¼

R
X

Uir0 Vjr0 ;ð3Þrwhere U 0 ¼ fUir0 g, V 0 ¼ fVjr0 g and r ¼ 1; . . . ; R(R ¼ minðI ; JÞ). In this factorization, the i-th row
vector of C 0 (a set of variable metrics for the i-thmodel) is represented by a linear combination of
the R column vectors in V 0 , called basis vectors or
EOFs, with the corresponding i-th row vector of U 0that represents a set of their coe‰cients that scores
reproducibility of the i-th model.
As in the case of PCA, non-negative matrix factorization (NMF; Lee and Seung 1999) decomposes
the CPI matrix C in (1). Unlike PCA, however,
NMF decomposes C directly:
C A PQ T ;ð4Þtaking advantage of the fact that every element of
C is nonnegative. In (4), P and Q are nonnegativeI  R and J  R matrices, respectively, but notnecessarily orthogonal. Here, a positive integer R
satisﬁes R < IJðI þ JÞ1 .
Figure 2 schematically compares basis vectors

obtained through (a) PCA and (b) NMF applied
to a hypothetical two-variable metric data set. The
origin of the PCA basis vectors is situated at thecenter of balance between the two model groups
that corresponds to the RMS bias in (1). The leading PCA vector is in the direction of the maximum
variability of the metrics, and the second PCA vector must be orthogonal to the leading vector. In
contrast, the NMF basis vectors are not orthogonal
mutually. In a hypothetical situation where there
are only two groups of climate models as in Fig. 2,
the two NMF basis vectors are inclined to point
those groups. In the particular phase space illustrated in Fig. 2, a model with lower reproducibilityof the current climatic state tends to be more distant from the origin2. The particular distance cantherefore be regarded as a general performance
metric, and the projection of the state vector of agiven model onto a NMF basis vector can thus beconsidered as a new variable metric that comprises
multiple variables showing similar behaviors. A
general performance metric thus deﬁned should be
subject to a certain degree of redundancy, which
can nevertheless be reduced in synthesizing theseprojections. This contrasts with the PCA vectorsthat do not necessarily point the origin of the phasespace but may rather represent trade-o¤ reproducibility among the variables.
While some suggestions have been made on how
many basis vectors should be retained for PCA, no4

Here we assume that both internal climate variability and observational errors are much smaller thanthe model bias, as is likely the case for most of themodels.

Journal of the Meteorological Society of Japan92

objective criterion has been proposed thus far for
determining R in NMF. In fact, Schlink and Thiem(2009), who applied NMF to identify dominantpatterns of atmospheric variability, determined R
empirically after several trials in varying R. Whilerelative importance of a given set of PCA basis
vectors can be assessed with the corresponding eigenvalues, the order of NMF basis vectors cannot be uniquely determined. With this peculiarity
of NMF, all the basis vectors should be treatedevenly.
3. Relationship among multiple variable metrics
3.1 Cluster analysis
Figure 3 shows a dendrogram based on our cluster analysis that was applied to a set of C after
standardizing inter-model variances. We adopted
a stopping rule of Calinski and Harabasz (1974).
Though not particularly distinct, the maximum of
the pseudo-F function in their deﬁnition, which is
the ratio of the inter-cluster variance based on theFig. 3.

Vol. 90A

means of the individual clusters to the mean of theintra-cluster variances, was found to be realized
when the model members were categorized into
two main clusters. This result of our cluster analysis
may be attributable to the artifact of RMSE-based
metrics where the signs of biases are neglected. Oneof the two main clusters consists of upper and midtropospheric temperature (T300, T500, T600, T700)
and lower-tropospheric humidity (Q850), whose
combination may be understandable except forhumidity. The other main cluster, which consistsof the 17 other variables, comprises several subclusters. One of them consists of lower-tropospherictemperature (T850), SAT and sea surface temperature (SST), whose close association in the climate
models is understandable. However, interpretationof some of the other sub-clusters is not necessarilystraightforward. It seems counterintuitive, for example, that model biases in surface sensible andlatent heat ﬂuxes are not closely related to those ineither SAT or SST. As argued by Yokoi et al.

Dendrogram of the cluster analysis that is applied to Cij deﬁned in (1).

February 2012

K. NISHII et al.

(2011), the mixture between variables that can yieldmodel biases in their global-mean values (e.g., SLP
and temperature ﬁelds) and those that cannot (e.g.,
meridional wind velocity) may complicate the interpretation.

93

3.2 PCAWe applied PCA to the same set of C as above
through the eigenvalue decomposition of its correlation matrix (Fig. 4). Fractions of the total variance explained by these modes are 36%, 21%, 9%,

Fig. 4. (a) Loadings of individual variables (abscissa) for the leading PCA modes, and (b) its scores for individual models (abscissa). (c, e) Same as in (a), but for second and third modes, respectively. (d, f ) Same asin (b), but for the second and third modes, respectively.

94

Journal of the Meteorological Society of Japan

7%, 6% and 5%. Thus more than 80% of the total
variance is explained by the six leading modes,
which means that most of the information of the
22 variables can be accounted for only by thesesix modes. The ﬁrst mode represents the overallmodel performance (Fig. 4a). Models that earnlarge negative scores of this mode tend to showhigh reproducibility in 16 out of the 22 variables
but not for upper and mid-tropospheric temperature (T300, T500, T600, T700), lower-tropospherichumidity (Q850) and cloud cover (Fig. 4d). Meanwhile, reproducibility of most of these six variables
is measured by the second PCA mode (Fig. 4b),and its large negative score represents high reproducibility of those variables (Fig. 4e). In contrast
to these two leading modes, the higher modes represent trade-o¤ relationships in reproducibility
among the 22 variables (Figs. 4c and 4f ), and therefore none of these modes alone can be used asa measure of the overall performance of a given
model. The trade-o¤ relationships found in theanalysis by Yokoi et al. (2011) and ours may suggest that one should not focus too much on themodel reproducibility only of a particular aspect,
in order to avoid its over-tuning at the expense ofother aspects. We should keep in mind, however,that the trade-o¤ relationships represented by thehigher modes tend to be more or less overemphasized due to an artifact of PCA (Lee and Seung
1999).
3.3 NMF
Our cluster analysis implies that the DOFs of thevariable metrics of C may be only two, while thesix leading modes are retained for our PCA. In recognition of this uncertainty, we repeatedly applied
NMF to the standardized C, changing R from two
to six. Figure 5 presents the results for R ¼ 2 as a
typical example. In Fig. 5, a small value in Pir suggests high reproducibility of the i-th model in a particular aspect represented by the r-th column vectorof Q. The ﬁrst NMF mode for R ¼ 2 measures the
reproducibility of upper and mid-tropospheric temperature and lower-tropospheric humidity, whereasthat of upper and mid-tropospheric humidity isscored e¤ectively by the second mode. The grouping of the variables into the two NNF modes isoverall consistent with the corresponding groupingin our cluster analysis and PCA. The characteristicof the ﬁrst mode for R ¼ 2 is fairly robust as it isreproduced in the second mode for R ¼ 3 (notshown). A positive score of the ﬁrst NMF modeVol. 90A

with R ¼ 3 corresponds to lower reproducibility ofupper and mid-tropospheric humidity, T850, SAT
and SST. The third mode implies better reproducibility of temperature ﬁelds in those models withlarge Q values at the expense of that of other variables.
4. Attempts for synthesizing multiple variable
metrics for reduced redundancy
Several methods have been proposed for synthesizing multiple variable metrics, but some of them,including an algebraic mean of the variable metrics,
are rather ad hoc. Utilizing the multivariate analyses discussed above, we make several attempts to
reduce redundant information in a set of multiple
variable metrics in deﬁning a scalar metric as ameasure of model’s general performance (‘‘general
metric’’), as in Yokoi et al. (2011). In our attempts,
we try to evaluate the overall performance of thei-th model with R (r ¼ 1; 2; . . . ; R) new variable
metrics deﬁned as:
Sj ojr Cij
C~ir ¼;
Sj ojrð5Þ

where ojr signiﬁes the weighting for the r-th metricthat has been deﬁned through one of the analysis
methods discussed above. For the cluster-analysisbased CPI, ojr ¼ 1 if the j-th variable belongs to
the r-th variable cluster or ojr ¼ 0 otherwise. For
the NMF-based metrics, ojr ¼ Qjr . A new general
metric for the i-th model with reduced redundancy
may thus be given as~2^ i ¼ Sr C ir :
D
R

ð6Þ

Our cluster analysis of the unnormalized C gives us
R ¼ 3, because the pseudo F reaches its maximum
for three main clusters, whereas PCA for the unnormalized C suggests R ¼ 4, because the four leading
modes explain more than 80% of the total variance
represented as the trace of the covariance matrix of
C. On the basis of these results R ¼ 3 and 4 aretested for our NMF, but their di¤erence is so smallthat only results for R ¼ 3 are discussed in the following.
We also utilize total energy (TE; Talagrand1981), which has been used as a norm for evaluating forecast errors. In our practice, TE is integrated
over the global domain A:

February 2012

K. NISHII et al.

95

Fig. 5. (a) First column vector of Q that represents weights of individual variables (abscissa) for measure of
the reproducibility of the CMIP3 models (abscissa) as represented by (b) the column vectors of P both forthe ﬁrst mode of NMF with R ¼ 2. (c, d) As in (a, b), respectively, but for the second mode.

1
TE ¼
2

ðð (

 0 2
Cp 02
p
02
02
T þ RTr su þv þ
Trpr)
L 2 02
þ
q
dA dp
Cp Tr

ð7Þ

where primes denote deviations from the observations, u westerlies, v southerlies, Cp speciﬁc heat atconstant pressure, L latent heat, R gas constant, Ttemperature, Tr reference temperature, and q speciﬁc humidity. In (7), the vertical integration was
performed between the p ¼ 200 and 1000 (hPa) levels. No evaluation was made, however, for the termthat includes surface pressure ( ps ), which is notavailable in some of the CMIP3 model output.
Strictly speaking, TE cannot be regarded as a general metric for model performance, since solar andterrestrial radiations, surface heat ﬂuxes and cloud
cover are all excluded from it. It can neverthelesso¤er a physically meaningful means for synthesizing dynamical and thermal variables in deﬁning ametric. As another general metric, we also adopt
the same deﬁnition as the Model Climate Performance Index (MCPI; Gleckler et al. 2008), which
is a simple summation of the conventional variable metrics but with the variable metrics listed inTable 1.
Figure 6 compares the model rankings based on
the aforementioned general metrics. Models thatare evaluated at higher rankings based on a particular general metric tend to be ranked at higher
positions based on the other general metrics. Although the TE-based model ranking tends to deviate slightly from those based on the other metrics,
the overall consistency among the model rankings

96

Journal of the Meteorological Society of Japan

Vol. 90A

Fig. 6. Ranking (ordinate) of the CMIP3 models (abscissa) determined through general metrics based on the
CPI (square), cluster analysis (rhombus), NMF (downward-pointing triangle), MCPI (upward-pointing triangle) and TE (rightward-pointing triangle), as indicated. See text for details.

based on the various general metrics implies thatthe reproducibility of the dynamical variables ismore or less related to that of the physical variables.
Figure 7 shows the numbers of variable metrics
that are ranked as the top ﬁve (squares with solidline) and bottom ﬁve (triangles with dotted line)
among the 24 CMIP3 models. The models arelisted in descending order according to the CPIbased general metric. The ﬁgure indicates an overall tendency for models with higher (lower) ranking
based on the CPI-based general metric to exhibit
higher (lower) reproducibility with respect to agreater number of variable metrics. For example,
ECHAM5/MPI-OM, the best model based on thegeneral metric, is ranked among the top ﬁve of theFig. 7. The number of variable metrics (metrics) that are ranked as the top ﬁve (squares with solid line) andbottom ﬁve (triangles with dotted line) among the models. Models (abscissa) are listed in descending orderaccording to the rank of CPI.

February 2012

K. NISHII et al.

24 models with respect to as many as 15 variable
metrics, while only a single variable metric ranks
this model (ECHAM5/MPI-OM) among the bottom ﬁve. In contrast, the three models that earn
the lowest scores of the general metric are notranked among the top ﬁve with respect to any ofthe variable metrics. Meanwhile, such models as
GFDL-CM2.1, MRI-CGCM2.3.2, CSIRO-Mk3.5
and GFDL-2.0 earn the top ﬁve scores in as many
variable metrics as the higher-ranked models basedon the general metric do so. Those models exhibit,
however, the relatively low reproducibility in air
temperature and humidity, whose inter-model variances tend to be large (Fig. 1). This is hinted at inFigs. 5b and 5d, where these models earn highscores in P. Our results suggest that a general metric based on an unnormalized matrix C may likely
be inﬂuenced substantially by the reproducibility ofvariables with large inter-model variances.
5. Discussion and conclusions
In this paper, we have compared several multivariate analysis methods that can be used for extracting relationships among variable metrics.
While details are dependent of speciﬁc analysis
methods, there are nevertheless some common features in the resultant grouping of the variable metrics. Some groups of the metrics obtained as theleading PCA or NMF modes are characterized byvariable metrics whose inter-model variances arelarge and can thereby score the overall performanceof the models. In contrast, other groups represent
trade-o¤ relationships among the variables in theirmodel reproducibility.
We have also proposed several methods to reduce redundancy in variable metrics before deﬁning
a general metric that scores the general performance of climate models. Model rankings are,however, rather insensitive to the particular deﬁnition of the general performance metric (Fig. 6).
These results suggest that (i) a general performance
metric that consists of a su‰ciently large number ofvariable metrics is unlikely to be inﬂuenced signiﬁcantly by the redundancy of variables, and (ii) goodmodels tend to show high reproducibility in various
aspects, at least based on the metrics used in thisstudy (Fig. 7).
Basically our metrics are based on RMSE fromthe observed climatology3, even in the estimation
with the total energy norm. Thus one may consider
that this similarity in the deﬁnition of the metrics
based on CPI, MCPI and TE may lead to the simi-

97

larity among the model rankings based on thosemetrics as shown in Fig. 6. We have compared themodel ranking based on CPI with those on the pattern correlations and RMSE of global-mean biases(Fig. 8). Although the similarity among those threerankings is weaker if compared to that among the
rankings shown in Fig. 6, there is still a tendency
that those models with higher rankings based on
CPI tend to be ranked also in higher positions
based on the pattern correlation and global-meanbiases.
As noted in the introduction, metrics that are related to future projections have been sought (Halland Qu 2006; Boe et al. 2009; Shiogama et al.
2011). Though it is beyond the scope of the present
study, it will be valuable to assess brieﬂy whether
the simple metrics deﬁned in this study may have
any relevance to future projection. Following Abe
et al. (2009), we compared inter-model similarity
of present-day climate simulation with that of projected future change. The inter-model similarity isevaluated between possible pairs of the CMIP3
models based on CPI (Fig. 9a) or single variable
metrics (Fig. 9b). The future change is based on
the di¤erence between the averages for the two periods, one for 2070–2099 of the A1B scenario experiment and the other for 1970–1999 of the
20C3M experiment. More speciﬁcally, the former
average is assigned to mijkl and the latter to oijkl in(1). sj is based on the current climate. Figure 9b
summarizes the correlation in the inter-model similarities between the present-day climate and projected future change based on the same scatter plot
as in Fig. 9a but based on respective variable metrics. The ﬁgure indicates fairly high correlation between current climate and future change projection
based on single variable metrics, especially in OLR,SWTOA and Prec, except for tropospheric temperatures. The high correlations of variable metrics
suggests that a pair of models that simulate similar
mean ﬁelds for the present-day climate tends toyield similar future projection in the mean ﬁeld, aslong as the similarity is measured by those variables. The correlation lowers if these variables aresynthesized in the form of CPI (0.21), while the correlation is improved slightly (0.31) if temperature
5

Note that RMSE-based metrics provide us with
mixture of information on the similarity in model–
simulated and observed climatological-mean ﬁeldsof a given variable from multiple perspectives: theglobal-mean bias and pattern similarities with respect
to spatial distribution and local amplitude.

98

Journal of the Meteorological Society of Japan

Vol. 90A

Fig. 8. Comparison of rankings of the CMIP3 models. Small squares, rhombuses and double triangles denote the rankings based on CPI, horizontal pattern correlation and RMSE of global-mean biases, respectively, between simulated and observed climatological ﬁelds. In the evaluation of the latter two, the patterncorrelations and global-mean biases for single variables are ﬁrst estimated, and then their rankings among
the models are averaged, respectively. Models (abscissa) are listed in descending order according to therank of CPI. Note that ECHO-G is not listed, whose humidity data were lost due to a computer trouble.

Fig. 9. (a) Scatter plot between inter-model similarity of the 20c3m experiment (abscissa) and that of the projected future changes (ordinate) for all possible pairs of the CMIP3 models. The similarity is measured by
CPI that has been evaluated without speciﬁc humidity. The future change is based on the di¤erence between the averages for the two periods, one for 2070–2099 of the A1B scenario experiment and the otherfor 1970–1999 of the 20C3M experiment. A line represents a regression line. (b) Correlations between theinter-model similarity of the 20c3m experiment and that of the future change, which is based on the same
scatter plots as in (a) but for variables used in this study. The last one ‘‘CPI w/o T’’ denotes CPI evaluated
without T300, T500, T600, T700, and T850.

metrics are excluded. This modest correlation implies that uncertainty that could emerge in the future projection may not be well constrained by us-

ing a synthesized metric that consists of multiple
aspects, even if each of the metrics shows high correlation between the present-day climate and futureFebruary 2012

K. NISHII et al.

projection. In our analysis, high correlations arefound in some variable metrics, but the physicalreasoning has not been uncovered.
Previous studies have pointed out that theCMIP3 models are not mutually independent andtheir e¤ective number is only between ﬁve andten (Jun et al. 2008a, 2008b; Pennell and Reichler2010). The estimation of the e¤ective model number by using PCA is equivalent to that of the number of e¤ective metrics or measures of inter-model
similarity, since the numbers of nonzero eigenvalues of inter-model and inter-variable covariancematrices of C are identical. As there are inﬁnite
ways to deﬁne metrics, incorporating additional
metrics may increase the e¤ective model number.
While precise estimation of the e¤ective numbersof models and variables may be of little worth, itwill be worthwhile to deepen our understanding ofinter-model and inter-metric relationships. In Section 3, linkages were revealed among di¤erent variable metrics for the CMIP3 models. Some of themseem to reﬂect physical relationships among thevariables or in parameterization schemes, whileothers may be mere artifacts of constraints among
the variables by a particular analysis method. Further investigation is needed to identify the originsof the revealed relationships. In Section 4, we attempted to reduce redundancy among the variable
metrics in quantifying general performance of theCMIP3 models. Still, no attempt has been madefor avoiding inter-model dependency that may distort the uncertainty (i.e., PDF) of the future projection in the ensemble of the CMIP3 models.
In the present study, we have focused on the
reproducibility of the climatological-mean ﬁelds,
whereas most of the studies on model reproducibility also focus on time-variability and long-term
trends. From a regional viewpoint, however, assessing the model reproducibility of atmospheric phenomena, including tropical and midlatitude cyclones and large-scale teleconnection patterns, isnecessary for reliable projection of their futurechanges. Several studies applied process-oriented
performance metrics to the CMIP3 models (e.g.
Yokoi and Takayabu 2009; Nishii et al. 2009). Especially, Kosaka and Nakamura (2011) found thatmodels with better reproducibility of the climatological-mean ﬁelds tend to show better reproducibility of the most dominant summertime anomaly
pattern over the western North Paciﬁc. Exploring
the relationships among process-oriented regional
metrics and global metrics based on climatological-

99

mean ﬁelds and their trends will be valuable in improving global climate models.
Acknowledgments
This study is supported by the Global Environment Research Fund (S-5) of the Ministry of Environment and by the Grant-in-Aid for ScientiﬁcResearch (B) #22340135 by the Ministry of Education, Culture, Sports, Science and Technology(MEXT). The ‘‘Data Integration and Analysis
System (DIAS)’’ Fund for the National Key Technology by the MEXT provides us with e‰cient environment for handling massive data. We acknowledge the modeling groups, the PCMDI and the
WCRP’s Working Group on Coupled Modelling(WGCM) for their roles in making available the
WCRP CMIP3 multi-model dataset. Support forthis dataset is provided by the O‰ce of Science,
U.S. Department of Energy.
References
Abe, M., H. Shiogama, J. Hargreaves, J. Annan, T. Nozawa, and S. Emori, 2009: Correlation between
inter-model similarities in spatial pattern for present and projected future mean climate. SOLA, 5,
133–136.
Annan, J. D., and J. C. Hargreaves, 2010: Reliabilityof the CMIP3 ensemble. Geophys. Res. Lett., 37,
L02703, doi:10.1029/2009GL041994.
Barkstrom, B., E. Harrison, G. Smith, R. Green, J. Kibler, R. Cess, and the ERBE Science Team, 1989:
Earth Radiation Budget Experiment (ERBE) archival and April 1985 results. Bull. Amer. Meteor.
Soc., 70, 1254–1262.
Boe, J. L., A. Hall, and X. Qu, 2009: September sea-ice
cover in the Arctic Ocean projected to vanish by
2100. Nat. Geosci., 2, 341–343.
Calinski, R. B., and J. Harabasz, 1974: A dendrite
method for cluster analysis. Communications in
Statistics, 3, 1–27.
Giorgi, F., and E. Coppola, 2010: Does the model regional bias a¤ect the projected regional climatechange? An analysis of global model projections.
Clim. Change, 100, 787–795.
Gleckler, P. J., K. E. Taylor, and C. Doutriaux, 2008:
Performance metrics for climate models. J.
Geophys. Res., 113, D06014, doi:10.1029/
2007JD008972.
Hall, A., and X. Qu, 2006: Using the current seasonal
cycle to constrain snow albedo feedback in futureclim. change. Geophys. Res. Lett., 33, L03502,
doi:10.1029/2005GL025127.
Jun, M., R. Knutti, and D. W. Nychka, 2008a: Local eigenvalue analysis of CMIP3 climate model errors.
Tellus, 60A, 992–1000.

100

Journal of the Meteorological Society of Japan

Jun, M., R. Knutti, and D. W. Nychka, 2008b: Spatial
analysis to quantify numerical model bias and dependence: How many climate models are there?
J. Amer. Stat. Assoc., 103, 934947, doi:10.1198/
016214507000001265.
Lee, D. D., and H. S. Seung, 1999: Learning the parts ofobjects by non-negative matrix factorization. Nature, 40, 788–791.
Knutti, R., R. Furrer, C. Tebaldi, J. Cermak, and G. A.
Meehl, 2010: Challenges in combining projections
from multiple climate models. J. Climate, 23,2739–2758.
Kosaka, Y., and H. Nakamura, 2011: Dominant modeof climate variability, intermodel diversity and projected future changes over the summertime western
North Paciﬁc simulated in the CMIP3 models. J.
Climate, 24, 3935–3955.
Meehl, G. A., C. Covey, T. Delworth, M. Latif, B. McAvaney, J. F. B. Mitchell, R. J. Stou¤er, and K. E.
Taylor, 2007: The WCRP CMIP3 multi-model dataset: A new era in climate change research. Bull.
Amer. Meteor. Soc., 88, 1383–1394.
Murphy, J. M., D. M. H. Sexton, D. N. Barnett, G. S.
Jones, M. J. Webb, M. Collins, and D. A. Stainforth, 2004: Quantiﬁcation of uncertainties in large
ensembles of climate change prediction. Nature,
430, 768–772.
Nishii, K., T. Miyasaka, Y. Kosaka, and H. Nakamura,
2009: Reproducibility and future projection of the
midwinter storm-track activity over the Far Eastin the CMIP3 climate models in relation to the occurrence of the ﬁrst spring storm (Haru-Ichiban)
over Japan. J. Meteor. Soc. Japan, 87, 581–588.
Onogi, K., J. Tsutsui, H. Koide, M. Sakamoto, S. Kobayashi, H. Hatsushika, T. Matsumoto, N. Yamazaki, H. Kamahori, K. Takahashi, S. Kadokura,
K. Wada, K. Kato, R. Oyama, T. Ose, N. Mannoji, and R. Taira, 2007: The JRA-25 reanalysis.
J. Meteorol. Soc. Japan, 85, 369–432.
Pennell, C., and T. Reichler, 2011: On the e¤ective number of climate models. J. Climate, 24, 2358–2367.
Rayner, N. A., P. Brohan, D. E. Parker, C. K. Folland,
J. J. Kennedy, M. Vanicek, T. Ansell, and S. F. B.
Tett, 2006: Improved analyses of changes anduncertainties in sea surface temperature measured
in situ since the mid-nineteenth century: The
HadSST2 dataset. J. Climate, 19, 446–469.
Reichler, T., and J. Kim, 2008: How well do coupledmodels simulate today’s climate? Bull. Amer. Meteor. Soc., 89, 303–311.
Rossow, W. B., and R. A. Schi¤er, 1999: Advances inunderstanding clouds from ISCCP. Bull. Amer.
Meteor. Soc., 80, 2261–2287.
Santer, B. D., K. E. Taylor, P. J. Gleckler, C. Bonﬁls,
T. P. Barnett, D. W. Pierce, T. M. L. Wigley, C.
Mears, F. J. Wentz, W. Brüggemann, N. P. Gillett,
S. A. Klein, S. Solomon, P. A. Stott, and M. F.

Vol. 90A

Wehner, 2009: Incorporating model quality information in climate change detection and attribution
studies. Proc. Natl. Acad. Sci. USA, 106, 14778–
14783.
Schlink, U., and A. Thiem, 2009: Non-negative matrix
factorization for the identiﬁcation of patterns of atmospheric pressure and geopotential for the Northern Hemisphere. Int. J. Climatol., 30, 909–925.
Shiogama, H., S. Emori, N. Hanasaki, M. Abe, Y. Masutomi, K. Takahashi, and T. Nozawa, 2011: Observational constraints indicate risk of drying in
the Amazon basin. Nat. Commun., 2:253. doi:
10.1038/ncomms1252.
Solomon, S., D. Qin, M. Manning, M. Marquis, K.
Averyt, M. M. B. Tignor, H. L. Miller Jr., and Z.
Chen, 2007: Climate Change 2007: The PhysicalScience Basis, Cambridge University Press, 996 pp.
Talagrand, O., 1981: A study of the dynamics of fourdimensional data assimilation. Tellus, 33, 43–60.
Uppala, S. M., P. W. Kallberg, A. J. Simmons, U.
Andrae, V. D. C. Bechtold, M. Fiorino, J. K.
Gibson, J. Haseler, A. Hernandez, G. A. Kelly,
X. Li, K. Onogi, S. Saarinen, N. Sokka, R. P.
Allan, E. Andersson, K. Arpe, M. A. Balmaseda,
A. C. M. Beljaars, L. Van De Berg, J. Bidlot, N.
Bormann, S. Caires, F. Chevallier, A. Dethof, M.
Dragosavac, M. Fisher, M. Fuentes, S. Hagemann,
E. Holm, B. J. Hoskins, L. Isaksen, P. A. E. M.
Janssen, R. Jenne, A. P. Mcnally, J.-F. Mahfouf,
J.-J. Morcrette, N. A. Rayner, R. W. Saunders,P. Simon, A. Sterl, K. E. Trenberth, A. Untch,
D. Vasiljevic, P. Viterbo, and J. Woollen, 2005:
The ERA-40 reanalysis. Quart. J. Roy. Meteor.
Soc., 131, 2961–3012.
Ward, J. H. Jr., 1967: Hierarchical grouping to optimize
an objective function. J. Amer. Stat. Assoc., 58,
236–244.
Whetton, P., I. Macadam, J. Bathols, and J. O’Grady,
2007: Assessment of the use of current climate patterns to evaluate regional enhanced greenhouse response patterns of climate model. Geophys. Res.
Lett., 34, L14701, doi:10.1029/2007GL030025.
Xie, P., and P. A. Arkin, 1997: Global precipitation: A
17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull. Amer. Meteor. Soc., 78, 2539–2558.
Yokoi, S., and Y. N. Takayabu, 2009: Multi-model projection of global warming impact on tropical cyclone genesis frequency over the Western North
Paciﬁc. J. Meteor. Soc. Japan, 87, 525–538.
Yokoi, S., Y. N. Takayabu, K. Nishii, H. Nakamura,
H. Endo, H. Ichikawa, T. Inoue, M. Kimoto, Y.
Kosaka, T. Miyasaka, K. Oshima, N. Sato, Y.
Tsushima, and M. Watanabe, 2011: Application of cluster analysis to climate model performance metrics. J. Appl. Meteor. Climatol., 50,
1666–1675.