----------------------------------------------------- --------------------------------------- ----------------------------------------------------
  ![](media/image1.jpeg){width="1.4791666666666667in"   ***Advanced Scientific\                 ![](media/image2.png){width="0.6666666666666666in"
  height="0.4791666666666667in"}                        Computing Research***                   height="0.4930555555555556in"}

  ----------------------------------------------------- --------------------------------------- ----------------------------------------------------

> **Bridging the Gap between Climate and Weather:\
> The 20th Century Reanalysis Project (2007)**

## Gil Compo, Jeffrey Whitaker, and Prashant Sardeshmukh, University of Colorado/CIRES and NOAA Earth System Research Lab 

***The 20th Century Reanalysis Project is building the first complete
database of three-dimensional global weather maps of the 20th century.
The new 3D atmospheric dataset will provide missing information about
the conditions in which early-century extreme climate events occurred,
such as the Dust Bowl of the 1930s and the Arctic warming of the 1920s
to 1940s. This dataset will also provide an important validation check
on the climate models being used to make 21st century climate
projections by comparing their simulations of the 20th century to the
observed variations in weather extremes.***

The distinction between climate and weather was expressed most
succinctly by science fiction writer Robert A. Heinlein: "Climate is
what you expect; weather is what you get." But as global warming
produces more noticeable changes on a planetary scale, how do we even
know what to expect in a particular region?

Climate change studies are increasingly focused on understanding and
predicting regional changes of daily weather statistics. But to predict
the next century's statistical trends with confidence, researchers have
to demonstrate that their forecasting tools can successfully recreate
the conditions of the past century. That requires a detailed set of
historical atmospheric circulation data---not just monthly averages, but
statistics for at least every six hours, so that phenomena like severe
storms can be analyzed.

Although there is scant atmospheric data from weather balloons and none
from satellites for the first half of the 20th century, there is an
enormous amount of observational data collected at the Earth's surface
by a variety of sources, from meteorologists and military personnel to
volunteer observers and ships' crews. Until recently, this
two-dimensional data was widely available only on hand-drawn weather
maps. Despite many errors, these maps are indispensable to researchers.

Now, using the latest data integration and atmospheric modeling tools
and an INCITE award of 2 million supercomputing hours at NERSC,
scientists from the NOAA Earth System Research Lab and the Cooperative
Institute for Research in Environmental Sciences (CIRES) are building
the first complete database of three-dimensional global weather maps of
the 20th century.

Called the 20th Century Reanalysis Project, the new dataset will double
the number of years for which a complete record of three-dimensional
atmospheric climate data is available, extending the usable digital
dataset from 1948 back to 1892. The final maps will depict weather
conditions every six hours from the Earth's surface to the level of the
jet stream (about 11 km or 36,000 ft high), and will allow researchers
to compare the patterns, magnitudes, means, and extremes of recent and
projected climate changes with past changes.

"We expect the reanalysis of a century's worth of data will enable
climate researchers to better address issues such as the range of
natural variability of extreme events including floods, droughts,
hurricanes, extratropical cyclones, and cold waves," said principal
investigator Dr. Gil Compo of CIRES, a joint institute of NOAA and the
University of Colorado.

Dr. Compo and his colleagues have discovered that using only surface air
pressure data, it is possible to recreate a snapshot of other variables,
such as winds and temperatures, throughout the troposphere, from the
ground or sea level to the jet stream. This discovery makes it possible
to extend two-dimensional weather maps into three dimensions.

The computer code used to combine the data and reconstruct the third
dimension has two components. The forecast model is the atmospheric
component of the Climate Forecast System, which is used by the National
Weather Service's National Centers for Environmental Prediction (NCEP)
to make operational climate forecasts. The data assimilation component
is the Ensemble Kalman Filter.

Data assimilation is the process by which raw data such as temperature
and atmospheric pressure observations are incorporated into the
physics-based equations that make up numerical weather models. This
process provides the initial values used in the equations to predict how
atmospheric conditions will evolve. Data assimilation takes place in a
series of analysis cycles. In each analysis cycle, observational data is
combined with the forecast results from the mathematical model to
produce the best estimate of the current state of the system, balancing
the uncertainty in the data and in the forecast. The model then advances
several hours, and the results become the forecast for the next analysis
cycle.

The Ensemble Kalman Filter is one of the most sophisticated tools
available for data assimilation. Generically, a Kalman filter is a
recursive algorithm that estimates the state of a dynamic system from a
series of incomplete and noisy measurements. Kalman filters are used in
a wide range of engineering applications, from radar to computer vision
to aircraft and spacecraft navigation. The 20th Century Reanalysis
Project uses the Ensemble Kalman Filter to remove errors in the
observations and to fill in the blanks where information is missing,
creating a complete weather map of the troposphere.

Rather than making a single estimate of atmospheric conditions at each
time step, the Ensemble Kalman Filter reduces the uncertainty by
covering a wide range---it produces 56 estimated weather maps (the
"ensemble"), each slightly different from the others. The filter blends
the forecasts with the observations, giving more weight to the
observations when they are high quality, or to the forecasts when the
observations are noisy. The NCEP forecasting system then takes the
blended 56 weather maps and runs them forward six hours to produce the
next forecast. Processing one month of global weather data takes about a
day of computing on 2400 cores, with each map running on its own
processor. The Kalman filter is flexible enough to change continuously,
adapting to the location and number of observations as well as
meteorological conditions, thus enabling the model to correct itself in
each analysis cycle.

The team aims to reduce inconsistencies in the atmospheric climate
record, which stem from differences in how and where atmospheric
conditions are observed. Until the 1940s, for example, weather

![](media/image3.png){width="3.1368055555555556in"
height="1.9472222222222222in"}

*Figure 1. The 20th Century Reanalysis Project successfully recreated
the deadly Knickerbocker Storm of January 28, 1922: (a, b) sea level
pressures with color showing uncertainty and dots indicating measurement
locations; (c) precipitation; (d) temperature.*

and climate observations were mainly taken from the Earth's surface.
Later, weather balloons were added. Since the 1970s, extensive satellite
observations have become the norm. Discrepancies in data resulting from
these different observing platforms have caused otherwise similar
climate datasets to perform poorly in determining the variability of
storm tracks or of tropical and Antarctic climate trends. In some cases,
flawed datasets have produced spurious long-term trends.

The new 3D atmospheric dataset will provide missing information about
the conditions in which early-century extreme climate events occurred,
such as the Dust Bowl of the 1930s, the Arctic warming of the 1920s to
1940s, and the deadly Knickerbocker Storm of January 28, 1922 (Figure
1). But the most important use of weather data from the past will be the
validation of climate model simulations and projections into the future.

**Publications:**

G. P. Compo, J. S. Whitaker, and P. D. Sardeshmukh, "The 20th Century
Reanalysis Project," Proceedings of the American Meteorological Society
88^th^ Annual Meeting, 20--24 January 2008, New Orleans, LA.

J. S. Whitaker, G. P. Compo, and P. D. Sardeshmukh, "Ensemble Data
Assimilation for Historical Reanalysis," Proceedings of the Third WCRP
International Conference on Reanalysis, 28 January--1 February, 2008,
Tokyo, Japan.

**Funding:**

INCITE, CIRES, NOAA.