**Discussion on representativeness error for OSSEs**

**June 2006-November 14,2006**

**June 16, 2006**

**Ronald Errico**

I think it is important that all members in the OSSE group have a common
understanding about certain critical aspects of OSSE experiments.  For
this reason, I think it helpful to open up some of these issues for
either explicit recognition or debate.\
\
1. For all observation types currently being assimilated forward models
of some sort are already available, because they are employed within the
data assimilation system.  They may not be the best forward models and
perhaps some aspects are even crude, but they exist and have a degree of
proven utility.  The simulated obs, however, may be most realistic with
respect to the nature run if they are determined using the best forward
models we can design.  Subsequently, we can\
also consider whether we want the OSSE DAS to use some of these better
forward models.  But, using different forward models creates some other
problems for consideration, as described below.\
\
2. More difficult, is to create realistic observation and forward model
errors, especially biases or gross errors. The problem is, we really do
not know what these error characteristics are for many\
observation types as well as we would like to know them.  Also,
typically what we call observation error statistics are actually
composites of forward model (including representativeness) and
instrument error.  As we change forward models compared to those
employed in the data assimilation, the contribution by the forward model
error will be changed.\
So, as we develop forward models to simulate the observations, we must
consider the implications for how we specify the subsequent observation
errors.\
\
3. Specifying these characteristics is therefore as much art as science
at this time. The more sophisticated error characteristics we try to
incorporate, the more validation will be required. In particular, in
past attempts, it has been very difficult to add gross errors and error
biases to observations in ways that yield quality control decisions in
the OSSEs that have equivalent characteristics to those in corresponding
real data assimilation systems.  For example, the\
OSSE should have similar observation acceptance rates and geographical
distributions as in real assimilations with each data type.  There are
work-arounds for achieving this result, such as adding no gross errors
or biases to the simulated observations and relying on quality control\
decisions provided by a corresponding (in this case one-to-one) real
data. With this technique, however, it may be inappropriate to use very
different forward models than already in the assimilation, since any
really significant difference in the models likely produces biases and
some gross errors which will remain undetected as the quality control is
rendered independent of the OSSE innovations (and only dependent on the
innovations in the real system)\
\
The point is the issues of forward model selection, observation error
design, data selection criteria, and quality control in the OSSE are all
connected.  Although OSSES have been performed in the community for a
long time, the importance of these issues have often been neglected,
with the result that many researchers find the results untrustworthy.
So, we should provide the attention they deserve.  It requires
considerable understanding of the observations as well as the DAS that
determines how the data is used.

**September 25, 2006**

**Gert-Jan Marseille**\
\
An important point of discussion during the (teleconf) meeting last
Thursday on NCEP-OSSE was the simulation of observation errors and more
in particular their representativeness errors. I remembered a discussion
with my colleague Ad Stoffelen on this issue some years ago, where Ad
convinced me that the representativeness error is not directly related
to the model grid resolution, but to the spatial scales that the NWP
model can resolve.\
\
I hope to make this clear through an example.\
For instance, assume that the representativeness error is directly
related to the model grid resolution and in addition computer power
resources would be unlimited, then one could in theory increase the
model grid resolution to infinitely small grid boxes. The 
representativeness error of observations would then reduce to zero since
each observation would then be fully representative for the particular
grid box to which it coincides. If, in addition instrument errors are
negligible then the total observation error (instrument +
representativeness error) is extremely small thus giving the observation
an extremely large weight in the analysis. In other words the model
trajectory is forced to fit the observations perfectly giving small
scale structures in the resulting analysis (overfitting). This would not
be a problem if the model can handle (/represent)/ these scales, but
from practice we know that this is not the case for nowadays NWP
models.\
\
The */effective/* resolution of nowadays models is mainly determined by
the spatial scales of the structure functions of the  background error
covariance matrix and is in the order of 150-200 km, if I am correct. A
spectral analysis applied to NWP model fields as described in (Lorenc,
1992) may provide a more precise value. In addition, a spectral analysis
provides an estimate of representativeness error.\
As such, representativeness error is related to the spatial scales that
the model can resolve, hence giving a correct weight to observations in
the analysis.\
\
I hope these thoughts may serve as a starting point for the discussion
on correct modelling of observation representativeness errors for the
NCEP-OSSE.\
\
Lorenc,A.C., Graham,R.J., Dharssi,I., MacPherson,B., Ingleby,N.B.,
Lunnon,R.W.,1992, Preparation for the use of a Doppler wind lidar
information in meteorological assimilation systems, ESA-CR(P)-3454\
\
Regards, Gert-Jan

ESA provided pdf file for Lorenc et al (1992) and posted at

<http://www.emc.ncep.noaa.gov/research/osse/NR/references/Lorenc.1992.TIDCCR4129.pdf>

September 25, 2006

Dave Emmitt

Good points, Gert-Jan. In our DWL simulations we do not prescribe any

representativeness errors. The errors we track is the instrument error

derived from signal strength and the variation in LOS estimates along an

integration line (for a staring instrument). Along with the LOS wind

estimate we provide to the DAS the instrument error and the physical

length of integration. The issue that has not been resolved, in my mind,

is how these two pieces of information are used to provide the

observational weight (Obs error) to the individual measurements in the

cost function.

I suggest a series of experiments where we explore how best to use these

new data and not rely on simple extensions of our current way of

handling observations that vary in quality in a known manner. Working

with NASA/GSFC (Reale and Terry), we are experimenting with RAOB single

component vs. DWL LOS data to see just how the LOS algorithms are

weighting and using the data. Clearly a LOS measure of a single

component of the wind over a 75 km line has a different value to the

model than a RAOB observation along a thin slanted column.

Let\'s continue this discussion tomorrow at the workshop.

Dave

**September 27, 2006**

**Jack Woollen**

I thank Gert for his viewpoint, and the same goes for Tom, Ron, Dave and
everyone else who have shared their thoughts on this subject. In effect,
we have already had several lectures on representative errors. If we are
going to have more lectures, why not expand the topic to include all
error specifications and statistics. And along with, in particular, the
practical derivation of the error statistics. In this regard,
operational experience is the most relevant. Some well defined empirical
techniques are employed to derive operational NWP background and
observation error statistics (including representativeness), and it
seems reasonable to follow on these efforts for \"credible\" OSSE
calibration. I vote for some lectures which detail current procedures to
develop the operational error statistics, and how the procedures may
apply to the OSSE context.\
\
Jack

**September 25, 2006**

**Ron Errico**

Tom Schlatter called me today and we talked for more than an hour. I
think we both agree that this issue of what is rep. error, how is it
introduced and modeled, and what is truth need to be addressed before
people begin to simulate all kinds of observations. This is a critical
issue and the hard part of doing OSSEs.  I am willing to put together a
presntation about it, but can\'t before Nov. It will take at least an
hour, including discussion which I will try to focus.

The question is, what kind of rep. error should you allow to be
introduced at this stage!!!!  That is why the question need s to be
answered now.  People need to understand what they are doing.  Tom did
not realize this before I explained it to him, but now he does.  We all
need to understand this before we proceed.

The point is, when people simulate observations from the nature run,
they are already implicitly introducing representativeness errors with
respect to the DAS, even before they add further random errors  They
must understand this and its implications before developing any specific
design.

Ron

**September 27, 2006**

**Michiko Masutani**

Ron Errico has accepted to coordinate this discussion.  He will discuss
with Tom to make sure his writing will not confuse management. I will
include this in my meeting summary.  Please send any of your discussion
about \"Representativeness errors\"  to Ron. If you CC to me I will put
together as record.\
\
Thank you very much.\
\
Michiko

**October 6, 2006**

**Erik Andersson**

This is what I would do about representativeness error in OSSEs. I hope
my interpretation won\'t offend Ron Errico (!)\
\
The observation input (y) to an assimilation system is provided in terms
of departures, d=y-H(xb), from what is already known (the background,
xb), using the observation operator (H).  Lorenc(1986) has explained
that errors in this observation minus model comparison can be due to\
1) pure observation error\
2) errors in H, contributing to representativeness error\
3) observed features that are not resolved by the model, also
contributing to representativeness error\
4) background error\
\
In every-day DA the sum of the first three error sources is represented
 by the \'total observation error\', sigma_o. The fourth one is the
subject of background error modelling.\
\
In OSSEs we replace y with H(NR)+P where P is a perturbation, so that\
d_OSSE = H(NR) + P - H(xb)\
\
For a well calibrated OSSE it is essential that d_OSSE has the same
variance as real departures (d) have (for existing observing systems),
and would have (for simulated observing systems). THIS MUST BE VERIFIED
at the start of an OSSE. Agreement is achieved by adding a perturbation
P to each observation, drawn randomly from a sample with the appropriate
error variance, with three contributions:\
1) pure observation error\
2) errors in H, contributing to representativeness error (footnote: this
has to be added unless the OSSE would use a different observation
operator to generate the NR observations H1(NR), than the one used in
the assimilation H2(xb),  such that the difference between H1 and H2 was
typical of actual observation operator uncertainty - but this is not
what is usually done)\
3) observed features that are not resolved by the NR model, also
contributing to representativeness error. In our case all OBSERVED
atmospheric variability on scales smaller than the T511 resolving power.
In the free atmosphere this source of variance can be nicely estimated
from variance spectra - for near-surface observations one component of
the perturbation could be dependent of the sub-grid-scale orography (at
T511).  Note that this contribution to P is different for different
observing systems, depending on their spatial and temporal averaging and
sampling characteristics. For example, a radiosonde is a point
measurement so its representativeness error would be quite a bit larger
than that of a line-averaged Doppler Wind as provided by the ADM (as
discussed thoroughly by Ad Stoffelen and Gert-Jan Marseille in their
simulations).\
\
Hope this helps\
Best regards and have a nice weekend (the weekend has just arrived in
Europe!)   /Erik

**October 30**

**Ad Sttofelen**

Dear Erik, All\
\
Your synthesis below is very good. Indeed, the realism of modeling the
error contributions of instrument error, H operator, and NWP model
spectral truncation are fundamental to a useful OSSE.\
\
However, Gert-Jan pointed me to a phrase in your mail he thought was not
capturing the whole problem and I agree. Andrew Lorenc et al (1992;
report distributed by Michiko) define spatial representativeness error
as those spatial scales not resolved by the NWP model. These include as
you say \"all OBSERVED atmospheric variability on scales smaller than
the T511 resolving power\", but also scales LARGER than T511 present in
the real atmosphere, but not in the NWP model at T511 truncation. This
contribution tends to be larger than the contribution of scales below
the spectral truncation in my (wind) experience.\
\
Experience shows that the \"resolving power\" of high resolution models
is not right down to the scale of truncation. To illustrate this, Lorenc
et al compare temporal and spatial spectra of observation systems with
collocated NWP model data spectra. As you do say, such comparisons can
(should) be used to estimate the spatial scales OBSERVED but not present
in the NWP fields. Besides the aircraft data referred to by Lorenc et
al, many sets of temporally/spatially uncorrelated data exists to
\"measure\" this spectral gap, e.g., in situ surface data (temporal),
scatterometer winds (spatial), automatic profilers (temporal).\
\
If a background model in data assimilation contains variance at a
certain scale, it does not mean that it verifies with observations, but
in this case it will contribute to the B error and not to the
observation error. Therefore, to keep B errors down, NWP models are
spatially filtered in order to suppress the unobserved (or \"unforced\")
scales of motion. Gert-Jan eluded to this aspect before (see below).\
\
I hope this helps too.\
\
Best regards,\
\
Ad

**October 30, 2006**

**Michiko Masutani**

Jack:

I think you have comments on this subject.\
\
In the meeting at NCEP, Jack pointed out temporal representativeness
error, which may be missing from Lorenc (1992).\
In OSSE we  did not generate the data by model but generated data from
archive using interpolation.  Even with hourly archives are used there
will be misrepresented representativeness error in OSSEs.

**October 30,2006**

**Ad Sttofelen**

Good point Michiko,\
\
For temporal:\
1) I implicitly used something like a \"frozen turbulence\" hypothesis,
i.e., small spatial scales are transported by the mean flow along a
measurement point and therefore spatial smoothness will result in
temporal smoothness. As such, spatial and temporal representativeness
errors are highly correlated and one should not account them
independently; in fact, one could account the spatial and neglect the
temporal problem. It follows that temporal spectra comparisons may be
used to estimate spatial smoothness, which is why I mentioned temporal
spectra in the first place (following Lorenc et al). This hypothesis
further implies that the spatial representativeness error may be reduced
(somewhat, since only 1D) by temporal averaging of the observation.\
\
2) A practical way to get around time interpolation is to presume that
all observations are only valid at the archive times, i.e., round
simulated observation times to the nearest archive point. As such, the
information content of the GOS is preserved, while the interpolation
error is avoided.\
\
More ideas may exist on this!\
\
Best regards,\
\
Ad

**November 1^st^, 2006**

**Jack Woollen**

A simple example of the time interpolation error I was talking with
Michiko about is shown by the difference in max daily temperature found
within 3 hourly model archives versus that found in continuous
observations. More contributions from larger scales, as Ad mentions
below. My point, which also may be similar, is just that even given
methodical application of spectral analyses of \"background\"
representativeness errors, it is still a very difficult problem to
generate, and install, complete and convincing representations of error
spectra for the entire range of existing and non-existing observation
sources which need to be synthesized for OSSEs, in general.\
\
Another important part of the solution then I think, is that the realism
of errors installed in synthetic observations, including (and
especially)  in analogs of existing data sources, should be verified in
some convincing way. For example, the technique used to estimate
observation errors in an operational analysis system, can be applied to
evaluate the errors in synthetic data assimilated in an OSSE experiment.
A variation of the \"GIGO\" principle, say \"EIEO\", or just that the
\"errors-in\" with the synthetic data, should be mirrored in the
\"errors-out\" produced by the error diagnostics. Here at NCEP, Wan-Shu
Wu has applied a procedure described by Desroziers and Ivanov in QJRMS
(2001), 127, pp. 1433-1452, to update estimates of observation errors
(including representative errors) for the NCEP assimilation system. We
can also apply that method in diagnosing the OSSE, to validate the error
levels installed in various synthetic data sources, and to compare
consistency with real world analogs.\
\
Best regards,\
\
Jack

**November 1^st^**

**Ad Stoffelen**

Right Jack,\
\
Even if we try to apply a consistent model of contributing errors for
all observing systems like suggested in earlier mails (e.g., Erik\'s,
mine), and we should, we cannot precisely determine all contributing
errors terms.\
\
Applying common tools to operations for characterizing the o-a and o-b
distributions in one way or another in the OSSE thus appears necessary
indeed to verify that the OSSE is carried out realistically for all
observing systems. This is in my view an important aspect of
calibration.\
\
Best regards,\
\
Ad

**November 1^st^**

**Michiko Masutani**

A DWL measurement is generated from many shots in a volume or a line. I
wonder how thisaffect the representativeness errors.

I wonder these thought for representativeness error will give some guide
lines for the both temporal and spatial resolution for data assimilation
systems used for the particular nature run.

Michiko

**November 2^nd^**

**Erik Andersson**

As I mentioned, this has been extensively explored by Stoffelen and
Marseille - so I consider this a solved issue. You need to get the
details from them, (and add their document to the web-site.)

Michiko said "I wonder these thought for representativeness error will
give some guide lines for the both temporal and spatial resolution for
data assimilation systems used for the particular nature run."

I HOPE so. That was the whole point!\
\
You said earlier that someone has been asked to summarize this
discussion on representativeness error. Would be nice to have such a
summary document by the time of the next tele conf.\
\
Erik

**November 2^nd^**

**Ad Stoffelen**

Below my perspective:

Line: The total wind variability contribution to the representativeness
error may be split into two orthogonal parts. If one of the directions
is well sampled, then only the spatial representativeness variability
error contribution to the other component remains, i.e., sqrt(2)
reduction in SDE contribution wrt a point measurement.

Volume: The sampling area cell size is defined by the spectral
truncation of the NWP model. If the measurement sampling is uniform and
providing N samples in such cell, then effectively the spatial
representativenes variability scale is reduced by sqrt(N). For example,
we have a cell size of 100 km and 4 samples are provided in it which are
well spread. Then, the 100-km scale variation in the cell is well
sampled in both directions, but the 50-km scale variations remain
unresolved. As such, the estimated real wind variability spectrum should
not be integrated from 0-100 km, but from 0-100\*sqrt(4) km = 0-50 km in
order to compute the representativeness contribution.

The nature run is indeed temporally and spectrally truncated. As such,
spatial representativeness variance should be added randomly (the DAS
will not resolve these variations anyway) in order to realistically
complete the small-scale spectrum. For T, p, u, v, q this is pretty well
known, for cloud and aerosol variables (if needed) it is more dodgy. For
DWL, we implemented in the KNMI simulator the cloud overlap constraint
of ECMWF and the assumption of vertically independent aerosol layers
drawn from an aerosol backscatter PDF profile and taken constant over
50km in the horizontal (i.e., layered). As such, we get a realistic
observation data base.

Subsequently, it will be simplest if the OSSE DAS NWP model has a
similar spectral truncation to the one of the nature run. In that case
we may assume that the representativeness variance from which we did
draw the random small-scale variance contribution is in fact equal to
the estimated representativeness error contribution. When the OSSE DAS
NWP model truncates more, the representativeness error contribution
should be increased accordingly. (The OSSE DAS NWP model should not
truncate less!!)

In the ADM OSSE, we stored the error parameters and random realised
errors in the observation simulation files. We found, for example, that
we slightly underestimated the radiosonde wind error and as such these
somewhat had relatively large impact. We documented this in our reports.

So, with the above variance model in mind, still practical choices will
need to be made in order to limit the amount of work.

I hope the above is clear. Further comments are welcome.

Best regards,

Ad

**November 2^nd^**

**Michiko Masutani**

I have posted a report from KNMI in 2006.  I should add more references
to\
<http://www.emc.ncep.noaa.gov/research/osse/NR/references/references.html>\
Please suggest more references we can include.\
\
I wonder the representativeness error for radiance data will be quite
different.    I played around representativeness error of TOVS data. 
The impact were quite sensitive to the error assigned.   Simulated TOVS
radiance is considered to be too good.  In simulated experiment gave
better analysis when I increased rep-error used by NCEP. I wonder
rep-errors used for operation for radiance data are under estimated.\
\
I will summarize the discussion in one file so we can look at more
easily.\
\
Ron Errico agreed to give a lecture on representativeness error.   It
seems he has a lot to cover.\
\
Michiko

**November 1^st^ 2006**

**Tom Schlatter**

The observation: $y = y_{t} + \varepsilon_{m}$

*y* is the observed value, measured by some instrument. The subscript
*t* refers to the true atmospheric value. We define the true value as
the weighted average of the true atmospheric values within the volume
sampled by the instrument. (Daniel P. Petersen defined the "true"
observation in this way, but quantitatively, in his 1968 paper "On the
Concept and Implementation of Sequential Analysis for Linear Random
Fields," *Tellus*, **20**, 673-686.) Different instruments sample
different volumes so that the true value of temperature appropriate for
a rawinsonde may not match the true value appropriate for the AMDAR
system aboard a commercial jet, even if the two observations are
assigned to the same location and time. Thus, the "truth" is very much
scale-dependent, but defining it in this way saves a lot of anguish
later.

*ε~m~* refers to errors incurred during measurement or subsequent data
processing. The errors can be random or systematic (biased).

The model state: $x = x_{t} + \varepsilon_{f}$

The model state is defined by a set of parameters stored at the points
of a model grid, or, alternatively, by a set of spectral coefficients.
Following Andrew Lorenc (1986) ("Analysis Methods for Numerical Weather
Prediction," *Quarterly Journal Royal Meteorological* *Society*,
**112**, 1177-1194), the true state is defined as the true atmospheric
state containing all scales from long waves down to cloud microphysics,
but spectrally truncated to the model resolution. Scales of motion that
cannot be captured by the model grid (or within the spectral truncation)
are not included in the definition of the true state. The numerical
model forecasts the state *x*, but the forecast is subject to error
*ε~f~* , the result of truncation associated with finite differencing,
imperfect dynamics, and flawed recipes for physical processes, whether
parameterized or not.

The forward model: *H(x)*

Forecasts are usually verified against observations (sometimes against
an analysis). Because observations hardly ever fall on model grid
points, it is necessary to map the model forecast to the observation in
order to make a direct comparison. The forward model *H* does this.
Another name for *H* is *observation operator* because *H* operates on
the model grid to generate a pseudo-observation, a best estimate of the
observed value. It relies on the parameters computed by the model on the
model grid in order to make a best estimate of the observed value.
Sometimes the calculation is as simple as 3-D linear interpolation, but
if the observed quantity does not match one of the predicted quantities,
then *H* will also involve a transformation of variables. For example,
the model may predict relative humidity, but the observed quantity is
column integrated water vapor. In this case, in addition to
interpolation, the forward model has to convert the predicted relative
humidity and temperature to a specific humidity and integrate specific
humidity vertically from the surface to the top of the model atmosphere.

Representativeness: $H(x_{t}) = y_{t} + \varepsilon_{r}$

This says that if we apply the forward model to the true values on the
model grid, we should expect to get the true value at the observation
location plus an error *ε~r~*, called the representativeness error. This
error has two causes: 1) The grid volume does not match the atmospheric
volume that is the object of measurement. If the observed volume is
small compared to the model grid volume, the measurement will represent
scales of motion that the model grid cannot resolve. From the model's
standpoint, the observation contains sub-grid scale noise, and this will
contribute to the value of *ε~r~*. If the observed volume is larger than
the model grid volume (e.g., a measurement of radiance in the microwave
portion of the electromagnetic spectrum could involve a volume of
atmosphere larger than the model grid volume), then the forward model
will be an averaging operator rather than an interpolation operator.
From the model's standpoint, the observation is too smooth. 2) If a
transformation of variables is included in *H*, the relationship is
imperfectly known or it is approximated in order to minimize the number
of computations. This also contributes to the value of *ε~r~*. To
summarize, representativeness error arises from the mismatch between the
model grid volume and the volume sampled by the instrument and sometimes
also between a mismatch between the observed and predicted variables.

Application to OSSEs:

In practice, real observations come with only an instrument error; they
are inherently *representative* of the volume of atmosphere sampled. The
representativeness error arises from the forward operator and has the
two components mentioned above. We account for instrument error and, if
we are rigorous, also for the representativeness error, when we specify
the observation error covariance in the penalty function. In practice,
we compute$H(x)$, not $H(x_{t})$.

Note that
$H(x) = H(x_{t}) + H(\varepsilon_{f}) = y_{t} + \varepsilon_{r} + H(\varepsilon_{f})$
for linear *H*. If *H* just involves interpolation, then
$H(x) = y_{t} + \varepsilon_{r} + {\hat{\varepsilon}}_{f}$ where the \^
indicates an interpolated value.

In an OSSE, we are using a forward model to *generate* an observation.
After the forward model is applied to the gridpoint values of the nature
run, we have an "observation" that contains representativeness error
(precisely as defined above) but no instrument error. I think we should
add an appropriate instrument error to this quantity and then account
for both instrument and representativeness errors in the observation
error covariance matrix used during the assimilation.

The finer the resolution of the nature run and the more accurate the
forward model, the smaller the respresentativeness error will be.
Ideally, we should use the most sophisticated forward model available in
generating observations from the nature run, and a different operational
forward model in the assimilation phase of the OSSE.

Tom Schlatter

01 November 2006

**November 14, 2006**

**Michiko Masutani**

Gert-Jan:\
I did not know you are woking on a paper for representativeness error.

**November 14, 2006**

**Gert-Jan Marsellie**

Well, not a peer reviewed paper. But just to have something in hand as a
guideline for the discussion. Erik Andersson also had a look and he
agrees with the content. We are working on a striking figure now,
clearly demonstrating the issue. Once it is in, we\'ll send you the
document.