**\[Identical Twin NR\]**

**June 4, 2007**

If OSSEs are conducted with a Nature Run produced by the same model and
resolution used for the DA, this is called the \"Identical Twin
problem\". At UMD this is called \"perfect model OSSEs\". The
presentation was posted at

http://space.hsv.usra.edu/LWG/Feb07/Papers.feb07/Liu.feb07.pdf

The diagram showed at the OSSE meeting and GSI meeting is

http://www.emc.ncep.noaa.gov/research/osse/NR/May07/Lui_Kalnay_3DVAR_LETKF.jpg

They show that LETKF is much better than 3D-Var and that 2% of the data
can produce 90% of the impact if a LETKF-based targeted observation is
used. General agreement is \"in an identical twin the DAS background may
be too close to the truth (closer than B suggests), such that all
observations have too little impact, inclusive the new one.\"

Emmitt said some time ago that OSSE community people have discussed and
agreed that identical twin OSSEs are not valid and we cannot conclude
anything from them. At least a fraternal twin OSSE should be used.
Zoltan said we have to show that identical twin OSSEs are indeed invalid
but no one has done that. He said it should be easy for ECMWF to try and
show if identical twin OSSEs are indeed invalid. That will give the data
impact when the model become perfect. If calibration does not work we
can just adjust the observational error to make it work. Tom Schlatter
said we should not spend any time on invalid OSSEs since the model will
never become perfect.

***[Michiko]{.underline}*** 070528

There must be many unpublished works on OSSEs that use identical twin
OSSEs. Identical twin OSSEs or fraternal twin OSSEs may be useful as
references but we cannot draw conclusions from them. The diagram shows
LETKF behaved as expected but 3D-VAR did not. 3DVar keeps a too large
analysis error with identical twin OSSEs. This may be useful results but
it is dangerous to make any recommendation for an observational system
from this results.

In the presentation they did not mention clearly that identical twin
OSSEs were conducted. Since the results will be affected by the choice
of NR, this should be clearly mentioned.

Note Miyoshi and Sato performed comparison between LETKF and 4DVAR

[[http://www.emc.ncep.noaa.gov/research/osse/NR/Feb07/miyoshi_gsmletkf_revised.pdf]{.underline}](http://www.emc.ncep.noaa.gov/research/osse/NR/Feb07/miyoshi_gsmletkf_revised.pdf)

Zoltan said running analysis is easy job once observation is simulated.
However, simulated obs from T213 NR was left unused for long time due to
the lack of resource to conduct analysis. It was not trivial job to run
analysis with Jack\'s simulated obs. Once radiance get involved, this
get much harder. Many thing have to be set up specially for OSSE.
Evaluation take time as well. So it identical twin OSSE to be done at
ECMWF, knowing we cannot use the results for recommendation, we really
have to think carefully.

I am interested in fraternal twin OSSEs. I do not expect there is
realistic MJO or QBO in NR.  I expect NR and DA model are doing their
own tropical or stratospheric oscillation..   As Oreste
presented synoptic scale waves look quite realistic in T511 NR.  
Fraternal twin OSSE  may be useful to evaluate how much model error is
affecting the OSSE results and predictability. However, these OSSEs
should be only presented with results from full OSSEs.

***[Ad Stoffelen]{.underline}***070529

I agree that they can certainly not be regarded as measures of absolute
impact; only full OSSE can, but after calibration. On the other hand, it
is useful to scientifically document academic studies such as identical
twin studies. In this light I made some comments to the book chapter on
this topic, since the motivation not to do identical twin experiments
was rather dogmatic in my view. I would expect that indeed few
observations are required in an identical twin OSSE to analyse close to
the truth. Adding a new system to the GOS (Global Observing System)
would then not deliver much additional value. However, while I would
think they may underestimate impact, it is often motivated that
identical twin OSSE overestimate new OS? I would encourage ECMWF running
an identical twin experiment to show the effects on O-B (Observation
minus Back ground) and O-A (Observation minus Analysis) statistical
distributions, such that we increase our understanding of what effects
an identical twin leads to while forcing a too truthful model trajectory
with observations. Perhaps a full GOS and a much reduced GOS experiment
could show the identical twin effect I mention above? I\'m not aware of
anybody having done this type of experiment.

***[Erik Andersson]{.underline}***070529

Firstly, I think it is worth including in the minutes the actual
definitions of these terms, so we all talk about the same thing. Use the
definitions stated in Arnold and Dey (1986). I think all the OSSEs we
are planning so far will fall in their \'fraternal twin\' category !\
\
Like Tom Schlatter I think that we should concentrate on conducting
OSSEs that are as realistic as possible. Especially if resources are
limited. And not spend too much time or degraded scenarios - unless
there is a special interest in some specific issue.\
\
You can add to the minutes that I said that ECMWF would be willing to
conduct some perfect-model scenario OSSEs if there is a specific goal
with such a study (Zoltan\'s suggestion). E.g. to have as a reference
against which the real OSSE results should be contrasted.  Having said
that, I don\'t quite know what kind of comparison could be made in
practice. The OSSE NWP system would differ from the ECMWF system in so
many ways, not just in its use of a different NWP model
(perfect/non-perfect scenario respectively). Also the DA system itself
would be different. So there would be too many contributing factors that
would be hard to disentangle from the results. Any idea for something
concrete and more straight forward?\
Another crucial aspect in identical twin experiments is to specify a
perturbed state to start from. Presumably the results would depend on
the choice of this initial state perturbation.\
\
In real NWP there are three main source for DA error: initial condition
error, model error and observation error (incuding represntativeness
error). All of them contribute to the observed obs-background
departures. In perfect model OSSE the model error source of noise is
absent. And the initial condition error (for the first DA cycle) has to
be defined somehow. The observation error is simulated similarly in both
types of OSSE. The criticism of identical-twin OSSE should emanate from
both of these simplifications (initial condition error and model error).
After some time of DA-cycling the choice of initial condition
perturbation might not matter, once the system has asymptoted. At that
stage it is the neglect of model error that is problematic. As Ad
pointed out, this will always tend to underestimate the observation
impact. More generally, it makes OSSE calibration more difficult,
because one source of error is absent. It would not be correct to simply
transfer that source of noise to the observation error or
representativeness error terms. By the way - identical twin OSSES tend
to be presented without any calibration results ! In such cases they
should be treated as \'for illustration only\'. Not necessarily
transferable to the full NWP environment.

***[Erik Andersson]{.underline}*** 070529

Comments on Meeting summary

\'Zoltan said we have to show that identical twin OSSEs are indeed
invalid\...\' I would like to rephrase this to say: Those who choose to
do an identical twin study have to show that that simplified approach is
valid for their purpose. By choosing to do the more complete OSSE study
we have avoided having to do that. What we have to do is to show that
the OSSE is well calibrated - and that should be quite a lot more
convincing.

\'That will give the data impact when the model has become perfect\' -
and we could add: and there are no atmospheric features and no sources
of forecast error (!) at unresolved scales. This is another frequently
neglected discussion point in simplified OSSEs.

\'If calibration does not work we can just adjust the observation error
to make it work\' NO this is not correct. Observation error is a source
term at observation locations. Model error is a source term everywhere,
and especially where some physics is going on. One error source cannot
be replaced by the other through any simple tuning procedure.

***[Ron Errico]{.underline}*** 070531

I would like to see an identical twin OSSE performed, in order to learn
somehow about how important the effects of model error are. But, it is
not that simple, as Erik makes clear.  For example, do you retune your B
for the likely reduced forecast error?  Then you have 2 changes: a
different B and a different model.\
\
The main point is, from well designed exps. you can learn some things.\
You want well-posed questions and exps designed to actually answer the
question, rather than a different question. Then you want results to be
carefully interpreted.  The problem with most exps. is that (1) they
actually have been intentionally or erroneously designed to answer a
different question than the one originally posed, and their results are
then misinterpreted and misrepresented.

***[Jeff Whitaker]{.underline}*** 070601

Michiko:  Thank you for this information.  I look forward to discussing
DAS strategies at the workshop.  I think your point about
\'identical-twin\' OSSEs is valid - much of the differences between
3DVar, the EnKF and 4DVar disappear when unrepresented model errors
become large.  I do think that in general one would expect the EnKF and
4DVar to extract more information from fewer observations than 3DVar, so
that in an OSSE environment new data may have a lesser impact (since the
new obs are more likely to be redundant in the  more advanced DA
systems).\
\
Regards,\
\
-Jeff

***[Takemasa Miyoshi]{.underline}*** 070604

\> Yamaguchi-san of JMA agree to look after 250GB low resolution data
from\
\> Nature run.   I hope you will be able to have some fun work using
that\
\> data.\
\
It is great that Yamaguchi-san takes an action to get the data.\
I think I mentioned before that all I need for my OSSEs are simulated
obs and verification fields (e.g., 1.25 deg x 1.25 deg pressure-level
NR), not the original or low-resolution NR data. Specifically, it is
crucial to have the simulated obs dataset, which I guess is not included
in the 250GB data.\
\
Since obs dataset may be rather small (if we exclude satellite obs), you
could distribute them through the internet. Then, I may be able to
perform OSSEs using the simulated obs. At least 1.5-month obs data are
preferred for OSSEs.\
\
\> Erik seems to be ready to perform Identical Twin OSSEs at ECMWF. I
think\
\> he will use just ECMWF 4DVAR.   Since you can compare 4DVAR, 3Dvar
and\
\> LETKF, it will be more interesting if you can work together.\
\
This sounds great to me. If all use the same simulated obs, we can make
inter-comparison among different systems at different centers including
test versions such as LETKF now under development. However, it is
important to consider that comparison with ECMWF system is not fair
since the NR is produced by the ECMWF model.\
\
\
\> I wonder if you have a manuscript they are trying to submit for\
\> publication. She sent me a copy and my main comments was\
\> \"They have to make it very clear that they did Identical Twin OSSE
and\
\> discuss how various errors are effected their results.\"\
\
I do not follow what Junjie and Eugenia have been doing recently. I do
not know about their paper either. Sorry that I cannot help.\
\
I find the discussion on OSSEs very interesting. I have a few comments
on some of the discussion.\
\
1.\
General agreement is \"in an identical twin the DAS background may be
too close to the truth (closer than B suggests), such that all
observations have too little impact, inclusive the new one.\"\
\
I do not agree. In the EnKF, B is always underestimated even in an
identical twin exp, thus we need the error covariance inflation. We
could tune the inflation parameter to have an appropriate B.\
\
2.\
Zoltan said we have to show that identical twin OSSEs are indeed invalid
but no one has done that.\
\
I agree. If someone says the identical twin OSSEs are not valid, s/he
must prove it.\
\
\
I agree that identical twin OSSEs are too optimistic and not simulating
the real situation. However, it enables clear discussion and simulates
the perfect case. If we fail the identical twin OSSEs, there would be no
chance that we succeed in the real case. I think identical twin OSSEs
are valuable in many senses, even if it is too optimistic and has
limitations. I agree that we have to be cautious about the limitations,
but we cannot say that identical OSSEs are meaningless.\
\
I think OSSEs are useful to test verification methods. For example, we
usually use anomaly correl score, RMS errors against radiosondes, RMS
errors against own initial conditions. All of the verification scores
have limitations and it is not clear how they represent actual forecast
accuracy. In the OSSEs, since we have the NR field, we could test how
forecast errors appear in each verification score, which can be compared
to the true RMS errors against the NR.\
\
These are my interests on the OSSEs that I think up right now.\
\
Best wishes,\
\
Takemasa