# Presentation: 458584

## ENSEMBLE PRODUCTS:DATABASE, PRODUCT GENERATION TOOLS, DELIVERY, VERIFICATION

- Zoltan Toth
**Environmental Modeling Center**
**NOAA/NWS/NCEP**
- Acknowledgements: Steve Lord, Ensemble Team
*http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html*

## OUTLINE / SUMMARY

**REQUIREMENTS **
- Critical in driving changes in operational organization like NWS
- Must be reformulated and formalized for ensemble era
**BASIC ENSEMBLE DATABASE**
- Raw operational and hind-cast ensemble member data
- Bias corrected / downscaled member dat
**PRODUCT GENERATION**
- Internal
- Distribute derived products
- External – Decision Support Systems
- Distribute basic dataset
**DISTRIBUTION**
- Passive
- All basic data must be available via ftp
- On demand
- Subsets of basic data
- Derived products
**VERIFICATION**
- User relevant - Bias corrected, downscaled
- Unified system?

## REQUIREMENTS

- How to effect change in operational environment?
- NWS is operational organization
- Operations driven by requirements
- How to change requirements without new supporting infrastructure?
- Chicken and egg dilemma
- New requirements are needed to support change from
- Old traditional paradigm
- Single value forecast to
- New paradigm
- To include forecast uncertainty
- Areas to be considered
- Observing system – Observational uncertainty
- Numerical guidance products – Ensembles
- Centralized (NCEP) and distributed (WFO) forecast products – Design new suit
- Adaptive procedures throughout forecast process, with 2-way feedback loop
- Resource implications
- Computational needs
- Telecommunication needs
- Training needs
- Etc

## ENSEMBLE DATABASE

- Types of ensemble data
- Raw ensemble output – operational
- Hind-cast ensemble for use in bias correction
- Bias corrected ensemble data (centrally corrected)
- Member by member
- Distributional characteristics for selected variables
- Downscaled ensemble data
- Members
- Distributions only
- Requirements (ideally)
- Ensemble member information needed for
- Temporal/spatial covariance and cross-variable forecast information
- Assumption: ensemble has valuable information – needs to be verified
- General user
- Bias corrected / downscaled data for all members, all variables of interest
- Specialized user who wants to do special bias correction
- Raw operational & hind-cast data for all members/variables

## INTERROGATION / PRODUCT GENERATION TOOLS

- User interface
- Users can ask all “meaningful” questions based on ensemble
- Answers delivered in specified format
- Data access
- Utility that returns requested ensemble forecast data
- Output to be used by
- Product generator (internal)
- Decision Support System (external)
- Product generation
- Prepares derived products based on ensemble data
- Probabilities, etc
- 10+ functionalities within NAWIPS
- Applications within NOMADS
- Decision Support Systems
- User specific application / product generator / decision support information

## WHAT PRODUCTS?

- MUST SUPPORT HIERARCHY OF PROGRESSIVELY MORE COMPLEX PRODUCTS
- First moment only – traditional approach
- Best estimate of first moment (univariate mean or median)
- “Most Likely Forecast” (multivariate)
- Close to mean at short lead time
- Display value only in highly non-linear phase (no unique solution)
- Second moment related information added
- Two bounds of distribution (10 & 90 percentile of pdf)
- Likely scenarios (with likelihood), if they exist (based on statistical tests)
- Multiple modes (univariate)
- Clusters (multivariate)
- Display value
- Probability of events that are defined by
- Single variable (univariate)
- Multiple variables (multivariate)
- Quantitative use
- Full information – preferred approach
- Pdf (univariate)
- Ensemble member “trajectories” (multivariate)
- All relevant information in explicit form
- Ideal for downstream user applications

## HOW TO DISSEMINATE PRODUCTS?

- MULTIPLE CHANNELS
- Routinely prepare and actively distribute often-used basic products
- Uni-variate ensemble mean, spread, 10-90 percentiles, PQPF, etc
- AWIPS, NAWIPS, ftp, web
- Stage all raw and bias-corrected ensemble forecasts on ftp sites
- Internally, for product generation engine
- On ftp sites, for professional users
- Include hind-casts to permit user specific bias correction
- Downscaled information will require orders of magnitude more storage
- On-demand access to basic and derived information – web/ftp access
- Partner with public & private sector service providers to serve their data needs
- Use data access software to serve up required ensemble data for
- Ftp data access (eg, NOMADS – NCDC plans)
- User processes information (Decision Support Systems, etc)
- Product generator
- Computation of derived products (on schedule or on demand)

## VERIFICATION

- Purposes
- Assess value added at each step of forecast process
- Assess value of newly developed components or configuration
- Evaluation criteria
- User relevant verification
- What matters to different user groups
- Based on best performance and other constraints
- System that produces best downscaled information at given computational cost
- Infrastructure requirement
-  Unified verification software for ensemble / probabilistic forecasts
- Part of unified general verification framework
- Shared across EMC, NCEP, NWS, NOAA, broader community?
- THORPEX connection

## OUTLINE / SUMMARY

**REQUIREMENTS **
- Critical in driving changes in operational organization like NWS
- Must be reformulated and formalized for ensemble era
**BASIC ENSEMBLE DATABASE**
- Raw operational and hind-cast ensemble member data
- Bias corrected / downscaled member dat
**PRODUCT GENERATION**
- Internal
- Distribute derived products
- External – Decision Support Systems
- Distribute basic dataset
**DISTRIBUTION**
- Passive
- All basic data must be available via ftp
- On demand
- Subsets of basic data
- Derived products
**VERIFICATION**
- User relevant - Bias corrected, downscaled
- Unified system?

## BACKGROUND

## OUTLINE / SUMMARY

**WHY WE NEED ENSEMBLES? **
- Scientifically 	Capture case dependent variations in uncertainty
- Users		Downstream applicatns forced by nonlinear trajectories
**WHEN ARE THEY CRITICAL?**
- Case-dependent variations in chaotic and model error growth
**HOW CAN THEY BE GENERATED? **
- Multiple integrations
- Initial perturbations
- Model perturbations
- Bias correction
- Product generation
**ISSUES IDENTIFIED**
- Often no consensus solution, further research needed to refine issues
- Apparently most promising paths recommended

## WHY ENSEMBLES?

***TRADITIONAL PARADIGM***
- Single value forecast incomplete from viewpoints of
- Science – 		Inherently statistically inconsistent with observations
- Applications – 	Significantly fewer users, with less value
- Probabilistic forecasts needed – Generate them through
- Single forecast integration
- Accumulate error statistics over many cases (“bias correction”, eg, MOS)
- Pro: Maximum possible fidelity in forecast - all comp. resources go into one solution
- Improved statistical reliability; Slight increase in statistical resolution
- Cons:	Aggregate statistics - no case dependent variations in uncertainty captured
- 		As errors become nonlinear, single solution becomes unrepresentative
- Loss of statistical resolution
- Liouville equations
- Theoretically proper solution in perfect model framework
- Pdf of initial state integrated in time
- Impractical, enormous computational costs
- Ensemble forecasts
- Multiple integrations started with sample from estimated initial pdf
- Provides multiple trajectories for critical downstream applications
- Time evolution of pdf captured in truncated form (how many members needed?)
- Ad-hoc methods aimed at capturing model related uncertainty
***	******	******	******	******	******	******	******ENSEMBLE APPROACH***

## ENSEMBLES: WHEN?

- Single forecast approach favored when
- Case-dependent variations are weak in
- Level of linear error growth at short lead times
- Pdf evolution at short lead times (ie, quasi-linear behaviour)
- Model-related error behaviour (at any lead time)
- Aggregate bias-correction algorithms adequate
- Use ensembles otherwise
- review criteria above for each application
- Bias-correct both single value & ensemble forecasts (ie, pdf)
- Decide on forecast configuration based on results
- “Generic” configuration
- Higher resolution control for short lead time if beneficial
- Lower resolution ensemble out to longer lead times
- Benefits from combining hi-re control & lo-res ensemble at shorter leads?
- Considerations
- Integrations must resolve phenomena of interest
- Unless sophisticated statistical down-scaling techniques can be developed

## ENSEMBLES: HOW?

- How to represent initial value related uncertainty?
- Perturb initial conditions
- How to represent model related uncertainty?
- Perturb model integration
- How many sample trajectories needed?
- Ensemble size
- How to convey forecasts?
- Trajectories and derived products
**Unified approach** **across all applications** when practical
- Based on general scientific principles
- Choices based on / supported by experimental results when possible
- Computational feasibility considered
- Facilitated by ESMF framework – common interface for
- Initial & model perturbations, bias correction, product generation
- Applicable in most cases
- Adjusted if/when necessary
- Maintenance economic

## Two distinct problems:
Estimate analysis uncertainty
All estimates ultimately sample based => difficult to disentangle from sampling
Implicit solutions (ie, not explicit pdf)
Choice among sampling strategies, given an estimate
Brute force (“Monte Carlo”) sampling – “Perturbed Observations” method
Run multiple analysis cycles with perturbed observations (Canadian approach)
Both growing and non-growing error space sampled with realistic amplitude
Very poor sampling of myriad non-growing directions
Noise hurts analysis performance
Directed sampling
Singular vectors – fastest growth for pre-selected time period
Transient growth emphasized
Computationally expensive
No general solution: If transient growth is
Important - 	Need different perturbations for various lead times
Not important - 	No need for SVs
Most often norm used is uncoupled from analysis error estimates
Relevant dynamics identified via growth optimization calculations
No successful DA applications
Random sampling in growing sub-space
Based on principle of breeding: Cycle growing perturbations
Capture dynamics of system responsible for error growth
Ignore noise
Successfully used in most ensemble-based DA efforts: eg, ETKF, etc

- Estimate analysis uncertainty
- All estimates ultimately sample based => difficult to disentangle from sampling
- Implicit solutions (ie, not explicit pdf)
- Choice among sampling strategies, given an estimate
- Brute force (“Monte Carlo”) sampling – “Perturbed Observations” method
- Run multiple analysis cycles with perturbed observations (Canadian approach)
- Both growing and non-growing error space sampled with realistic amplitude
- Very poor sampling of myriad non-growing directions
- Noise hurts analysis performance
- Directed sampling
- Singular vectors – fastest growth for pre-selected time period
- Transient growth emphasized
- Computationally expensive
- No general solution: If transient growth is
- Important - 	Need different perturbations for various lead times
- Not important - 	No need for SVs
- Most often norm used is uncoupled from analysis error estimates
- Relevant dynamics identified via growth optimization calculations
- No successful DA applications
- Random sampling in growing sub-space
- Based on principle of breeding: Cycle growing perturbations
- Capture dynamics of system responsible for error growth
- Ignore noise
- Successfully used in most ensemble-based DA efforts: eg, ETKF, etc

**HOW TO REPRESENT INITIAL VALUE RELATED UNCERTRAINTY?**

## HOW TO REPRESENT INITIAL VALUE RELATED UNCERTRAINTY?

**Proposed solution:** Random sampling in growing sub-space – ET / ETKF
- Link with DA
- GSI – ET
- Take error variance from GSI to specify ensemble perturbation level
- Feed back information from ensemble into background error covariance
- Ensemble-based DA – ETKF
- Same ensemble principles, except 2-way interactions tuned simultaneously
- “Ensemble Filter” or
- Variational solutions
- General applicability
- As long as transient behaviour is not dominating
- Ideal for “downstream” applications
- ET provides series of perturbed analyses consistent in time
- Important for wave, land surface, etc ensembles where
- Instantaneous error/perturbation dependent on *forcing history*
- Need for collaboration where DA & ensemble overlap
*Analysis is not complete until uncertainty estimate provided and assessed*
- Consider ensemble-based DA alternative for revised product suite

## Theoretically not well understood problem
Numerical model different from reality
Truncated model representation in resolved
Space and time scales (dynamics, physics)
Processes (physics)
Other approximations (numerics)
Other approximations/errors in representing nature due to
Lack of full understanding of nature
Mistakes (science and coding bugs)
Lack of accounting for model related uncertainties (for ensemble applications)
Deficiency in perturbation growth
Performance metrics for modelling guidance differ depending on use
Traditional, single forecasts
Minimize single forecast error
Ensemble requirement
Account for case dependent model related uncertainty
Systematic effort needed 
Incorporate capability of simulating model related uncertainty
Strong collaboration between modelling and ensemble communities
Model is incomplete until all model related uncertainty can be simulated

- Numerical model different from reality
- Truncated model representation in resolved
- Space and time scales (dynamics, physics)
- Processes (physics)
- Other approximations (numerics)
- Other approximations/errors in representing nature due to
- Lack of full understanding of nature
- Mistakes (science and coding bugs)
- Lack of accounting for model related uncertainties (for ensemble applications)
- Deficiency in perturbation growth
- Performance metrics for modelling guidance differ depending on use
- Traditional, single forecasts
- Minimize single forecast error
- Ensemble requirement
- Account for case dependent model related uncertainty
- Systematic effort needed
- Incorporate capability of simulating model related uncertainty
- Strong collaboration between modelling and ensemble communities
*Model is incomplete until all model related uncertainty can be simulated*

**HOW TO REPRESENT MODEL RELATED UNCERTRAINTY?**

## Current approaches
Stochastic perturbations
“Catch-all” efforts to represent effect of unresolved scales of motion on resolved scales
Increase growth of spread (ie, properly simulate reaql level of predictability)
Multi-model (version) method
Pragmatic effort
Works to minimize effect of unidentified modelling errors
Possibly reduces case dependent biases (that cannot be removed statistically)
High development/maintenance costs
Effort given up by pioneering center (MSC)
Scientifically not appealing
Admits fractured nature of our knowledge
Must transcend
Proposed solution 
Continue development of stochastic perturbation method
Perturb resolved scales within ensemble subspace
Continue use of multi-model approach
Share development/maintenance costs with other centers
NAEFS: MSC, FNMOC, others
SREF / High impact modelling: MSC, FNMOC?
Embrace new modelling paradigm for ensemble era
Enable models to simulate all known modelling related uncertainties
Develop/evaluate models in ensemble (and not single forecast) framework

- Stochastic perturbations
- “Catch-all” efforts to represent effect of unresolved scales of motion on resolved scales
- Increase growth of spread (ie, properly simulate reaql level of predictability)
- Multi-model (version) method
- Pragmatic effort
- Works to minimize effect of unidentified modelling errors
- Possibly reduces case dependent biases (that cannot be removed statistically)
- High development/maintenance costs
- Effort given up by pioneering center (MSC)
- Scientifically not appealing
- Admits fractured nature of our knowledge
- Must transcend
**Proposed solution **
- Continue development of stochastic perturbation method
- Perturb resolved scales within ensemble subspace
- Continue use of multi-model approach
- Share development/maintenance costs with other centers
- NAEFS: MSC, FNMOC, others
- SREF / High impact modelling: MSC, FNMOC?
- Embrace new modelling paradigm for ensemble era
- Enable models to simulate all known modelling related uncertainties
- Develop/evaluate models in ensemble (and not single forecast) framework

**HOW TO REPRESENT MODEL RELATED UNCERTRAINTY?**

## MEMBERSHIP VS. MODEL RESOLUTION?

***TRADE-OFF BETWEEN MODEL VS. PDF RESOLUTION (LIMITED RESOURCES)***
- Step-wise changes from single forecast to ensembles of increasing size: 1 => N
- Decrease in model resolution
- Degrading fidelity / statistical consistency
- Bias correction / downscaling becomes more demanding
- Increase in membership
- Improving statistical resolution (case-dependent variations in pdf captured)
- Potentially better forecasts
- Membership questions
- Fewer members needed in phase of
- Linear error growth (short lead time);
- Bias correction to generate pdf
- More members needed in phase of
- Nonlinear error growth (longer lead times)
- Highly nonlinear phenomena, eg, hurricane genesis
- Bias correction / downscaling to improve fidelity
- To resolve higher moments of pdf

## MEMBERSHIP VS. MODEL RESOLUTION?

*PROPOSED SOLUTION*
- Considerations
- Cannot increase membership with lead time
- Must compromise, considering entire time range of ensemble forecast
- Integrations must resolve phenomena of interest
- Unless sophisticated statistical down-scaling techniques can be developed
- Potential gain from more members capped by level of refinement in other parts of forecast process
- No point refining one aspect of forecast process – skill limited by weakest link
- No use of very large ensemble with poor model
- Bias correction / downscaling can interpolate/extrapolate pdf based on smaller ensemble
- “Generic” configuration guidelines for maximum overall benefits
- Higher resolution control for short lead time if beneficial
- Lower resolution ensemble out to longer lead times
- Benefits from combining hi-re control & lo-res ensemble at shorter leads?
- Ratio of ~1:2 horizontal resolution for ensemble vs. hires control
- O(10) membership

## STATISTICAL POST-PROCESSING

**Distinguish between**
- Bias correction on model grid
- Eliminate lead time dependency
- Coarse to coarse resolution mapping - Cheap
- Downscaling to (much) higher resolution grid (NDFD)
- Needed if effective model resolution is below desired output resolution
- Coarse to fine resolution mapping – Expensive
- Sub-grid variability must be added for ensembles
- Best done by dynamical methods
- LAM or variable resolution global model – Very expensive
**Background**
- Based on sample of forecast – truth pairs
- Model, nature must be stationary
- Quality depends on signal (“systematic error”) to noise (random error) ratio
- Improved when
- Random error smaller (short range)
- Sample size larger (hind-casts important for longer leads with larger random errors)
- Degraded when more details sought
- Need for larger sample

## Approaches
1-step “downscaling”
Potential advantage in reduced noise
2 steps
Lead-time dependent bias correction on model grid (cheap)
Diagnostic evaluation of model forecasts possible
Lead-time independent downscaling to finer grid (more expensive)
Applied on bias corrected forecasts - “Perfect prog” approach, not dependent on lead time
More flexibility
Applications differ in
Statistical method for extracting info from forecast – truth pairs
Linear regression, Bayesian, analog method 
What they bias correct statistically (skilful and not reliable statistically)
Only 1st, or also additional moments?
How rest of information from forecast treated
Retain from raw forecasts (if info statistically reliable)
Replace stochastically (if info not skilful, not reliable)
Analog approach - difficult to control retained vs. lost information
Stochastic generator – preferred solution
Filter out (not skilful)
MOS hedges toward climatology, not suitable for ensemble applications

- 1-step “downscaling”
- Potential advantage in reduced noise
- 2 steps
- Lead-time dependent bias correction on model grid (cheap)
- Diagnostic evaluation of model forecasts possible
- Lead-time independent downscaling to finer grid (more expensive)
- Applied on bias corrected forecasts - “Perfect prog” approach, not dependent on lead time
- More flexibility
- Applications differ in
- Statistical method for extracting info from forecast – truth pairs
- Linear regression, Bayesian, analog method
- What they bias correct statistically (skilful and not reliable statistically)
- Only 1st, or also additional moments?
- How rest of information from forecast treated
- Retain from raw forecasts (if info statistically reliable)
- Replace stochastically (if info not skilful, not reliable)
- Analog approach - difficult to control retained vs. lost information
- Stochastic generator – preferred solution
- Filter out (not skilful)
- MOS hedges toward climatology, not suitable for ensemble applications

**STATISTICAL POST-PROCESSING**

## STATISTICAL POST-PROCESSING – PROPOSED SOLUTION

- PROPOSED SOLUTION: Follow 2-step approach, develop centrally applied
- Bias correction on model grid
- Bayesian approach – can handle all non-Gaussian, non-linear situations
- Can optimally merge hires control, lores ensemble, and climate information
- Bias-corrected ensemble trajectories
- Downscaling to NDFD grid
- Current methods – no sub-grid processes considered/added
- Linear function of grid-scale info – limited utility
- Climate anomaly
- Downscaling vector
- Alternative methods – sub-grid processes considered/added
- More information / larger sample needed – *MDL, other collaborators*
- Local analogs
- Must mosaic together independent patches
- Cannot well control what information is retained vs. stochastically replaced in ensemble
- Stochastic generator
- More general solution
- Difficult to construct?
- Forecast configuration evaluation
- Eventually must be done on user relevant bias-corrected and downscaled information
- Selection criteria
- System that produces best downscaled information at given computational cost

## HIND-CASTING

- What
- With operational process, generate forecasts for past cases
- Must use operational procedures otherwise lost purpose
- Resource intensive
- Purpose
- Increase sample size for bias correction / downscaling
- Required for
- Longer lead bias correction
- Shorter lead bias correction with more details (regime dependent)
- Options
- Freeze operational system
- Generate hind-cast data set prior to use in operations
- Labor intensive
- Any improvements must wait until next hind-cast dataset can be prepared
- Generate hind-casts in real time, on continuous basis
- Can upgrade forecast system any time following a 2-month parallel experiment
- Computationally more expensive – Re-computes hind-casts with new system every year
- Logistically simpler, institutionalized process
- Cheaper in terms of human resources

## HIND-CASTING – PROPOSED SOLUTION

- Consider real-time generation arrangement as part of operations
- Use forecast process for hind-casts identical to operations
- Cannot share hind-casts across applications if operational forecasts are not shared
- Assumption: bias at longer lead does not depend on analy
- Critical for longer range applications
- Global ensemble
- Highly nonlinear applications such as river flow forecasting
- Assume bias at long lead independent of analysis technique (use reanalysis)
- Coupled ensemble
- Needed for refined, regime dependent bias correction / downscaling for
- Regional ensemble, etc
- Short-range bias depends on analysis technique
- Must regenerate reanalysis with current DA system
- Can reanalysis also be done in real time on next machine (2010)?

## REAL-TIME GENERATION OF HIND-CAST DATASET?

- Today’s Julian Date
- TJD

- TJD + 30

- TJD - 30

- Actual ensemble **generated today**

- Hind-casts for TJD+30 **generated today**

- Hind-casts (or its statistics) for TJD+/- 30 **saved on disc**

- Time

- 2005

- 2004

- 2003

- 1980

- 1979

- 2006

## WHAT PRODUCTS?

- HIERARCHY OF PROGRESSIVELY MORE COMPLEX PRODUCTS
- First moment only – traditional approach
- Best estimate of first moment (univariate mean or median)
- “Most Likely Forecast” (multivariate)
- Close to mean at short lead time
- Display value only in highly non-linear phase (no unique solution)
- Second moment related information added
- Two bounds of distribution (10 & 90 percentile of pdf)
- Likely scenarios (with likelihood), if they exist (based on statistical tests)
- Multiple modes (univariate)
- Clusters (multivariate)
- Display value
- Probability of events that are defined by
- Single variable (univariate)
- Multiple variables (multivariate)
- Quantitative use
- Full information – preferred approach
- Pdf (univariate)
- Ensemble member “trajectories” (multivariate)
- All relevant information in explicit form
- Ideal for downstream user applications

## HOW TO DISSEMINATE PRODUCTS?

- MULTIPLE CHANNELS
- Routinely prepare and actively distribute often-used basic products
- Univariate ensemble mean, spread, 10-90 percentiles, PQPF, etc
- AWIPS, NAWIPS, ftp, web
- Stage all raw and bias-corrected ensemble forecasts on ftp sites
- Internally, for product generation engine
- On ftp sites, for professional users
- Include hind-casts to permit user specific bias correction
- Downscaled information will require orders of magnitude more storage
- On-demand access to derived information – web/ftp access
- Use product generation engine
- Accesses bias-corrected ensemble database
- Derives any desired product
- NAWIPS software developed in collaboration with NCO
- NOMADS functionalities serve User Support Systems
- Strongly encouraged by NRC panel report
- Conflict with some private sector partners?

## INTERFACE OF UNIFIED ENSEMBLE APPROACH WITH DIFFERENT FORECAST SYSTEMS

- Experts working on different aspects of unified ensemble approach (columns)
- Others responsible for pulling together all pieces for specific systems (rows)
- Critical area: High-Impact ensemble systems
- Unified framework for very high resolution ensemble, embedded into regional ensemble
- Tropical storm; Severe weather; Storm surge; Fire weather; Air Quality, etc applications
- Must share basic infrastructure to prevent proliferation of systems
- Can adapt basic structure as needed for special applications (eg, different model versions used)
- Suggested long term goal: Variable resolution modeling
- Single framework to address multiscale processes, replaces current global and multitude of LAM integrations
- Simplified modeling, DA, ensemble infrastructure
- Scientifically challanging - 5+ years ( global resolution higher by then)
- Adaptively configure model to serve all high impact cases with pre-defined priorities
- THORPEX research/development resources may be available

## BACKGROUND

**USER REQUIREMENTS:****PROBABILISTIC FORECAST INFORMATION IS CRITICAL**

## Slide 32

## BACKGROUND-2

## EVALUATION OF FORECAST SYSTEMS

- Some statistics based on forecast system only
- Other statistics based on comparison of forecast and observed systems =>
- 		***FORECAST SYSTEM ATTRIBUTES***
- Abstract concepts (like length)
**Reliability and Resolution**
- Both can be measured through different statistics
- Statistical properties
- Interpreted for large set of forecasts (ie, describe behavior of forecast ***system***),
- 							not for a single forecast
- For their definition
- Assume that forecasts:
- Can be of any format
- Take a finite number of different “classes”
- Consider empirical frequency distribution of
- Verifying observations corresponding to large number of forecasts of same class =>
- 			Observed Frequency Distribution (ofd)

## STATISTICAL RELIABILITY – STAT AGGREGATE STATISTICAL CONSISTENCY OF FORECASTS WITH OBSERVATIONS 

**BACKGROUND:**
- Consider particular forecast class – Fa
- Consider distribution of observations *O**a* that follow forecasts Fa
**DEFINITION:**
- If forecast Fa has the exact same form as Oa, for all forecast classes,
- 		the forecast system is statistically consistent with observations =>
- 			*The forecast system is perfectly reliable*
**MEASURES OF RELIABILITY:**
- Based on different ways of comparing Fa and Oa

## STATISTICAL RESOLUTION – TEMPORAL EVOLUTION ABILITY TO DISTINGUISH, AHEAD OF TIME, AMONG DIFFERENT OUTCOMES

**BACKGROUND:**
- Assume observed events are classified into finite number of classes
**DEFINITION:**
- If all observed classes are preceded by distinctly different forecasts, 		the forecasts “resolve” the problem =>
- 			*The forecast system has perfect resolution*
**MEASURES OF RELIABILITY:**
- Based on degree of separation of distributions of observations that follow various forecast classes
- Measured by difference between ofd’s & climate distribution
- Measures differ by how differences between distributions are quantified

*FORECASTS*

*OBSERVATIONS*

**EXAMPLES**

## QUALITY OF BIAS CORRECTION

**REAL WORLD APPLICATIONS***** ***
- More complex problem, processes not stationary but influenced by
- Seasonal cycle
- Bias estimate as function of seasonal cycle
- Can be done but increases sample size requirement
- Regime changes
- Bias estimate as function of regime
- Can be done
- Use of recent data works well for short lead
- Increases sample size requirement for long lead
- Changes in forecast process (periodic upgrades of NWP DA & modeling)
- Estimation can be done but may require regeneration of large hind-cast dataset
- How to balance influence of different factors?
- Update model continuously
- Best statistical resolution
- Lack of large hind-cast dataset corrupts ability to bias correct
- Generate large hind-cast dataset
- Best reliability
- Corrupts statistical resolution for not using best available NWP system
- Can we have the good aspects of both worlds?

## ISSUES

- How much skill we lose by using an 8-10 year old forecast system?
- 2 days skill at D3, 3 days at D7.5, 4+ days at 9D
- Estimate 2% rms error reduction per year in recent yrs
- How much skill we gain due to use of larger sample?
- Reduction depends on level of bias in forecast
- CDC hind-casts have much larger bias
- Large bias reduction observed
- Still does not compensate for loss of skill until 10+ day lead time
- Operational forecast system
- Not known
- Estimate via synthetic data
- Use synthetic data to estimate bias error reduction with more data & current fcst
- What is current level of NH extratropical time mean error (estimated bias)?
- ~6-14% of climate standard deviation (2m temp)
- Gain from expanding sample
- From 100 days (Kalman Filter method) to 5,000 days (50 yrs hind-cast)
- 5-10% (depending on method) of random rms error
- This is equivalent to 2.5-5 yrs of model development
- If DA/model has to be frozen for 3-5 yrs
- Gain in reliability due to larger sample &
- Loss in resolution and reliability due to use of frozen system
- Are of similar magnitude
- Is it worth it?

## Raw, Optimal & Actual Bias Corrected Ensembles  

-    bias correction improves RPSS for all lead time vs. raw oper. ens.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains  significant improvement for all lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-     oper. fcst is better than the bias-corrected reforecast out to 9-10 days.  Beyond 10 days, bias-corrected  reforecast becomes competitive to or better than oper. fcst

-  Sign of improving
-    larger for CDC reforecast

- Annual Mean RPSS ( 20040301 – 20050228 )
- 500 mb Height over Northern Hemisphere

## Gain per year = 

*Gain in 5 yrs = 0.06 or*
*	**	**~15 hrs*

*Gain per year = *
*2% rms error reduction*

*Gain in 5 yrs = 15 hrs or*
*	**6 m (10%)*

## Slide 41

## ISSUES - 2

- Regime dependent vs. climate mean bias correction
- Regime dependent (with small sample)
- Better at short lead
- Climate mean (with much larger sample)
- Better at long lead
- Could try regime dependent correction using hind-cast sample
- Combined method may be best?
- Is it worth the effort to generate hind-cast dataset?
- Questionable
- Are there additional factors/gains to consider?
- Bias correction for highly non-linear downstream applications and for rare events
- River flow forecasting?
- Can we have the good aspects of both worlds?
- Latest model AND large hind-cast dataset?
- Real time generation of hind-cast data
*Assume* biggest problem is bias in first moment
- Can be estimated using large sample of *single* forecasts
- Use recent data only to correct bias in higher moments of ensemble
- Focus on long lead bias estimation problem
- Regime dependent short lead estimation ok with most recent small sample
*Assume* long lead bias depends only on model (and not quality of initial condition)

## Raw, Optimal & Actual Bias Corrected Ensembles  

- CDC reforecast
-  decaying method gives better results than climate mean bias estimation for short range (~day 5), value of regime dependent correction
-  some gain from climate mean bias correction after _5 days

- Raw, Optimal & Actual Bias Corrected Ensembles

- RPSS of 500 mb Height
- Northern Hemisphere, 2004 Summer

## Raw, Optimal & Actual Bias Corrected Ensembles  

- Regime-dependent bias correction superior to climate mean correction (with hindcast data) out to D3
- Climate mean correction superior to regime-dependent method beyond D4

- Raw, Optimal & Actual Bias Corrected Ensembles

- RPSS of 500 mb Height
- Northern Hemisphere, 2004 Summer

## Slide 45

## REAL-TIME GENERATION OF HIND-CAST DATASET?

- When computing facilities are upgraded (2007 at NCEP)
- Forgo increasing resolution/membership
- Instead generate hind-cast “ensemble”
- Arrangement
- Generate control forecast from re-analysis initial conditions
- Initialized from 30 days ahead of current date
- For each year in re-analysis dataset (last 27 yrs)
- Additional cost is 2-3 times current ensemble configuration - *DOABLE*
- Bias correction methodology
- Same as currently, except
- Estimate climate mean bias as mean of difference between
- Re-analysis & hind-cast forecast (+/-30 days x 30 yrs = 1,800 sample)
- Use centrally weighted recursive filter to save on disc storage requirements
- Combine regime dependent Kalman filter + climate mean bias estimates
- More weight on Kalman filter at short
- More weight on climate mean estimate at long leads
- Explore possible regime dependent corrections at long lead
- Save & use entire sample of 1,800 using hind-casts in research
- Discard 30+ day old hind-cast data
- Implement DA/model upgrades at wish
- Generate regime dependent & climate mean bias statistics during
- ~30-day parallel testing prior to operational implementation

## REAL-TIME GENERATION OF HIND-CAST DATASET?

- Today’s Julian Date
- TJD

- TJD + 30

- TJD - 30

- Actual ensemble **generated today**

- Hind-casts for TJD+30 **generated today**

- Hind-casts (or its statistics) for TJD+/- 30 **saved on disc**

- Time

- 2005

- 2004

- 2003

- 1968

- 1967

- 2006

## LIMITATIONS

- Only first moment can be corrected
- Single hind-cast dataset (hindcast ensemble prohibitively expensive)
- Second & higher moments will be corrected using only recent forecast data
- Only longer range forecasts can be corrected
-  Hindcasts initialized with old reanalysis
- Short-range hindcast stat. characteristics different from operational fcsts
- Short-range forecasts will be corrected using only recent forecast data
**TESTS**
- Two alternatives with comparable cpu requirements to be tested:
- Hindcast generation (T126L64)
- Increased horizontal resolution, without hindcasts (T170L64)
- Decision based on verification results
- Each alternative evaluated after bias correction
- Expected result
- Hindcast better for longer ranges
- Increased resolution better for shorter ranges

## OTHER POTENTIAL APPLICATIONS OF REAL-TIME HINDCASTING

- Real-time generation of reanalysis?
- Consistency between reanalysis and operational analysis
- Allows for testing applicability of hindcasts for short-range applications
- Allows for testing utility of hind-cast ensembles
- Test if high cost justified by improved utility of forecasts
- Real-time generation of coupled ocean-atmosphere forecasts?
- Allows periodic upgrades to coupled DA/model

## BACKGROUND

## QUALITY & UTILITY OF FORECASTS

- Quality of forecast process depends on its
- Statistical resolution
- Ability to distinguish (provide unique signals before) future events
- Temporal sequence foreseen - Inherent value of forecast process
- NWP methods used in 6 hr – 15 days range, statistical methods are not viable
- Can be improved by NWP DA & model development
- Statistical reliability
- Ability to simulate (not predict) nature faithfully
- Realism, fidelity - But no info on temporal sequence
- Can be improved by
- NWP model development
- Use of statistical bias correction methods
- Better statistical methods
- Larger data sample - Can be perfectly corrected with large enough sample
- Utility of forecasts depends on both resolution and reliability
- Dual requirement of
- Continuously improved DA & model
- Routinely done couple times per year
- Large hind-cast data sample
- Tried in US once with 8-10 year old system
- May conflict
- What is best way to proceed?

## STATISTICAL POST-PROCESSING ISSUES

**GOAL**
- Improve reliability while maintaining resolution in NWP forecasts
- Reduce systematic errors (improve reliability) while
- Not increasing random errors (maintaining resolution)
- Retain all useful information in NWP forecast
**METHODOLOGY** - Requirements
- Use bias-free estimators of systematic error
- Need methods with fast convergence
**APPROACH** – Computational efficiency
- Remove *lead-time dependent bias* on model grid
- Working on coarser model grid allows use of more complex methods
- Feedback on systematic errors to model development
- Downscale bias-corrected forecast to finer grid
- Further refinement/complexity added
*No dependence on lead time*
- Mapping from coarse to fine grid

## STATISTICAL POST-PROCESSING ISSUES - 2

**APPLICATION** – User requirements
- Statistically correct all forecast information before transmitted to users
- Modest reduction in bias can have large impact on users (prob. fcst)
- Make raw forecast data also available for specialized users
**OUTPUT FORMAT**
- Adjusted ensemble trajectories
- Corrected pdf
**BIAS CORRECTION TECHNIQUES**  – Array of methods
- Frequentist's approach
- Estimate/correct bias moment by moment (e.g., D. Unger et al.)
- Simple approach, implemented partially
- May be less applicable for extreme cases
- Bayesian approach (e.g., Roman Krzysztofovicz)
- Allows simultaneous adjustment of all modes considered
- More complex, currently under development
- BMA, Ridging are variants of approach

## STATISTICAL POST-PROCESSING ISSUES - 3

**DOWNSCALING METHODS**
- Bias correction onto finer grid
- Cumulative distribution matching
- “Downscaling vector”
- Statistically generated spatio-temporal variance on fine grid
**DATA NEEDS** – Hierarchy of possible adjustments
- More data allows better quality / more detailed bias estimation
- Hierarchy of adjustments based on amount of data
- Seasonally changing climate mean
- Regime dependent
- Requires much more data except
- When most recent data used
- Works for short lead time only since regimes change
- Situation dependent
- Use LAM integrations
- Dual-resolution ensemble approach of Jun Du

## Current method, Real data

*Before bias correction*

*After bias correction*

*Estimated*

- Current method, Simulated data

*Estimated*

*Before bias correction*

*After bias correction*

- Current & new methods, Simul. data

*Before bc.*

*After bc, new method*

*Actual*

- New method, Simulated data

*Estimated*

*Before bias correction*

*After bias correction*

*After bc., current method*

##  Bias correction methods  

-  Bias correction methods
- Decaying averaging bias correction ( ~ 46 day independent training data)
- Operational NCEP ensemble
- CDC reforecast data
- 25-yr climatological mean forecast error (Hamill & Whitaker)
- CDC reforecast
- 31-day centered running mean forecast error ( dependent data), used as “optimal” benchmark
- Operational NCEP ensemble
- CDC reforecast data
-  “Hybrid” system, 2 step adjustment

-    FCSTclibriated = FCSTOPR – BIAS Decaying OPR-RFC – BIASRFC-REA (25-YRS)

-  Data sets
-  NCEP operational global ens. Data (2004/05 analysis/modeling system)
-  CDC reforecast data set (1998 modeling system, 1978 – 2005)
- 500 mb height, 850 mb temp, 2m temp, 10m U and V, Mar. 2004-Feb. 2005

## Raw, Optimal & Actual Bias Corrected Ensembles  

-    bias correction improves RPSS for all lead time vs. raw oper. ens.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains  significant improvement for all lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-     oper. fcst is better than the bias-corrected reforecast out to 9-10 days.  Beyond 10 days, bias-corrected  reforecast becomes competitive to or better than oper. fcst

-  Sign of improving
-    larger for CDC reforecast

- Annual Mean RPSS ( 20040301 – 20050228 )
- 500 mb Height over Northern Hemisphere

## Raw, Optimal & Actual Bias Corrected Ensembles  

-  Decaying average
-    oper. ens. with bias correction has better performance than the raw fcst.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPRR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains significant improvement for most lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-    both the raw and post-processed oper. fcst. are better than the bias-corrected reforecast

- RPSS of 850 mb Temperature
- Northern Hemisphere, 2004 Summer

-   RMS error of OPR_DAV2% reduced for first week
-  Climate error removed
-   RMS error of RFC_COR   improvement for all lead  times vs. RFC_RAW

- Raw, Optimal & Actual Bias Corrected Ensembles

## Bias-correction algorithms

**Current method (method 1)**

- : Kalman Filter weight

**New method (method 2)**
- Bias   ~  weighted average of
- Bias Estimation
- Equal weight
- Kalman Filter
- Bias correction

## Slide 61

## Slide 62

## Slide 63