# Presentation: 178685

## ASSESSING FORECAST UNCERTAINTY

- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- Louis Uccellini, Steve Lord & Ensemble Team
- http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

## OUTLINE / SUMMARY

- TRADITIONAL NWP APPROACH
- REDUCE FORECAST UNCERTAINTY
- IGNORE REMAINING ERRORS
- Problem for users
- SOURCES OF FORECAST ERRORS
- INITIAL CONDITION
- NUMERICAL MODEL
- ESTIMATING AND SAMPLING FORECAST ERRORS
- INITIAL CONDITION
- Breeding technique / ET
- MODEL ERRORS
- No solid scientific basis, open research
- POTENTIAL VALUE OF ENSEMBLE APPROACH
- IMPROVED SINGLE VALUE ESTIMATE
- CASE DEPENDENT ESTIMATE OF UNCERTAINTY
- FULL PROBABILITY DISTRIBUTION / TRAJECTORIES

## Assess current weather situation
Before we can look into future, understand what is happening now
“Initial condition”

Digest observational information
Bring observed data into “standard” format
“Data assimilation”

Project initial state into future
Based on laws of physics
“Numerical Weather Prediction” (NWP) model forecasting

Apply weather forecast information
Statistical post-processing
“User applications”

FORECASTS ARE NOT PERFECT – WHY?

- Before we can look into future, understand what is happening now
*“**Initial condition”*
- Digest observational information
- Bring observed data into “standard” format
*“**Data assimilation”*
- Project initial state into future
- Based on laws of physics
*“**Numerical Weather Prediction”* (NWP) model forecasting
- Apply weather forecast information
- Statistical post-processing
*“**User applications”*
***FORECASTS ARE NOT PERFECT – WHY?***

**THE MAKINGS OF A WEATHER FORECAST –**
*HOW FORECASTS ARE MADE? *

## SOURCES OF FORECAST ERRORS

***IMPERFECT KNOWLEDGE OF***
**INITIAL CONDITIONS**
-  Incomplete observing system (not all variables observed)
-  Inaccurate observations (instrument/representativeness error)
-  Imperfect data assimilation methods
-  Statistical approximations (eg, inaccurate error covariance information)
-  Use of imperfect NWP forecasts (due to initial and model errors) –
-  Effect of cycling (forecast errors “inherited” by analysis – use breeding)
**GOVERNING EQUATIONS**:
-  Imperfect model
-  Structural uncertainty (eg, choice of structure of convective scheme)
-  Parametric uncertainty (eg, critical values in parameterization schemes)
-  Closure/truncation errors (temporal/spatial resolution; spatial coverage, etc)
***NOTES****:*
-  Two main sources of forecast errors hard to separate =>
-  Very little information is available on model related errors
-  Tendency to attribute all forecast errors to model problems
***CAN REDUCE, BUT NEVER ELIMINATE ERRORS***

## EVER IMPROVING, BUT ALWAYS IMPERFECT – WHY?

**EVER IMPROVING, BUT ALWAYS IMPERFECT – WHY?**

## WHY ERRORS AMPLIFY?

*Gain in 5 yrs = 0.06 or*
*	**	**~15 hrs*

*Gain per year = *
*2% rms error reduction*

*Gain in 5 yrs = 15 hrs or*
*	**6 m (10%)*

**WHY ERRORS AMPLIFY?**

## SCIENTIFIC NEEDS - DESCRIBE FORECAST UNCERTAINTY ARISING DUE TO CHAOS

*Buizza 2002*

## VALUE OF PROBABILISTIC FORECASTING

- “...the value of reliable – and even moderately unreliable – probabilistic forecasts generally exceeds the value of ... categorical forecasts” - Murphy 1977
- Potential economic value of ensemble forecasts
- “... a winder range of potential users can benefit from the ensemble than from the control forecasts ... the ensemble offers more economic value than the control forecasts” – Zhu el al. 2002
- Operational forecasting implications
- “...important implications for operational forecasting ... desirability of formulating and disseminating a wide variety of weather forecasts in probabilistic terms...” Murphy 1977
- “A weather forecast is ... not complete unless it is expressed in the form of probability distributions.” - Zhu el al. 2002
- “Uncertainty is thus a fundamental characteristic of weather, climate, and hydrological prediction, and no forecast is complete without a description of its uncertainty.” NRC Report: “Completing the Forecast”, Ban et al., 2006

**VALUE OF PROBABILISTIC FORECASTING**

**USER REQUIREMENTS:****PROBABILISTIC FORECAST INFORMATION IS CRITICAL**

## ASSESSING FORECAST UNCERTAINTY

- Initial condition, model not perfect
- Errors can be reduced, but never eliminated
- Main (only) NWP thrust so far: reduction of uncertainty
- Atmosphere is chaotic system
- Any error amplifies
- Predictability is finite and
- Varies from case to case
- Users need to know about expected forecast errors
- Serious limitation otherwise
- Errors can be assessed
- Statistically
- Climatology of errors in single forecast
- Dynamically
- Ensemble forecasts
***New thrust in NWP is assessing uncertainty***

**ASSESSING FORECAST UNCERTAINTY**

## MOTIVATION FOR ENSEMBLE FORECASTING

***FORECASTS ARE NOT PERFECT - *****IMPLICATIONS FOR:**
**USERS:**
- Need to know how often / by how much forecasts fail
- Economically optimal behavior depends on
- Forecast error characteristics
- User specific application
- Cost of weather related adaptive action
- Expected loss if no action taken
- EXAMPLE: Protect or not your crop against possible frost
- Cost = 10k, Potential Loss = 100k => Will protect if P(frost) > Cost/Loss=0.1
***NEED FOR PROBABILISTIC FORECAST INFORMATION***
**DEVELOPERS:**
- Need to improve performance   - 	*Reduce error in estimate of first moment*
- Traditional NWP activities (I.e., model, data assimilation development)
- Need to account for uncertainty -	*Estimate higher moments*
- New aspect – How to do this?
-  Forecast is incomplete without information on forecast uncertainty
***NEED TO USE PROBABILISTIC FORECAST FORMAT***
***FORECASTS ARE NOT COMPLETE UNLESS UNCERTAINTY ASSESSED ***

## FORECASTING IN A CHAOTIC ENVIRONMENT

***DETERMINISTIC APPROACH - PROBABILISTIC FORMAT***
*** ***
**SINGLE FORECAST** -	* ****One integration with an NWP model***
-  Is not best estimate for future evolution of system
- Except if constrained by data in 4DVAR
-  Does not contain all attainable forecast information
- Case-dependent variations in forecast uncertainty missed
- 4DVAR does not come with an ensemble generation algorithm
-  Can be combined with past verification statistics to form probabilistic forecast
-  Gives **no estimate of flow dependent variations in forecast uncertainty**
**PROBABILISTIC FORECASTING** -	 ***Based on Liuville Equations***
-  Continuity equation for probabilities, given dynamical eqs. of motion
- Dynamical forecast of pdf based on conservation of probability values
- Initialize with probability distribution function (pdf) at analysis time
-  **Prohibitively expensive** -
-  Very high dimensional problem (state space x probability space)
-  Separate integration for each lead time
-  Closure problems when simplified solution sought

## FORECASTING IN A CHAOTIC ENVIRONMENT –

**PROBABILISTIC FORECASTING BASED A ON SINGLE FORECAST** –
***One integration with an NWP model, *****combined with past verification statistics**
***DETERMINISTIC APPROACH - PROBABILISTIC FORMAT***

- Does not contain all forecast information
- Not best estimate for future evolution of system
***UNCERTAINTY CAPTURED IN TIME AVERAGE SENSE -***
**NO ESTIMATE OF CASE DEPENDENT VARIATIONS IN FCST UNCERTAINTY**

## SCIENTIFIC NEEDS - DESCRIBE FORECAST UNCERTAINTY ARISING DUE TO CHAOS

*Buizza 2002*

## WHY ENSEMBLES?

***TRADITIONAL PARADIGM***
- Single value forecast incomplete from viewpoints of
- Science – 		Inherently statistically inconsistent with observations
- Applications – 	Significantly fewer users, with less value
- Probabilistic forecasts needed – Generate them through
- Single forecast integration
- Accumulate error statistics over many cases (“bias correction”, eg, MOS)
- Pro: Maximum possible fidelity in forecast - all comp. resources go into one solution
- Improved statistical reliability; Slight increase in statistical resolution
- Cons:	Aggregate statistics - no case dependent variations in uncertainty captured
- 		As errors become nonlinear, single solution becomes unrepresentative
- Loss of statistical resolution
- Liouville equations
- Theoretically proper solution in perfect model framework
- Pdf of initial state integrated in time
- Impractical, enormous computational costs
- Ensemble forecasts
- Multiple integrations started with sample from estimated initial pdf
- Provides multiple trajectories for critical downstream applications
- Time evolution of pdf captured in truncated form (how many members needed?)
- Ad-hoc methods aimed at capturing model related uncertainty
***	******	******	******	******	******	******	******ENSEMBLE APPROACH***

## PROPOSED CHANGE

- Incorporate assessment and communication of uncertainty in forecast process
- Is it a **major change in course** of “Weather Ship”?
- Ie, abandon course of ever improving single forecast scenario (expected value)?
- No – Expand, not abandon
- Keep improving fidelity of forecasts, PLUS
- Add new dimension
- Capture other possible scenarios – ensemble forecasting
- Use a flotilla, instead of one ship, in exploring nature
- Existing activities are subset of expanded forecast process
- Single value forecast is expected value of full probability distribution
- Can keep serving forecasts in old format to users who prefer that

**PROPOSED CHANGE**

## Single forecast (driven by GFS winds) example for drifting virtual ice floe

- Single forecast (driven by GFS winds) example for drifting virtual ice floe

*Bob Grumbine, EMC*

- Initial position

- 7 September 2006

## Ensemble forecast for drifting ice floe for same case

**Ensemble forecast** for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## Most likely forecast for drifting ice floe for same case

- Most likely forecast for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## WHY CHANGE IS NEEDED?

- They admittedly want minimal or no uncertainty in forecasts
- Distinction between no uncertainty in the forecast, vs. not talking about it
- Forecast uncertainty cannot be arbitrarily reduced
- Despite major ongoing & continuing efforts, they persist forever
- Chaotic nature of atmosphere - land surface – ocean coupled system + initial/model errors
- Level of uncertainty is determined by nature and level of sophistication in forecast system
- Forecast uncertainty can be ignored though
- Negative consequence on informed users
- Not able to prepare for all possible outcomes
- Assumes a certain scenario and remains vulnerable to others
- Possibly serious loss in social/economic value of forecast information
- Why forecasters (should) care about forecast uncertainty?
- Imperfect forecasts are consistent w. observations (reliable) only if in prob format
- If in other format, must be brought into probabilistic format through
- Verification / bias correction

**WHY CHANGE IS NEEDED?**

## ADVANTAGES OF PROBABILISTIC FORMAT

**Old paradigm**
- Convoluted forecaster-user decision process
- User expects forecaster to make decision for them in presence of uncertainty
- “Will it rain?” – “80%” – “But tell me, will it rain?”
**New paradigm**
- Forecaster and user decision processes enhanced and better linked
- Allows forecasters to capture all knowledge about future conditions
- Provision of information related to multiple decision levels in probabilistic format critical
- Provider helps interpret probabilistic info & and modify user decision process if needed
- Option to continue providing single value or other limited info until user ready
- Allows users to decide about most beneficial course of action given all possibilities
- Proper use of probability or other uncertainty information needed - Training
- User requests critical weather forecast info depending on their sensitivity

**ADVANTAGES OF PROBABILISTIC FORMAT**

## TRADITIONAL FORECAST PROCESS

- Reducing uncertainty in single forecast is main emphasis
- Loss of accuracy in forecast estimate of expected value of distribution
- Mean of ensemble cloud provides better estimate
- Ignores or simplifies forecast uncertainty
- Uncertainty assessed as statistically averaged error in single fcst (second thought)
- Ensemble cloud provides better estimate of case dependent variations in uncertainty
- Use of single value / categorical forecast format
- Difficulty in formulating/communicating plausible alternate scenarios
- Ensemble member forecasts can directly feed into Decision Support Systems
- One-way flow of information from observations to users
- Not adaptable to case dependent user requirements
- Ensemble can propagate back user requirements to adaptive
- Observing, assimilation, modeling/ensemble, post-processing and application components
- Applications in planning and execution of new CONOPS in high impact events

**TRADITIONAL FORECAST PROCESS**

## PROPAGATING FORECAST UNCERTAINTY

- z

*Single value*

*Distribution*

- Ensemble Forecasting:
- Central role – bringing the pieces together

**PROPAGATING FORECAST UNCERTAINTY**

## HOW CAN IT BE DONE? NEW PARADIGM

- Expand forecasting with new dimension of uncertainty
- Multiple scenarios (in place of single scenario)
- Provides best forecast estimate for both expected value (as before) and uncertainty (new)
- Unified scientific, technological, human approach
- Sharing resources across NWS & NOAA
- Ensemble is centerpiece both symbolically and figuratively in forecast process
- Ensembles act as a glue & two-way information channel
- Observing system, data assimilation, numerical modeling
- ENSEMBLES
- Statistical post-processing, product generation, decision making
- Design, develop, & implement missing components of new forecast process
- Gradual, measured steps
- Basic capability - Short-term, 2-3 yrs, leading to
- Full implementation - Long-term, 5-10 yrs

**HOW CAN IT BE DONE? NEW PARADIGM**

## FORECASTING IN A CHAOTIC ENVIRONMENT - 2DETERMINISTIC APPROACH - PROBABILISTIC FORMAT

**MONTE CARLO APPROACH** –* ****ENSEMBLE FORECASTING***
* ****IDEA*****:****	****	****Sample sources of forecast error**
- Generate initial ensemble perturbations
- Represent model related uncertainty
* ****PRACTICE***:	**Run multiple NWP model integrations**
- Advantage of perfect parallelization
- Use lower spatial resolution if short on resources
-  ***USAGE*****:**		**Construct forecast pdf based on finite sample**
- Ready to be used in real world applications
- Verification of forecasts
- Statistical post-processing (remove bias in 1st, 2nd, higher moments)
***CAPTURES FLOW DEPENDENT VARIATIONS***
***	******	******	******	******	******	****** ******IN FORECAST UNCERTAINTY***

## 6 hours breeding cycle

- T00Z
- 56m

- T06Z
- 56m

- T18Z
- 56m

- T12Z
- 56m

- 6hrs

- Next T00Z

**Re-scaling**

**Re-scaling**

**Re-scaling**

**Up to 16-d**

**Up to 16-d**

**Up to 16-d**

**Up to 16-d**

- T00Z
- 80m

- 6hrs

- T06Z
- 80m

- T12Z
- 80m

- T18Z
- 80m

**Up to 16-d**

**Up to 16-d**

**Up to 16-d**

**Up to 16-d**

**Re-scaling**

**Re-scaling**

**Re-scaling**

**Re-scaling**

-                  Production

-               Planned Change

- Next T00Z

**Re-scaling**

## SAMPLING FORECAST ERRORS =

***REPRESENTING ERRORS ORIGINATING FROM TWO MAIN SOURCES***
**INITIAL CONDITION RELATED ERRORS**** – *****“Easy”***
** **Sample initial errors
-  Run ensemble of forecasts
-  It works
-  Flow dependent variations in forecast uncertainty captured (show later)
-  Difficult or impossible to reproduce with statistical methods
**MODEL RELATED ERRORS**** – *****No theoretically satisfying approach***
** **Change structure of model (eg, use different convective schemes, etc, MSC)
-  Add stochastic noise (eg, perturb diabatic forcing, ECMWF)
-  Works? Advantages of various approaches need to be carefully assessed
-  Are flow dependent variations in uncertainty captured?
-  Can statistical post-processing replicate use of various methods?
-  Need for a
-  more comprehensive and
-  theoretically appealing approach

## SAMPLING INITIAL CONDITION ERRORS

***CAN SAMPLE ONLY WHAT’S KNOWN – FIRST NEED TO***
***ESTIMATE INITIAL ERROR DISTRIBUTION***
***THEORETICAL UNDERSTANDING*** – THE MORE ADVANCED A SCHEME IS
- 					(e. g., 4DVAR, Ensemble Kalman Filter)
-  The lower the overall error level is
-  The more the error is concentrated in subspace of Lyapunov/Bred vectors
***PRACTICAL APPROACHES***** – **
**ONLY SOLUTION IS MONTE CARLO (ENSEMBLE) SIMULATION**
-  ***Statistical approach*** (dynamically growing errors neglected)
-  Selected estimated statistical properties of analysis error reproduced
-  Baumhefner et al – Spatial distribution; wavenumber spectra
-  ECMWF – Implicite constraint with use of Total Energy norm
*** ******Dynamical approach*** – Breeding cycle (NCEP)
-  Cycling of errors captured
-  Estimates subspace of dynamically fastest growing errors in analysis
-  ***Stochastic-dynamic approach*** – Perturbed Observations method (MSC)
-  Perturb all observations (given their uncertainty)
-  Run multiple analysis cycles
-  Captures full space (growing + non-growing) of analysis errors

## SAMPLING INITIAL CONDITION ERRORS

***THREE APPROACHES – SEVERAL OPEN QUESTIONS***
**RANDOM SAMPLING** – **Perturbed observations method** (MSC)
- Represents all potential error patterns with realistic amplitude
- Small subspace of growing errors is well represented
- Potential problems:
-  Much larger subspace of non-growing errors poorly sampled,
-  Yet represented with realistic amplitudes
**SAMPLE GROWING *****ANALYSIS***** ERRORS** – **Breeding** (NCEP)
- Represents dynamically growing analysis errors
- Ignores non-growing component of error
- Potential problems:
- May not provide “wide enough” sample of growing perturbations
- Statistical consistency violated due to directed sampling? Forecast consequences?
**SAMPLE FASTEST GROWING *****FORECAST***** ERRORS** – **SVs** (ECMWF)
- Represents forecast errors that would grow fastest in linear sense
- Perturbations are optimized for maximum forecast error growth
- Potential problems:
- Need to optimize for each forecast application (or for none)?
- Linear approximation used
- Very expensive

## ESTIMATING AND SAMPLING INITIAL ERRORS:THE BREEDING METHOD

**DATA ASSIM:** Growing errors due to cycling through NWP forecasts
**BREEDING:** - Simulate effect of obs by rescaling nonlinear perturbations
- Sample subspace of most rapidly growing analysis errors
- Extension of linear concept of Lyapunov Vectors into nonlinear environment
- Fastest growing nonlinear perturbations
- Not optimized for future growth –
- Norm independent
- Is non-modal behavior important?

## LYAPUNOV, SINGULAR, AND BRED VECTORS

**LYAPUNOV VECTORS (LLV):**
- Linear perturbation evolution
- Fast growth
- Sustainable
- Norm independent
- Spectrum of LLVs
**SINGULAR VECTORS (SV):**
- Linear perturbation evolution
- Fastest growth
- Transitional (optimized)
- Norm dependent
- Spectrum of SVs
**BRED VECTORS (BV):**
- Nonlinear perturbation evolution
- Fast growth
- Sustainable
- Norm independent
- Can orthogonalize (Boffeta et al)

## PERTURBATION EVOLUTION

**PERTURBATION GROWTH**
- Due to effect of instabilities
- Linked with atmospheric phenomena (e.g, frontal system)
**LIFE CYCLE OF PERTURBATIONS**
- Associated with phenomena
- Nonlinear interactions limit perturbation growth
- Eg, convective instabilities grow fast but are limited by availability of moisture etc
**LINEAR DESCRIPTION**
- May be valid at beginning stage only
- If linear models used, need to reflect nonlinear effects at given perturb. Amplitude
**BREEDING**
- Full nonlinear description
- Range of typical perturbation amplitudes is only free parameter

## 
Estimate analysis uncertainty
Choices among sampling strategies, given an estimate
Monte Carlo type sampling – “Perturbed Observations” method
Run multiple analysis cycles with perturbed observations (Canadian approach).
Both growing and non-growing error space sampled with realistic amplitude.
Noise introduced hurts analysis performance.

Directed sampling
Singular vectors – fastest growth for pre-selected time period (ECMWF)
Transient growth emphasized.
Computationally very expensive.
No general solution: depending time interval and norm.
Norm most frequently used is uncoupled from analysis error estimates.
No success in DA applications.

Dynamical sampling in growing sub-space (NCEP)
Based on principle of breeding: Cycle growing perturbations
Capture dynamics of system responsible for error growth.
Ignore noise.
Successfully used in most ensemble-based DA efforts: eg, ETKF, etc.

- Estimate analysis uncertainty
- Choices among sampling strategies, given an estimate
- Monte Carlo type sampling – “Perturbed Observations” method
- Run multiple analysis cycles with perturbed observations (Canadian approach).
- Both growing and non-growing error space sampled with realistic amplitude.
- Noise introduced hurts analysis performance.
- Directed sampling
- Singular vectors – fastest growth for pre-selected time period (ECMWF)
- Transient growth emphasized.
- Computationally very expensive.
- No general solution: depending time interval and norm.
- Norm most frequently used is uncoupled from analysis error estimates.
- No success in DA applications.
- Dynamical sampling in growing sub-space (NCEP)
- Based on principle of breeding: Cycle growing perturbations
- Capture dynamics of system responsible for error growth.
- Ignore noise.
- Successfully used in most ensemble-based DA efforts: eg, ETKF, etc.

**HOW TO REPRESENT INITIAL VALUE RELATED UNCERTRAINTY?**** **

## HOW TO REPRESENT INITIAL VALUE RELATED UNCERTRAINTY?

**Proposed solution:** Dynamical sampling in growing sub-space – ET / ETKF
- Link with DA (GSI – ET)
- Need collaboration between DA and ensemble teams.
- Take error variance from GSI to specify ensemble perturbations
- Feed back information from ensemble into background error covariance.
- ET provides series of perturbed analyses consistent in time
- Important for wave, land surface ensembles etc  where perts depends on the history.
- Ensemble-based DA – ETKF
- Same ensemble principles, except 2-way interactions tuned simultaneously.

## Ensemble Transform Bred Vector

- P1

- N1

- P2

- N2

- P#, N# are the pairs of positive and negative
- P1 and P2 are independent vectors
- Simple scaling down (no direction change)

- _Ensemble Transform Bred Vector_
- (New)

- P1, P2, P3, P4 are orthogonal vectors
- No pairs any more
- To centralize all perturbed vectors (sum of all vectors are equal to zero)
- Scaling down by applying mask,
- The direction of vectors will be tuned by ET.

- _Rescaling_

**ANL**

**ANL**

**ANL**

- _Bred Vector_
- (Current)

_**P1 forecast**_

_**P4 forecast**_

_**P3 forecast**_

_**P2 forecast**_

- t=t1

- t=t0

- t=t0

- t=t2

- t=t2

- t=t1

- _Rescaling_

## Unified EFS and DA

*   **EFS and DA systems must be consistent for best *
*      **performance of both.** *
*  *SSI/GSI currently* ** provides ** **best estimate of analysis, GSI will be *
*       **used to derive ** **analysis uncertainties (error variance) for EFS.    *
*  **EFS produces flow dependent forecast (background) error *
*       **covariance to be tested in GSI later.          *

- GSI

- EFS

- Best analysis error variances

- Accurate forecast error covariance

- A Hybrid DA-EFS System

- or

## 4.   Summary of Perturbation Properties

- (a).  Perts are centered around the analysis to improve ensemble mean.
- (b). They have simplex structure, not paired. Ensures that perts  will have maximum number of  effective degrees of freedom. The variance will
-        be maintained in as many directions as possible  within the ensemble subspace.
- (c).  They are uniformly centered and distributed in different directions. The larger the ensemble, the more orthogonal they become.  They become orthogonal if the number of  members approaches to infinity.
- (d). The initial perts have flow dependent spatial structure if the analysis error variance is derived from operational DA system at every cycle.
- (e).  The covariance constructed from the perts is approximately consistent with the analysis covariance from the DA if the number of ensemble members is large.
- References:   Wei *et al.*   2005,   ***WMO TD No.1237****, ***WWRP  THORPEX No. 6***, *2005. p227-230.
-                                         2006,   **US Department of Commerce, NOAA/NCEP Office Note 453**,
-                                         33pp,  September 2006,    (also submitted to Tellus A, 2007).

**Notes:**
CIRA operational and research experience

## Slide 38

## Slide 39

## SOURCES OF FORECAST ERRORS 

***IMPERFECT KNOWLEDGE / REPRESENTATION OF  ***
_**GOVERNING LAWS**_
**USE OF IMPERFECT MODELS LEADS TO**:
-  Closure/truncation errors related to:
-  Spatial resolution
-  Time step
-  Type of physical processes explicitly resolved
-  Parameterization scheme chosen
- Structure of scheme
- Choice of parameters
- Geographical domain resolved
- Boundary condition related uncertainty (Coupling)
***NOTES****:*
-  Two main (initial cond. vs. model) sources of forecast errors hard to separate =>
-  Very little information is available on model related errors
-  Tendency in past to attribute all forecast errors to model problems
*Houtekamer, Buizza, Smith, Orrell, Vannitsem, Hansen, etc*

## WHAT HAPPENS IF MODEL ERRORS ARE IGNORED? 

**NCEP ENSEMBLE RESULTS:**
-  	***Bias in first moment******	******	******	******Bias in second moment***
- All members shifted statistically	*	*Perturbation growth lags error growth

- Y. Zhu

## The impact of using a second model at MSC

**The warm bias was reduced substantially and the U-shape disappeared by combining the two ensembles into the ****16-SEF/GEM ensemble****.**

- P. Houtekamer

## SAMPLING FORECAST ERRORS =

***REPRESENTING ERRORS DUE TO USE OF***
**IMPERFECT MODELS - 1**
**CURRENT METHODS**
**1)****	****Change structure of model (use different convective schemes, etc, ****MSC****)**
- Perturbation growth not affected?
- Biases of different model versions cancel out in ensemble mean?

- Spread

- Oper: 3 model versions
- Para: More model diversity

- Based on Houtekamer

- J. Du

## Spread

- Oper: 3 model versions (ETA, ETA/KF, RSM)
- Para: More model diversity

- RMS error

## SAMPLING FORECAST ERRORS =

***REPRESENTING ERRORS DUE TO USE OF***
**IMPERFECT MODELS – 2**
**CURRENT METHODS**
- Change structure of model (eg, use different convective schemes, etc, MSC)
**2)**	**Add stochastic noise (eg, perturb diabatic forcing, ****ECMWF****)**
- Modest increase in perturbation growth for tropics
- Some improvement in ROC skill for precip, for tropics

- 850 hPa Temp, NH

- Spread

- ROC Area

- Summer

- Winter

- Oper vs. Stochastic perturbations

- Buizza

## 850 hPa Temp

- Spread

- ROC Area

- Oper vs. Stochastic perturbations

- NH

- Tropics

- Summer

- Winter

- Buizza

## RESULTS FROM COMBINED USE OF RAS & SAS

**RESULTS FROM COMBINED USE OF RAS & SAS**

**500 hPa height RMS error, NH extratr.****SAS****, ****RAS,**** & Combination**

***NO POSITIVE EFFECT ON PRECIP OR HEIGHT SCORES***

- D. Hou

## RESULTS FROM COMBINED USE OF RAS & SAS

**RESULTS FROM COMBINED USE OF RAS & SAS**

- Rank histogram comparing distributions of sub-ensembles relative to each other
**AFTER BIAS CORRECTION, SAS & RAS SUB-ENSEMBLES COVER SAME SUBSPACE**

- 500 hPa height NH extratrop. RMS error for RAS, SAS, and NAS (no convection)
**NO DIFFERENCE WHETHER CONVECTIVE SCHEME IS USED OR NOT**

***CONVECTIVE SCHEME DOES NOT SEEM TO HAVE PROFOUND INFLUENCE ON FORECASTS EXCEPT PRECIP***

- D. Hou

## REPRESENTING MODEL RELATED UNCERTAINTYA STOCHASTIC PERTURBATION (SP) SCHEME

**General Approach: **Adding a stochastic forcing term in to the tendencies of the model equations.
**Strategy**: Generate the S terms from (random) linear combinations of the conventional perturbation tendencies.

**Desired Properties **
- 1.    Forcing applied to all variables
- 2.   Approximately balanced
- 3.   Smooth variation in space and time
- 4.    Flow dependent
- 5.    Quasi-orthogonal

*Example of Combination Coefficients*

*Reduced Outlier*

*Reduced Bias*

*Reduced Absolute Systematic error*

**RMSE**

*Increased Spread*

- ---- Without SP
- ---- With SP
- ---- Without SP
-     but optimal pp
-     (upper limit)

- Stochastic Parameterization (SP)

## STOCHASTIC PERTURBATIONS

- ECMWF operational (Buizza et al, 1999), A random  numbe (sampled from a uniform distribution) multiplied to the parameterized tendency
- ECMWF research (Shutts and Palmer, 2004), Cellular Automaton Stochastic Backscatterused to determine the perterbation
-  Simple Model Experiment (Peres-Munuzuri, 2003), multiplicative and additive stochastic forcing
-  NCEP METHOD UNDER TESTING
- Addition of flow-dependent perturbations to tendencies in course of integration
-  DETAILS – Add to each perturbed member:
- Difference between single high & low-res forecasts (after scaling and filtering)
- Perturbation based on the differences among the ensemble members at previous 																	step in integration
- Use global or localized perturbation approach
- Random or guided selection of members (e.g., use difference between 														most similar members)

**STOCHASTIC PERTURBATIONS**

## REPRESENTING MODEL RELATED UNCERTAINTYA STOCHASTIC PERTURBATION SCHEME

**Strategy**: Generate the S terms from (random) linear combinations of the conventional perturbation tendencies.

**General Approach: **Adding a stochastic forcing term in to the tendencies of the model equations.
**Assumption: **The perturbations (difference between ensemble members and the control) in the conventional tendencies provide a sample of realizations of the additional stochastic forcing S.

**Desired Properties **
- 1.    Forcing applied to all variables
- 2.   Approximately balanced
- 3.   Smooth variation in space and time
- 4.    Flow dependent
- 5.    Quasi-orthogonal

**Expected Results**
- Increased spread
- Reduced systematic error
- Improved probabilistic forecast

*Reduced Outlier*

*Example of Combination Coefficients*

## Outliers: H500, day 6 forecast, 20041002 

**Without SP**
- large number of outliers with
- negative and positive forecast bias

**With SP**
- the number of outliers is
- significantly reduced

## Statistics: Ensemble Spread and Error of Ensemble MeanIncreased Spread, Reduced Mean Error (ME)Reduced Mean Absolute Systematic Error (MASE)----- Without SP      ------- With SP

- MASE

- Mean Error

- Mean Error

- Solid, rmse
- Dash: spread

- Solid: rmse
- Dash: spread

## Comparison with Post-Processing (PP) RPSS: Improved in both cases (SP and PP)SP is more effective in week 2 forecast

- ---- Without SP
- ---- With SP
- ---- Without SP
-     but optimal pp
-     (upper limit)

- ---- Operation
- ---- Operation +1%PP
- ---- Operation +
-        optimal pp
-        (upper limit)

- Stochastic Parameterization (SP)

- Bias-correction (1%PP)

## SAMPLING FORECAST ERRORS =

***REPRESENTING ERRORS DUE TO USE OF***
**IMPERFECT MODELS – 3**
**CURRENT METHODS**
- Change structure of model (eg, use different convective schemes, etc, MSC)
- 	Model version fixed, whereas model error *varies in time*
- 	Random/stochastic errors not addressed
- 	Difficult to maintain
- Add stochastic noise (eg, perturb diabatic forcing, ECMWF)
- 	Small scales perturbed
- 	If otherwise same model used, larger scale biases may not be addressed
- Do they work? Advantages of various approaches need to be carefully assessed
-  Are flow dependent variations in uncertainty captured?
-  Can statistical post-processing replicate use of various methods?
** *****NEED NEW***
***MORE COMPREHENSIVE AND***
***THEORETICALLY APPEALING ***
***                                             ******	****** ******	******APPROACH***

##     MODEL ERRORS ARE PART OF LIFE, WILL NEVER GO AWAY

**REPRESENTING MODEL RELATED UNCERTAINTY**

**MODEL ERRORS ARE DUE TO:**
-  Truncation in spatial/temporal resolution –
-  Need to represent stochastic effect of unresolved scales
-  Add parameterized random noise
-  Truncation in physical processes resolved
-  Need to represent uncertainty due to choice of parameterization schemes
-  Vary parameterization schemes / parameter values
**    ****MODEL ERRORS ARE PART OF LIFE, WILL ****NEVER**** GO AWAY**
**IN ENSEMBLE ERA, **
***NWP MODELING PARADIGM NEEDS TO CHANGE***

## NEW APPROACH TO NWP MODELING –

**REPRESENTING MODEL RELATED UNCERTAINTY**

***IT IS NOT ENOUGH TO PROVIDE SINGLE (BEST) ******	******MODEL***
***	******	******	******	******	******	******	******FORECAST ***
**JOINT EFFORT NEEDED BETWEEN MODELING & ENSEMBLE COMMUNITY**
**FOR OPTIMAL ENSEMBLE PERFORMANCE,**
**MODELS NEED TO REALISTICALLY REPRESENT ALL MODEL-RELATED **
- 			***Resolution (time and space truncation)***
***	******	******	******Parameterization-type (unresolved physics)***
**	****	****	****	****UNCERTAINTY AT THEIR SOURCE** -
- 				Like in case of initial condition-related uncertainty
**FOR MODEL IMPROVEMENTS,**
**ENSEMBLE OFFERS TOOL TO SEPARATE INITIAL & MODEL ERRORS**
- 	Case dependent errors can be captured and corrected

## WILL NEW APPROACH ADD VALUE?

**WILL IT ENHANCE RESOLUTION OF PROBABILISTIC FCSTS?**
**WILL IT GIVE CASE-DEPENDENT ESTIMATES **
**(INSTEAD OF AVERAGE STATISTICAL MEASURE) OF**
** ****MODEL-RELATED UNCERTAINTY?**

## Slide 59

## OUTLINE / SUMMARY

- TRADITIONAL NWP APPROACH
- REDUCE FORECAST UNCERTAINTY
- IGNORE REMAINING ERRORS
- Problem for users
- SOURCES OF FORECAST ERRORS
- INITIAL CONDITION
- NUMERICAL MODEL
- ESTIMATING AND SAMPLING FORECAST ERRORS
- INITIAL CONDITION
- Breeding technique / ET
- MODEL ERRORS
- No solid scientific basis, open research
- POTENTIAL VALUE OF ENSEMBLE APPROACH
- IMPROVED SINGLE VALUE ESTIMATE
- CASE DEPENDENT ESTIMATE OF UNCERTAINTY
- FULL PROBABILITY DISTRIBUTION / TRAJECTORIES

## BACKGROUND

## Slide 62

## ENSEMBLES: WHEN?

- Single forecast approach favored when
***Case-dependent variations*** are weak in
- Level of linear error growth at short lead times
- Pdf evolution at short lead times (ie, quasi-linear behaviour)
- Model-related error behaviour (at any lead time)
- Aggregate bias-correction algorithms adequate
- Use ensembles otherwise
- Review criteria above for each application
- Bias-correct both single value & ensemble forecasts (ie, pdf)
- Decide on forecast configuration based on results
- “Generic” configuration
- Higher resolution control for short lead time if beneficial
- Lower resolution ensemble out to longer lead times
- Benefits from combining hi-re control & lo-res ensemble at shorter leads?
- Considerations
- Integrations must resolve phenomena of interest
- Unless sophisticated statistical down-scaling techniques can be developed

## PROPAGATING FORECAST UNCERTAINTY

- z

*Single value*

*Distribution*

- Ensemble Forecasting:
- Central role – bringing the pieces together

**PROPAGATING FORECAST UNCERTAINTY**

## NOAA

- NCEP

**OPERATIONS**

**RESEARCH TO OPERATIONS**

**OPERATIONS TO APPLICATIONS**

**USER SUPPORT**

**DIRECTED R&D**

**RESEARCH & DEVELOPMENT**

**SOCIETAL APPLICATIONS**

- General basic & applied R&D
- R&D directed toward operations
- Systematic transition to operations
- 7/24 Product generation
- Systematic transition to applications
- Delivery of products to end users
-  Decision making, feedback

**RESEARCH TO OPERATIONS TO APPLICATIONS FUNNEL**

**WHO**

**WHAT**

**Wide research community**
**Research Labs,            Grants**
**Environmental Modeling Center**
**NCEP Central Operations**
**NCEP Service Centers**
**      ****WFOs,        Weather Enterprise**
**Diverse user community**

## ENSEMBLES AND THE RESEARCH COMMUNITY

**SOCIOECON.**

**ENSEMBLES AND THE RESEARCH COMMUNITY**
- LINKED THROUGH THORPEX – MAJOR INTERNATIONAL RESEARCH PROGRAM
***GOAL: Accelerate improvements of high impact weather forecasts***

**SYSTEM**

- CLIMATE FORECASTING / CTB

***INTEGRATED**** DATA ASSIMILATION & FORECASTING*

***Days 15-60***

- NWS OPERATIONS

- GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS)

***GLOBAL OPERATIONAL***

***TEST CENTER***

***GLOBAL OPERATIONAL***

***TEST CENTER***

***MODEL ****ERRORS & HIGH IMPACT MODELING*

## ENSEMBLES AND NOAA SERVICES

- NWS operations is strictly requirement driven
- Culture must change to support evolution in operations
- New emphasis on high impact events
- W&W Goal & EMP Sub-Goal involvement
- High Impact Events Theme
- Adaptive and event driven
- Integrated across the spectrum of services
- Probabilistic approach
- Enhanced automated guidance
- New role for forecasters
- Environmental Information Repository
- “Establish comprehensive suite of ensemble forecast systems (“forecast engine”) that will facilitate the generation of automated forecast guidance products in the framework of the new NOAA CONOPS as the basis (“forecast engine”) for NOAA operations regarding high impact events:
- New automated “forecast engine that adapts to high impact events
- Adaptive observations
- Adaptive ensemble suite
- Statistical post-processing

**ENSEMBLES AND NOAA SERVICES**

- An Integrated Plan of Operations
- NOAA’s Weather and Water
- High Impact Events
- FY 2009 – 2013
- August 3, 2006

## CONSIDERATIONS FOR OPERATIONAL IMPLEMENTATIONS

- Offline research, parallel development, pre-implementation testing
**User relevant verification statistics** (ie, bias corrected & downscaled forecasts)
- Economy
- Operations is narrowest point in Research-Operations-Applications funnel
- Lots of research/development, **one system in operations**
**Computational efficiency**
- Maintenance
- Minimize work needed for transfer (R2O, O2A, from machine to machine, etc)
**Unified approaches** preferred if performance not sacrificed
- Interconnectedness
- Each piece of operations intimately connected with rest of system
**Incremental improvements** to existing system OR
- Very careful **long-term planning** for major upgrades

**CONSIDERATIONS FOR OPERATIONAL IMPLEMENTATIONS**

## ENSEMBLE DEVELOPMENT CONSIDERATIONS

**ENSEMBLE DEVELOPMENT CONSIDERATIONS**

- Common scientific principles - Chaos affects all spatial/temporal scales
- Quantify all forecast uncertainty - Inseparable from forecasting in general
- Links with observing system, data assimilation, numerical modeling, user applications
- Represent all forecast uncertainty at their source - Otherwise poor reliability
- Only chance to propagate true uncertainty through forecast process
- Unified approach
- Common techniques across applications wherever appropriate / possible
- Ensemble team members
- Work in implementation teams, coordinated with rest of EMC & NCO
- Interact with broader research and user communities