# Presentation: 288948

## COMPLETING THE FORECAST:ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY

- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- NRC Report; Louis Uccellini, Steve Lord & Ensemble Team
- http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

## OUTLINE

- REVIEW OF 3RD ENSEMLBE USER WORKSHOP
- MAJOR RECOMMENDATIONS REGARDING NRC REPORT
- WHAT, WHY, AND HOW TO CHANGE?
- SPECIFIC RECOMMENDATIONS

## 3rd ENSEMBLE USER WORKSHOP

- Oct. 30 – Nov. 1 2006, Laurel, MD
- Close to 100 participants
- NWS Regions (12), Headquarters (15), NCEP (37)
- OAR (6), other government (4), private (4), academic (8) sectors & international (8)
- For further info, see: _[http://](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[wwwt](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[.](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[emc](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[.](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[ncep](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[.](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[noaa](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[.](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[gov](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[/](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[gmb](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[/](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[ens](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[/](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[UserWkshop](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)__[_Oct2006.html](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)_
- Topics
- Assessing & propagating uncertainty throughout entire forecast process
- From observations to users
- Working group discussions
- Ensemble configuration
- Statistical post-processing
- Data depository / Interrogation / Product generation/dissemination / Verification
- User support / Outreach / Training
- Outcome
- Enthusiastic discussions
- Convergence on number of topics
- Open questions needing further research identified
- Strong support for sustained effort/engagement, annual meetings, etc
- Recommendations
- Course of action in response to NRC report
- Presented here first – will solicit and incorporate feedback from participants

**3****rd**** ENSEMBLE USER WORKSHOP**

## MAJOR RECOMMENDATIONS - 1

**All major recommendations embraced**
- Develop roadmap for assessing & communicating forecast uncertainty
- Based on science, technology, workforce considerations
- Consistent with NOAA’s mission, NWS plans, embraced by “Enterprise”
- Define end goal, 5-10 years horizon
- Adopt ensemble-based forecast process
- Maximize forecast skill
- Unify scientific, software, and technological infrastructure across NOAA
- Weather, water, climate applications
- Form high level planning and oversight team – March 2007
- Each NWS office to delegate one representative
- Team reports to Corporate Board
- Name programmatic and technical leads
- Develop roadmap – Sept. 2007

**MAJOR RECOMMENDATIONS - 1**

## MAJOR RECOMMENDATIONS - 2

- Revise operational requirements to make them probabilistic
- Make probabilistic format the primary requirement
- For each forecast application:
- Replace single value/categorical format with probabilistic format as primary requirement
- Revise/supplement corresponding performance measures (GPRA)
- Essential for orderly transition from traditional to new forecast process
- NWS is requirement-driven organization
- Without clear new requirements, process is doomed
- Phased implementation schedule in consultation with responsible offices
- Allows orderly transition to new requirements
- New requirements prepared by Planning & oversight team (or its designate)
- Assisted by responsible NWS office
- Presented to Corporate Board for approval – By March 2008

**MAJOR RECOMMENDATIONS - 2**

## MAJOR RECOMMENDATIONS - 3

- Revise operational requirements to make them probabilistic (1 year)
- Design, develop, and gradually implement new forecast process
- Focus on missing scientific, technological, and human components
- Identify self-contained components with
- Clear requirements and interfaces between components
- Define basic capabilities achievable in 2-3 years
- Limited but consistent with and leading to end goal
- Full capabilities in 5-10 years
- Interface with NOAA **THORPEX** program
- Research/development to improve skill & utility of probabilistic forecasts
- Leverage related major NOAA, national, and international efforts
- Integrate with NWS, NOAA, national activities
- New NOAA **CONOPS** process
- NOAA-wide regional service plans
**NUOPC** planning/development
- Provide long-term funding support through **PPBES**
- Overlap with W&W High Impact Event Theme Team
- Form **development teams** for specific tasks/components following workshp
- Identify potential contributors within & outside of NWS & NOAA

**MAJOR RECOMMENDATIONS - 3**

## PROPOSED CHANGE

- Incorporate assessment and communication of uncertainty in forecast process
- Is it a **major change in course** of “Weather Ship”?
- Ie, abandon course of ever improving single forecast scenario (expected value)?
- No – Expand, not abandon
- Keep improving fidelity of forecasts, PLUS
- Add new dimension
- Capture other possible scenarios – ensemble forecasting
- Use a flotilla, instead of one ship, in exploring nature
- Existing activities are subset of expanded forecast process
- Single value forecast is expected value of full probability distribution
- Can keep serving forecasts in old format to users who prefer that

**PROPOSED CHANGE**

## Single forecast (driven by GFS winds) example for drifting virtual ice floe

- Single forecast (driven by GFS winds) example for drifting virtual ice floe

*Bob Grumbine, EMC*

- Initial position

- 7 September 2006

## Ensemble forecast for drifting ice floe for same case

**Ensemble forecast** for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## Most likely forecast for drifting ice floe for same case

- Most likely forecast for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## WHY CHANGE IS NEEDED?

- They admittedly want minimal or no uncertainty in forecasts
- Distinction between no uncertainty in the forecast, vs. not talking about it
- Forecast uncertainty cannot be arbitrarily reduced
- Despite major ongoing & continuing efforts, they persist forever
- Chaotic nature of atmosphere - land surface – ocean coupled system + initial/model errors
- Level of uncertainty is determined by nature and level of sophistication in forecast system
- Forecast uncertainty can be ignored though
- Negative consequence on informed users
- Not able to prepare for all possible outcomes
- Assumes a certain scenario and remains vulnerable to others
- Possibly serious loss in social/economic value of forecast information
- Why forecasters (should) care about forecast uncertainty?
- Imperfect forecasts are consistent w. observations (reliable) only if in prob format
- If in other format, must be brought into probabilistic format through
- Verification / bias correction

**WHY CHANGE IS NEEDED?**

## ADVANTAGES OF PROBABILISTIC FORMAT

**Old paradigm**
- Convoluted forecaster-user decision process
- User expects forecaster to make decision for them in presence of uncertainty
- “Will it rain?” – “80%” – “But tell me, will it rain?”
**New paradigm**
- Forecaster and user decision processes enhanced and better linked
- Allows forecasters to capture all knowledge about future conditions
- Provision of information related to multiple decision levels in probabilistic format critical
- Provider helps interpret probabilistic info & and modify user decision process if needed
- Option to continue providing single value or other limited info until user ready
- Allows users to decide about most beneficial course of action given all possibilities
- Proper use of probability or other uncertainty information needed - Training
- User requests critical weather forecast info depending on their sensitivity

**ADVANTAGES OF PROBABILISTIC FORMAT**

## TRADITIONAL FORECAST PROCESS

- Reducing uncertainty in single forecast is main emphasis
- Loss of accuracy in forecast estimate of expected value of distribution
- Mean of ensemble cloud provides better estimate
- Ignores or simplifies forecast uncertainty
- Uncertainty assessed as statistically averaged error in single fcst (second thought)
- Ensemble cloud provides better estimate of case dependent variations in uncertainty
- Use of single value / categorical forecast format
- Difficulty in formulating/communicating plausible alternate scenarios
- Ensemble member forecasts can directly feed into Decision Support Systems
- One-way flow of information from observations to users
- Not adaptable to case dependent user requirements
- Ensemble can propagate back user requirements to adaptive
- Observing, assimilation, modeling/ensemble, post-processing and application components
- Applications in planning and execution of new CONOPS in high impact events

**TRADITIONAL FORECAST PROCESS**

## PROPAGATING FORECAST UNCERTAINTY

- z

*Single value*

*Distribution*

- Ensemble Forecasting:
- Central role – bringing the pieces together

**PROPAGATING FORECAST UNCERTAINTY**

## HOW CAN IT BE DONE? NEW PARADIGM

- Expand forecasting with new dimension of uncertainty
- Multiple scenarios (in place of single scenario)
- Provides best forecast estimate for both expected value (as before) and uncertainty (new)
- Unified scientific, technological, human approach
- Sharing resources across NWS & NOAA
- Ensemble is centerpiece both symbolically and figuratively in forecast process
- Ensembles act as a glue & two-way information channel
- Observing system, data assimilation, numerical modeling
- ENSEMBLES
- Statistical post-processing, product generation, decision making
- Design, develop, & implement missing components of new forecast process
- Gradual, measured steps
- Basic capability - Short-term, 2-3 yrs, leading to
- Full implementation - Long-term, 5-10 yrs

**HOW CAN IT BE DONE? NEW PARADIGM**

## ENSEMBLES AND THE RESEARCH COMMUNITY

**SOCIOECON.**

**ENSEMBLES AND THE RESEARCH COMMUNITY**
- LINKED THROUGH THORPEX – MAJOR INTERNATIONAL RESEARCH PROGRAM
***GOAL: Accelerate improvements of high impact weather forecasts***

**SYSTEM**

- CLIMATE FORECASTING / CTB

***INTEGRATED**** DATA ASSIMILATION & FORECASTING*

***Days 15-60***

- NWS OPERATIONS

- GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS)

***GLOBAL OPERATIONAL***

***TEST CENTER***

***GLOBAL OPERATIONAL***

***TEST CENTER***

***MODEL ****ERRORS & HIGH IMPACT MODELING*

## SPECIFIC RECOMMENDATIONS - 1

- Focus on adaptive methods applicable for high impact events
- Collection & use of observations (targeted observations)
- Data assimilation (case dependent background error estimation)
- Numerical modeling (adaptive resolution & high impact modeling)
- Ensemble forecasting (case dependent variations in membership & composition)
- Decision support systems (flexible user actions depending on forecast probabilities)
- Bias correction & downscaling methods for ensembles
- Estimate/correct lead-time dependent bias in ensemble forecasts (on model grid)
- Generate fine resolution (NDFD grid) uncertainty/ensemble data
- Establish connection between (bias corrected) coarse model vs. fine user grids
- Use reanalysis & hind-casts with operational systems as needed
- Define summary ensemble information to be used to (ST)
- Collapse vast amount of ensemble data for inclusion in expanded NDFD/NDGD
- E.g., 10, 50, and 90 percentile of forecast distribution (in place of single value)
- Manually inspect/modify summary ensemble information

**SPECIFIC RECOMMENDATIONS - 1**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS - 2

- Expand NDFD/NDGD database to include forecast uncertainty (ST)
- Develop capability to hold all ensemble trajectories (LT)
- All members, variables, lead times
- Develop ensemble interrogation, modification, & product generation tools to
- Derive summary information from ensemble (ST)
- Manually modify summary ensemble info (ST)
- Derive additional statistics from summary info (product generation, ST)
- Automatically modify ensemble trajectories based on modified summary info (LT)
- Derive any info from full ensemble data (product generation, LT)
- Develop telecommunication facilities to access data
- Summary info & limited derived products (ST)
- All ensemble forecasts & derived products (LT)

**SPECIFIC RECOMMENDATIONS - 2**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS - 3

- Assess statistical reliability and resolution for
- Computing official performance measures
- Evaluating value added along forecast chain
- Assessing value in newly developed vs. operational techniques
- Develop & implement comprehensive training to
- Prepare all participants for their new roles in expanded forecast process, incl.
- Statistical background
- Ensemble methods
- Best forecast practices in assessing uncertainty
- Applications of probabilistic & other uncertainty information
- Develop outreach program on use & communication of uncertainty
- In partnership with entire Weather, Water & Climate Enterprise
- Determine best ways of communicating uncertainty
- Compile sample of Decision Support Systems using uncertainty information
- Establish close partnership with public sector users (e.g., emergency, water management)
- User feedback on new activities
- Explore how forecast process can be adapted according to user requirements

**SPECIFIC RECOMMENDATIONS - 3**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## NFUSE PRIORITIES?

- NRC Recommendations
- Do we agree on major recommendations?
- If not, let’s discuss any issues
- If so, can we accept those as the basis/guidelines for our work
- 3rd Ensemble User Workshop Recommendations
- Three major recommendations in response to NRC Report
- High level roadmap guiding our collaboration (Sept 2007)
- New operational requirements (March 2008)
- Design, develop, and gradually implement new forecast process
- Basic capabilities (2009-2010)
- Full capabilities (2012-2017)
- Nine specific recommendations on design, development, implementation
- Work areas identified, links to be worked out
- Continue development of expanded forecast process
- Bias correction & downscaling methods for ensembles
- Define summary ensemble information to be used (ST)
- Contribute to establishment of NOAA-wide environmental data depository
- Develop ensemble interrogation, modification, & product generation tools
- Develop telecommunication facilities to access data
- Develop unified NWS/NOAA probabilistic verification package (ST)
- Develop & implement comprehensive training
- Develop outreach program on use & communication of uncertainty
***Do we need to modify/augment etc these recommendations?***

**NFUSE PRIORITIES?**