# Presentation: 288939

## REVIEW OF RECOMMENDATIONS FROM THE 3RD ENSEMBLE USER WORKSHOP (OCT. 2006)

- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- Louis Uccellini, Steve Lord & Ensemble Team
- Paul Schultz, DJ Seo, Tom Hamill, Steve Mullen, Julie Demargne
- http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

## MAJOR RECOMMENDATIONS - 1

**All major recommendations embraced**
- Develop roadmap for assessing & communicating forecast uncertainty
- Based on science, technology, workforce considerations
- Consistent with NOAA’s mission, NWS plans, embraced by “Enterprise”
- Define end goal, 5-10 years horizon
- Adopt ensemble-based forecast process
- Maximize forecast skill
- Unify scientific, software, and technological infrastructure across NOAA
- Weather, water, climate applications
- Suggested in Mar 2007 to develop roadmap by Sept. 2007

**MAJOR RECOMMENDATIONS - 1**

## MAJOR RECOMMENDATIONS - 2

- Revise operational requirements to make them probabilistic
- Make probabilistic format the primary requirement
- For each forecast application:
- Replace single value/categorical format with probabilistic format as primary requirement
- Revise/supplement corresponding performance measures (GPRA)
- Essential for orderly transition from traditional to new forecast process
- NWS is requirement-driven organization
- Without clear new requirements, process is doomed
- Phased implementation schedule in consultation with responsible offices
- Allows orderly transition to new requirements
- Suggested in Mar 2007 to prepare plan by March 2008

**MAJOR RECOMMENDATIONS - 2**

## MAJOR RECOMMENDATIONS - 3

- Revise operational requirements to make them probabilistic (1 year)
- Design, develop, and gradually implement new forecast process
- Focus on missing scientific, technological, and human components
- Identify self-contained components with
- Clear requirements and interfaces between components
- Define basic capabilities achievable in 2-3 years
- Limited but consistent with and leading to end goal
- Full capabilities in 5-10 years
- Interface with NOAA **THORPEX** program
- Research/development to improve skill & utility of probabilistic forecasts
- Leverage related major NOAA, national, and international efforts
- Integrate with NWS, NOAA, national activities
- NOAA-wide regional service plans
**NUOPC** planning/development
- Provide long-term funding support through **PPBES**
- Overlap with W&W High Impact Event Theme Team
- Form **development teams** for specific tasks/components following workshp
- Identify potential contributors within & outside of NWS & NOAA

**MAJOR RECOMMENDATIONS - 3**

## SPECIFIC RECOMMENDATIONS – 1-3

- Focus on adaptive methods applicable for high impact events
- Collection & use of observations (targeted observations)
- Data assimilation (case dependent background error estimation)
- Numerical modeling (adaptive resolution & high impact modeling)
- Ensemble forecasting (case dependent variations in membership & composition)
- Decision support systems (flexible user actions depending on forecast probabilities)
- Bias correction & downscaling methods for ensembles
- Estimate/correct lead-time dependent bias in ensemble forecasts (on model grid)
- Generate fine resolution (NDFD grid) uncertainty/ensemble data
- Establish connection between (bias corrected) coarse model vs. fine user grids
- Use reanalysis & hind-casts with operational systems as needed
- Define summary ensemble information to be used to (ST)
- Collapse vast amount of ensemble data for inclusion in expanded NDFD/NDGD
- E.g., 10, 50, and 90 percentile of forecast distribution (in place of single value)
- Manually inspect/modify summary ensemble information

**SPECIFIC RECOMMENDATIONS – 1-3**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS – 4-6

- Expand NDFD/NDGD database to include forecast uncertainty (ST)
- Develop capability to hold all ensemble trajectories (LT)
- All members, variables, lead times
- Develop ensemble interrogation, modification, & product generation tools to
- Derive summary information from ensemble (ST)
- Manually modify summary ensemble info (ST)
- Derive additional statistics from summary info (product generation, ST)
- Automatically modify ensemble trajectories based on modified summary info (LT)
- Derive any info from full ensemble data (product generation, LT)
- Develop telecommunication facilities to access data
- Summary info & limited derived products (ST)
- All ensemble forecasts & derived products (LT)

**SPECIFIC RECOMMENDATIONS – 4-6**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS – 7-9

- Assess statistical reliability and resolution for
- Computing official performance measures
- Evaluating value added along forecast chain
- Assessing value in newly developed vs. operational techniques
- Develop & implement comprehensive training to
- Prepare all participants for their new roles in expanded forecast process, incl.
- Statistical background
- Ensemble methods
- Best forecast practices in assessing uncertainty
- Applications of probabilistic & other uncertainty information
- Develop outreach program on use & communication of uncertainty
- In partnership with entire Weather, Water & Climate Enterprise
- Determine best ways of communicating uncertainty
- Compile sample of Decision Support Systems using uncertainty information
- Establish close partnership with public sector users (e.g., emergency, water management)
- User feedback on new activities
- Explore how forecast process can be adapted according to user requirements

**SPECIFIC RECOMMENDATIONS – 7-9**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## NEW DEVELOPMENTS SINCE WG RECOMMENDATIONS

- NOAA/NWS Forecast Uncertainty Team (NFUSE) formed
- Early 2007
- Assess and discuss status, search for solutions
- AMS ACUF Team formed
- Enterprise planning activities
- NOAA/NWS Forecast Uncertainty program formed
- Outgrowth of NFUSE
- Oversight panel established
- Budget planning for out-years
- Uncertainty-related activities wide-spread, affect all forecasting
**What is next?**
- Coordinated
- Work level plans
- Related activities

## 2008 WORKSHOP CONTRIBUTIONS

- Agree on** long term plan**
- What we want in 5-10 yrs – desired end result/framework
- Not how we achieve goal
- Multiple avenues can be tested within single framework
- Agree on what we can achieve in **next 2-3 years** with expected resources
- Limited version of long term goal – interim result/framework
- Development of short term solution will contribute to long term plan
- Contribute to **forming of Work Teams** addressing well defined sub-problems
- Separate pieces that can be worked on independently
- Leads/principles provide overall coordination

## DISCUSS / AGREE ON LONG TERM GOAL

*Capability to answer any question related to future weather, climate, water, including forecast uncertainty*
- Considering state of related science, what do we need for that?
- Bias corrected, downscaled ensemble forecasts – ***Ultimate dataset***
- Allows derivation of single/joint probabilities
- Information on spatial/temporal/cross-variable correlations
- Trajectories (scenarios)
- Work independent of particular product/service needs (all questions covered)
- Major technical requirements
- Storage for ensemble data
- 20-100 times more data compared to current “NDFD”
- Distribution of / access to ensemble data
- 20-100 fold increase in bandwidth
- Technical tools to display / manipulate ensemble data
- Special AWIPS-2 capabilities
- Forecasters trained to work with ensemble data
- 2-step approach warranted
- First implement limited capability
*Will provide large portion of total benefit with relatively small investment*

## DISCUSS / AGREE ON INTERMEDIATE GOAL

*Capability to provide uncertainty/probabilistic info on any weather element with some approximation*
- Considering state of related science, what do we need for that?
- Summary info that allows the generation of single-variable cdf/pdf
- Bias corrected, downscaled
- Can be generated based on a single or ensemble NWP
- Allows derivation of single-variable probabilities
- Including information on spatial/temporal/cross-variable correlations
- Technical requirements
- Storage for summary info
- 3-fold increase in NDFD data
- Distribution of / access to ensemble data
- 3-fold increase in bandwidth
- Technical tools to display / manipulate ensemble data
- Subset of long term toolkit in AWIPS-2
- Connect summary stats & cdf/pdf
- Forecasters trained to work with ensemble data
- With right choice of summary info, forecasters eased into new paradigm

## PHASE 1 PLAN

***Definition of basic format for forecast uncertainty***
- Advantages of using a common format
- Facilitates flow of info between three major elements in forecast process
- Numerical guidance
- Human forecasters
- External users
- Provides basis for collaboration among various groups
- Inter-comparison of different approaches
- Requirements
- Can be easily interpreted by users
- Intuitive meaning
- Can be manipulated by forecasters
- Related to current forecast modification practices
- Can be easily communicated
- Limited data volume
- Can be stored in NDFD
- Related to current forecast format
- Additional formats can be easily derived
- Based on standard statistical considerations
- Choice of exact format not critical
- As long as software available to generate alternate formats

**PHASE 1 PLAN**

## CHOICE OF FORMAT FOR ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY

- Recommended by Predictability WG at WR/NWS SOO/DOH Workshop (Salt Lake City, May 2003)
- Discussed and recommended by Predictability Workshop at Univ. Wisconsin (Madison, Mar 2004)
- Studied in detail and adopted at UKMET Office (2006)
**Suggested format**
- Beyond mid-point estimate (50%, or mode, or mean?), include
- Two additional fields for each NDFD variable
- 10 percentile value of forecast distribution
- 90 percentile value of forecast distribution
- Readiness
- Numerical guidance
- Can be generated based on ensemble data, using NAWIPS software
- Forecasters
- Can use same tools to manipulate 2 new fields as with most likely field
- Users
- UKMET research indicates this is intuitive

**CHOICE OF FORMAT FOR ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY**

## BACK

- _[BACK](slide9.xml)_

## EXAMPLE FROM UK METOFFICE - TEMP

**EXAMPLE FROM UK METOFFICE - TEMP**

*Courtesy of Mark Roulston*

## EXAMPLE FROM UK METOFFICE - PRECIP

**EXAMPLE FROM UK METOFFICE - PRECIP**

*Courtesy of Mark Roulston*

## QUESTIONS ABOUT SUGGESTED FORMAT

- No need to worry (too much)
- Just need a common format to connect three main areas
- Alternative formats can/will be generated
- What NDFD grids describe currently?
- Most likely value (mode of pdf)?
- Expected value (mean of pdf)?
- 50% exceedence value (median of pdf)?
- What mid-point parameter an expanded NDFD should have?
- What do users need?
- What is most intuitive to forecasters?
- Would median (50 percentile) be best choice? Similar to 10, 90 percentile value
- What threshold may work best for limits?
- 10 & 90 - not too narrow, not too wide?
- 5 & 95 percentile may be too extreme for forecasters?
- 20 & 80 may not be inclusive enough?
- What software needs to be developed to generate other formats?
- Fit parametric distribution to 3 points
- Roman Krzysztofowicz’ library of 40+ distributions
- Any probability or its inverse can be derived for single variables
- Basic probabilistic forecast data-base

**QUESTIONS ABOUT SUGGESTED FORMAT**

## TOOLS NEEDED FOR INTERMEDIATE STEP

- Modify summary statistics (forecasters’ role)
- Fit parametric distribution to modified summary statistics
- Adjust ensemble members to be consistent with official forecast
- For users who need trajectories
- Derive probabilities from fitted cdf/pdf
- Same statistical tools must be used across organization
- Ensemble generation, display, manipulation, product generation, etc
- Same tools can be used in final stage to manipulate ensemble distribution
- Summary statistics inspected & modified by forecasters
- Changes back-propagated to ensemble data set

## WORK AREAS

- All topics need to be developed in coordination
- Plenary sessions for interactions
- Topics A & B are especially connected in each WG
- Same groups discussing both topics in 2 sessions
- Third WG session to focus on connections
- Working Group 1
- Topic A	Operational requirements
- Topic B	Corporate outreach
- Working Group 2
- Topic A	Ensemble forecasting
- Topic B	Statistical post-processing
- Working Group 3
- Topic A	Ensemble data depository / access
- Topic B	Database interrogation / forecaster tools
- Working Group 4
- Topic A	Forecaster’s role
- Topic B	Forecaster training

## WORK AREAS

**Operational requirements**
- General requirements first to “justify” effort
- Detailed requirements later as effort unfolds
**Corporate outreach**
- Link with ACUF process
- Promote quantitative use of uncertainty information in Decision Support Systems
**Ensemble forecasting**
- Centerpiece of uncertainty effort
- THORPEX
- NOAA, national, international research/development
- NAEFS, NUOPC, TIGGE, GIFS

## WORK AREAS – CONT

**Statistical post-processing**
- Problem
- Relate coarse resolution biased forecast to user relevant fine resolution information
- Break up problem to facilitate collaboration
- Bias correction of coarse resolution forecast grid wrt NWP analysis
- Cheap
- Sample of forecasts / hind-casts needed
- Downscaling
- Relate coarse resolution NWP and fine resolution observationally based analyses
- Perfect prog approach
- No need for hind-casts
- Creation of observationally based fine resolution analysis
- Estimate of truth
- Groups can collaborate on each topic
- Addressing all 3 problems in one swoop limits collaboration

## NAEFS RESULTS

**NAEFS RESULTS**

- 8 days total gain in skill
- Downscaling more important than bias correction
- Less need for hindcasting?
*Need for local observationally based analysis*
- Multicenter approach adds 1-2 days skill

*NCEP raw-downscaled*

***NCEP biascorrected-downscaled***

*NCEP raw*

*NAEFS final*

## WORK AREAS – CONT 2

**Ensemble data depository / access**
- Create NOAA digital forecast database (linked with other NOAA data)
- Summary statistics in phase 1
- All ensemble members in phase 2
- Provide easy access to internal / external users
- NOMADS, etc?
- Link with multi-center ensembles
- NAEFS – NUOPC – GIFS
**Database interrogation / forecaster tools**
- Derive information from summary stats / ensembles
- Modify summary stats
- Back-propagate modified info into ensemble
- Verify added value at each step of forecast process
- Need for unified verification capability across forecast process
- Common requirement across all WGs

## WORK AREAS – CONT 3

**Forecaster’s role**
- Traditional role
- Generate forecast, or modify numerical guidance
- Changing environment
- Need to generate very large amount of forecast data
- Quality of automatic numerical guidance improves
- New role:
- Direct and quality control forecast process
- Identify high impact events
- Assign adaptively configurable observational, modeling, etc resources
- Interfere with automated process only during critical high impact events
- User outreach
- Interpret probabilistic forecast
**Forecaster training**
- Prepare for new role related to ensemble / probabilistic forecasting
- Modification
- Interpretation
- Outreach

## RTMA Analysis

- _GMOS forecast_

- _NAEFS products_

- Surface Temperature MAE
- CONUS, Sept. 2007
-  12Z NDFD vs. 00Z MOS/_GMOS_/_NAEFS_

- Surface Temperature MAE
- CONUS, Sept. 2007
-  00Z _GMOS_ vs. 00Z _NAEFS _

- METAR  obs. 1221 sites

**0.5°C**

**0.5°C**

## RTMA Analysis

- CONUS, Sept. 2007
-  12Z NDFD vs. 00Z MOS/_GMOS_/_NAEFS_

- Surface Temperature Pointwise Bias
- CONUS, Sept. 2007
-  00Z _GMOS_ vs. 00Z _NAEFS_

- _GMOS forecast_

- _NAEFS products_

- METAR  obs. 1221 sites

- Pointwise Bias

- Area Mean Bias

**0.6°C**

**0.6°C**

## BACKGROUND

## Example for using new mask      

- Example for using new mask

## MDL GMOS  & NAEFS Downscaled Forecast  

- MDL GMOS  & NAEFS Downscaled Forecast
- Mean Absolute Error w.r.t. RTMA  Average For Sept. 2007

- 12-h GMOS
- Forecast

- 12-h NAEFS Forecast

- For CONUS:
- NAEFS(1.01) : GMOS(1.59)
- 36% impr. over GMOS

## MDL GMOS  & NAEFS Downscaled Forecast  

- 24-h GMOS
- Forecast

- For CONUS:
- NAEFS(1.45) : GMOS(1.72)
- 15% impr. over GMOS

- MDL GMOS  & NAEFS Downscaled Forecast
- Mean Absolute Error w.r.t. RTMA  Average For Sept. 2007

- 24-h NAEFS Forecast

## RTMA Analysis

- _GMOS forecast_

- _NAEFS products_

- Surface Temperature MAE
- CONUS, Sept. 2007
-  12Z NDFD vs. 00Z MOS/_GMOS_/_NAEFS_

- Surface Temperature MAE
- CONUS, Sept. 2007
-  00Z _GMOS_ vs. 00Z _NAEFS _

- METAR  obs. 1221 sites

**0.5°C**

**0.5°C**

## RTMA Analysis

- CONUS, Sept. 2007
-  12Z NDFD vs. 00Z MOS/_GMOS_/_NAEFS_

- Surface Temperature Pointwise Bias
- CONUS, Sept. 2007
-  00Z _GMOS_ vs. 00Z _NAEFS_

- _GMOS forecast_

- _NAEFS products_

- METAR  obs. 1221 sites

- Pointwise Bias

- Area Mean Bias

**0.6°C**

**0.6°C**

##  Western Rgn Pointwise Bias

- _ __Western Rgn Pointwise Bias_

- _ __Western Rgn Area Mean Bias_

- _ __Western Rgn MAE_

- _ __Western Rgn MAE_

##  Central Rgn Pointwise Bias

- _ __Central Rgn Pointwise Bias_

- _ __Central Rgn Area Mean Bias_

- _ __Central Rgn MAE_

- _ __Central Rgn MAE_

##  Eastern Rgn MAE

- _ __Eastern Rgn MAE_

- _ __Eastern Rgn MAE_

- _Eastern Rgn Pointwise Bias_

- _ __Eastern Rgn Area Mean Bias_

##  Southern Rgn MAE

- _ __Southern Rgn MAE_

- _ __Southern Rgn MAE_

- _ __Southern Rgn Pointwise Bias_

- _ __Southern Rgn Area Mean Bias_

## BACK

*NCEP-raw*

*NAEFS-final*

*NCEP-raw-downs*

***NCEP-biasc-downs***

***BACK***

*Bo et al.*

## BACKGROUND

## NFUSE GOALS

- Premise
- We accept and plan to follow major NRC recommendations
- What’s needed?
- Design, develop, and gradually implement changes in forecast process for
- 				*assessing, communicating, and using forecast uncertainty*
- Basic capabilities (by 2009-2010)
- Detailed draft plan (including low hanging fruit) by Oct 07
- Full capabilities (by 2012-2017)
- Conceptual draft plan by Oct 07
- Corporate commitment
- Funding for both phases
- Reallocation / reprioritization for phase 1
- PPBES for phase 2
- New / revised operational requirements
- Part of planning / development

**NFUSE GOALS**

## TWO-PHASE PLAN

- Limited time and resources - Build on existing and emerging efforts
- Not all users within/outside NWS can access ensemble data
- Limited capabilities
- Ability to express basic forecasts in probabilistic terms
- Uni-variate pdf
- Joint probab. computation assumes probabilities for different variables are independent
- Other formats derived from uni-variate pdf
- Phase 2 (5-10 yrs)
- Full capabilities
- All interested users within/outside NWS have access to ensemble data
- Joint probabilities derived from ensemble
- Exhaustive set of forecast formats/products supported

**TWO-PHASE PLAN**

## BACKGROUND

## LINKS WITH NFUSE PLANS

**Single value format**
- Short-term (2-3 yrs) plan – **3 values format** (pdf)
- Provide best (bias corrected) numerical guidance in agreed upon format
- Provide bias corrected ensemble data to external and selected internal users
- Human forecasters modify numerical guidance using agreed upon format
- Distribute and store agreed upon format
- Convert format into single variable pdf format
- External users provided with products in format of their choice
- Multiple options
- Software needed to derive user requested format from single value pdf
- Long-term (5-10 yrs) plan – **ensemble format**
- Provide best numerical guidance in agreed upon format
- Human forecasters modify numerical guidance using agreed upon format
- Propagate information to modify bias corrected ensemble data
- Modified bias corrected ensemble data is complete and final forecast dataset includes
- Uncertainty information regarding spatial, temporal, cross-variable co-variances
- Human forecasters need access to ensemble data to assist in
- Manipulation (added value)
- Interpretation (user outreach)
- External users provided with products in format of their choice
- Multiple options
- Software needed to derive user requested format from ensemble data

**LINKS WITH NFUSE PLANS**

## Slide 43

## SYNERGIES

- Verification using common framework possible
- Added value can be traced
- Choice of THORPEX performance measures
- Related to major application areas (previous discussion)
- Use 3-value format to verify uncertainty/probabilistic information?

**SYNERGIES**

## INTRODUCTION

- Probabilistic numerical guidance for high impact events (THORPEX)
- Provides objective numerical guidance
- Human forecasters
- Add value to numerical guidance
- Interpret guidance / products
- Products to be distributed to and used by external community
- Interface with “Weather enterprise”
- Format of forecast uncertainty estimate can/must link three major areas
- Numerical methods
- Must be able to produce chosen format
- Forecasters
- Must be able to manipulate guidance
- Must be simple and intuitive
- Users
- Must find clear value in product given in chosen format
- Choice of exact format not critical
- As long as software available to generate alternate formats

**INTRODUCTION**

## PROPOSED NEW / LEGACY PRODUCTS TO BE MONITORED

- Possible new probabilistic guidance products for high impact events
- Hydrometeorology
- Extreme hydro-meteorological events, incl. dry and wet spells (CONUS)
- Exceedance of 1 & 5 inch thresholds
- Exceedance of 6-hr flash flood guidance (or numerical outlook) thresholds
- Quantitative extreme river flow forecasting (OCONUS)
- Tropical / winter storm prediction
- Extreme surface wind speed
- Radii of gale, storm, hurricane force wind
- Maximum surface wind (for intensity)
- Extreme precipitation (related to wet spells)
- Storm surges
- Storm surge values along US Coast
- Aviation forecasting
- Flight restriction
- Icing, visibility, fog, clear air turbulence
- Health and public safety
- Hot and cold spells
- “Legacy” internal probabilistic scores to assess long-term progress
- With years of existing archive and/or
- Can be easily recomputed
- General circulation
- Probabilistic 1000 & 500 hPa height forecasts
- Tropical storm
- Strike probability for track
- Probability of intensity (central pressure or wind-based)

## BACKGROUND

## NFUSE PRIORITIES?

- NRC Recommendations
- Do we agree on major recommendations?
- If not, let’s discuss any issues
- If so, can we accept those as the basis/guidelines for our work
- 3rd Ensemble User Workshop Recommendations
- Three major recommendations in response to NRC Report
- High level roadmap guiding our collaboration (Sept 2007)
- New operational requirements (March 2008)
- Design, develop, and gradually implement new forecast process
- Basic capabilities (2009-2010)
- Full capabilities (2012-2017)
- Nine specific recommendations on design, development, implementation
- Work areas identified, links to be worked out
- Continue development of expanded forecast process
- Bias correction & downscaling methods for ensembles
- Define summary ensemble information to be used (ST)
- Contribute to establishment of NOAA-wide environmental data depository
- Develop ensemble interrogation, modification, & product generation tools
- Develop telecommunication facilities to access data
- Develop unified NWS/NOAA probabilistic verification package (ST)
- Develop & implement comprehensive training
- Develop outreach program on use & communication of uncertainty
***Do we need to modify/augment etc these recommendations?***

**NFUSE PRIORITIES?**

## COMPLETING THE FORECAST:ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY

- Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- NRC Report; Louis Uccellini, Steve Lord & Ensemble Team
- http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

## OUTLINE

- REVIEW OF 3RD ENSEMLBE USER WORKSHOP
- MAJOR RECOMMENDATIONS REGARDING NRC REPORT
- WHAT, WHY, AND HOW TO CHANGE?
- SPECIFIC RECOMMENDATIONS

## 3rd ENSEMBLE USER WORKSHOP

- Oct. 30 – Nov. 1 2006, Laurel, MD
- Close to 100 participants
- NWS Regions (12), Headquarters (15), NCEP (37)
- OAR (6), other government (4), private (4), academic (8) sectors & international (8)
- For further info, see: _[http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html](http://wwwt.emc.ncep.noaa.gov/gmb/ens/UserWkshop_Oct2006.html)_
- Topics
- Assessing & propagating uncertainty throughout entire forecast process
- From observations to users
- Working group discussions
- Ensemble configuration
- Statistical post-processing
- Data depository / Interrogation / Product generation/dissemination / Verification
- User support / Outreach / Training
- Outcome
- Enthusiastic discussions
- Convergence on number of topics
- Open questions needing further research identified
- Strong support for sustained effort/engagement, annual meetings, etc
- Recommendations
- Course of action in response to NRC report
- Presented here first – will solicit and incorporate feedback from participants

**3****rd**** ENSEMBLE USER WORKSHOP**

## MAJOR RECOMMENDATIONS - 1

**All major recommendations embraced**
- Develop roadmap for assessing & communicating forecast uncertainty
- Based on science, technology, workforce considerations
- Consistent with NOAA’s mission, NWS plans, embraced by “Enterprise”
- Define end goal, 5-10 years horizon
- Adopt ensemble-based forecast process
- Maximize forecast skill
- Unify scientific, software, and technological infrastructure across NOAA
- Weather, water, climate applications
- Form high level planning and oversight team – March 2007
- Each NWS office to delegate one representative
- Team reports to Corporate Board
- Name programmatic and technical leads
- Develop roadmap – Sept. 2007

**MAJOR RECOMMENDATIONS - 1**

## MAJOR RECOMMENDATIONS - 2

- Revise operational requirements to make them probabilistic
- Make probabilistic format the primary requirement
- For each forecast application:
- Replace single value/categorical format with probabilistic format as primary requirement
- Revise/supplement corresponding performance measures (GPRA)
- Essential for orderly transition from traditional to new forecast process
- NWS is requirement-driven organization
- Without clear new requirements, process is doomed
- Phased implementation schedule in consultation with responsible offices
- Allows orderly transition to new requirements
- New requirements prepared by Planning & oversight team (or its designate)
- Assisted by responsible NWS office
- Presented to Corporate Board for approval – By March 2008

**MAJOR RECOMMENDATIONS - 2**

## MAJOR RECOMMENDATIONS - 3

- Revise operational requirements to make them probabilistic (1 year)
- Design, develop, and gradually implement new forecast process
- Focus on missing scientific, technological, and human components
- Identify self-contained components with
- Clear requirements and interfaces between components
- Define basic capabilities achievable in 2-3 years
- Limited but consistent with and leading to end goal
- Full capabilities in 5-10 years
- Interface with NOAA **THORPEX** program
- Research/development to improve skill & utility of probabilistic forecasts
- Leverage related major NOAA, national, and international efforts
- Integrate with NWS, NOAA, national activities
- New NOAA **CONOPS** process
- NOAA-wide regional service plans
**NUOPC** planning/development
- Provide long-term funding support through **PPBES**
- Overlap with W&W High Impact Event Theme Team
- Form **development teams** for specific tasks/components following workshp
- Identify potential contributors within & outside of NWS & NOAA

**MAJOR RECOMMENDATIONS - 3**

## PROPOSED CHANGE

- Incorporate assessment and communication of uncertainty in forecast process
- Is it a **major change in course** of “Weather Ship”?
- Ie, abandon course of ever improving single forecast scenario (expected value)?
- No – Expand, not abandon
- Keep improving fidelity of forecasts, PLUS
- Add new dimension
- Capture other possible scenarios – ensemble forecasting
- Use a flotilla, instead of one ship, in exploring nature
- Existing activities are subset of expanded forecast process
- Single value forecast is expected value of full probability distribution
- Can keep serving forecasts in old format to users who prefer that

**PROPOSED CHANGE**

## Single forecast (driven by GFS winds) example for drifting virtual ice floe

- Single forecast (driven by GFS winds) example for drifting virtual ice floe

*Bob Grumbine, EMC*

- Initial position

- 7 September 2006

## Ensemble forecast for drifting ice floe for same case

**Ensemble forecast** for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## Most likely forecast for drifting ice floe for same case

- Most likely forecast for drifting ice floe for same case

*Bob Grumbine, EMC*

- Initial position

## WHY CHANGE IS NEEDED?

- They admittedly want minimal or no uncertainty in forecasts
- Distinction between no uncertainty in the forecast, vs. not talking about it
- Forecast uncertainty cannot be arbitrarily reduced
- Despite major ongoing & continuing efforts, they persist forever
- Chaotic nature of atmosphere - land surface – ocean coupled system + initial/model errors
- Level of uncertainty is determined by nature and level of sophistication in forecast system
- Forecast uncertainty can be ignored though
- Negative consequence on informed users
- Not able to prepare for all possible outcomes
- Assumes a certain scenario and remains vulnerable to others
- Possibly serious loss in social/economic value of forecast information
- Why forecasters (should) care about forecast uncertainty?
- Imperfect forecasts are consistent w. observations (reliable) only if in prob format
- If in other format, must be brought into probabilistic format through
- Verification / bias correction

**WHY CHANGE IS NEEDED?**

## ADVANTAGES OF PROBABILISTIC FORMAT

**Old paradigm**
- Convoluted forecaster-user decision process
- User expects forecaster to make decision for them in presence of uncertainty
- “Will it rain?” – “80%” – “But tell me, will it rain?”
**New paradigm**
- Forecaster and user decision processes enhanced and better linked
- Allows forecasters to capture all knowledge about future conditions
- Provision of information related to multiple decision levels in probabilistic format critical
- Provider helps interpret probabilistic info & and modify user decision process if needed
- Option to continue providing single value or other limited info until user ready
- Allows users to decide about most beneficial course of action given all possibilities
- Proper use of probability or other uncertainty information needed - Training
- User requests critical weather forecast info depending on their sensitivity

**ADVANTAGES OF PROBABILISTIC FORMAT**

## TRADITIONAL FORECAST PROCESS

- Reducing uncertainty in single forecast is main emphasis
- Loss of accuracy in forecast estimate of expected value of distribution
- Mean of ensemble cloud provides better estimate
- Ignores or simplifies forecast uncertainty
- Uncertainty assessed as statistically averaged error in single fcst (second thought)
- Ensemble cloud provides better estimate of case dependent variations in uncertainty
- Use of single value / categorical forecast format
- Difficulty in formulating/communicating plausible alternate scenarios
- Ensemble member forecasts can directly feed into Decision Support Systems
- One-way flow of information from observations to users
- Not adaptable to case dependent user requirements
- Ensemble can propagate back user requirements to adaptive
- Observing, assimilation, modeling/ensemble, post-processing and application components
- Applications in planning and execution of new CONOPS in high impact events

**TRADITIONAL FORECAST PROCESS**

## PROPAGATING FORECAST UNCERTAINTY

- z

*Single value*

*Distribution*

- Ensemble Forecasting:
- Central role – bringing the pieces together

**PROPAGATING FORECAST UNCERTAINTY**

## HOW CAN IT BE DONE? NEW PARADIGM

- Expand forecasting with new dimension of uncertainty
- Multiple scenarios (in place of single scenario)
- Provides best forecast estimate for both expected value (as before) and uncertainty (new)
- Unified scientific, technological, human approach
- Sharing resources across NWS & NOAA
- Ensemble is centerpiece both symbolically and figuratively in forecast process
- Ensembles act as a glue & two-way information channel
- Observing system, data assimilation, numerical modeling
- ENSEMBLES
- Statistical post-processing, product generation, decision making
- Design, develop, & implement missing components of new forecast process
- Gradual, measured steps
- Basic capability - Short-term, 2-3 yrs, leading to
- Full implementation - Long-term, 5-10 yrs

**HOW CAN IT BE DONE? NEW PARADIGM**

## ENSEMBLES AND THE RESEARCH COMMUNITY

**SOCIOECON.**

**ENSEMBLES AND THE RESEARCH COMMUNITY**
- LINKED THROUGH THORPEX – MAJOR INTERNATIONAL RESEARCH PROGRAM
***GOAL: Accelerate improvements of high impact weather forecasts***

**SYSTEM**

- CLIMATE FORECASTING / CTB

***INTEGRATED**** DATA ASSIMILATION & FORECASTING*

***Days 15-60***

- NWS OPERATIONS

- GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS)

***GLOBAL OPERATIONAL***

***TEST CENTER***

***GLOBAL OPERATIONAL***

***TEST CENTER***

***MODEL ****ERRORS & HIGH IMPACT MODELING*

## SPECIFIC RECOMMENDATIONS – 1-3

- Focus on adaptive methods applicable for high impact events
- Collection & use of observations (targeted observations)
- Data assimilation (case dependent background error estimation)
- Numerical modeling (adaptive resolution & high impact modeling)
- Ensemble forecasting (case dependent variations in membership & composition)
- Decision support systems (flexible user actions depending on forecast probabilities)
- Bias correction & downscaling methods for ensembles
- Estimate/correct lead-time dependent bias in ensemble forecasts (on model grid)
- Generate fine resolution (NDFD grid) uncertainty/ensemble data
- Establish connection between (bias corrected) coarse model vs. fine user grids
- Use reanalysis & hind-casts with operational systems as needed
- Define summary ensemble information to be used to (ST)
- Collapse vast amount of ensemble data for inclusion in expanded NDFD/NDGD
- E.g., 10, 50, and 90 percentile of forecast distribution (in place of single value)
- Manually inspect/modify summary ensemble information

**SPECIFIC RECOMMENDATIONS – 1-3**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS – 4-6

- Expand NDFD/NDGD database to include forecast uncertainty (ST)
- Develop capability to hold all ensemble trajectories (LT)
- All members, variables, lead times
- Develop ensemble interrogation, modification, & product generation tools to
- Derive summary information from ensemble (ST)
- Manually modify summary ensemble info (ST)
- Derive additional statistics from summary info (product generation, ST)
- Automatically modify ensemble trajectories based on modified summary info (LT)
- Derive any info from full ensemble data (product generation, LT)
- Develop telecommunication facilities to access data
- Summary info & limited derived products (ST)
- All ensemble forecasts & derived products (LT)

**SPECIFIC RECOMMENDATIONS – 4-6**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*

## SPECIFIC RECOMMENDATIONS – 7-9

- Assess statistical reliability and resolution for
- Computing official performance measures
- Evaluating value added along forecast chain
- Assessing value in newly developed vs. operational techniques
- Develop & implement comprehensive training to
- Prepare all participants for their new roles in expanded forecast process, incl.
- Statistical background
- Ensemble methods
- Best forecast practices in assessing uncertainty
- Applications of probabilistic & other uncertainty information
- Develop outreach program on use & communication of uncertainty
- In partnership with entire Weather, Water & Climate Enterprise
- Determine best ways of communicating uncertainty
- Compile sample of Decision Support Systems using uncertainty information
- Establish close partnership with public sector users (e.g., emergency, water management)
- User feedback on new activities
- Explore how forecast process can be adapted according to user requirements

**SPECIFIC RECOMMENDATIONS – 7-9**

*ST – Short Term (2-3 yrs); **	**LT – Long Term (5-10 yrs)*