# Presentation: 251765

## 2ND ENSEMBLE USER WORKSHOPMay 18-20 2004, NCEP  DRAFT RECOMMENDATIONS Based on presentations and working group discussionsJune 1 2004

## WORKING GROUP PARTICIPANTS (26)

**CONFIGURATION**
*Co-leaders:**** ***Jun Du and Mozheng Wei
*Participants: *Rick Knabb, Richard Wobus, Ed O’Lenic, Dingchen Hou

**STATISTICAL POST-PROCESSING**
*Co-leaders:* Paul Dallavalle & Zoltan Toth
*Participants: *Keith Brill, Andy Lough, DJ Seo, David Unger

**DATA ACCESS**
*Co-leaders: *Yuejian Zhu and David Michaud
*Participants: David Bright, Minh Nguy, Kathryn Hughes*

**PRODUCTS & TRAINING**
*Co-leaders:* Jeff McQueen and Pete Manousos
*Participants: Paul Stokols, Fred Mosher, Paul Janish, Linnae Neyman, Bill Bua, Joe Sienkiewicz, Binbin Zhou*

***ADDITIONAL WORKSHOP PARTICIPANTS (15)***
- Steve Tracton, Mike Halpert, Brian Gockel, Brent Gordon, Mark Antolik, Barbara Strunder, Andrew Loughe, Michael Graf, Dave Plummer, Steve Schotz, Jon Mittelstadt, Malaquias Pena, Glen Zolph, Steve Lord, David Caldwell

## 2nd NCEP Ensemble User Workshop SUMMARY RECOMMENDATIONS

**OVERALL** - 	Enhance coordination of ensemble-related efforts
- Establish ensemble product working group
- Continue with monthly Predictability meetings
- Hold Ensemble User Workshops (part of reestablished SOO workshops)
**CONFIGURATION**
*	****Global ensemble:****	**	*Implement hurricane relocation for perturbed initial conditions
- 				Continue efforts to build multi-center ensemble
*	****Regional (SREF) ensemble:****	*Ensemble run should be coupled closer with hires control (same initial time)
- 				Run 4 cycles per day
**DATA ACCESS**
- Provide access to all ensemble data (including members)
- Facilitate user controlled access to data (e.g. NOMAD, on demand, not on rigid schedule)
**STATISTICAL POST-PROCESSING (BIAS CORRECTION)**
- Develop techniques for two-stage statistical post-processing
- Operationally implement post-processing techniques
**PRODUCTS**
- Develop a software toolbox for interrogating ensemble data
- Establish central/local operational product generation suites
**VERIFICATION **
- Design & develop unified and modular ensemble/probabilistic verification framework
**TRAINING**
- Establish NWS formal ensemble training requirements
- Contribute to Ensemble Training Workshops, international activities (AMS, WMO), etc

## OVERALL ISSUES / RECOMMENDATIONS

**Enhance coordination of ensemble-related efforts**
- Among NCO Service Center users
- Between users and NCO / EMC developers
- Between global and regional ensemble groups within EMC
- Share research, development, and operational procedures where possible
*Establish NCO / EMC / Service Centers Ens. Products Working Group*
*Continue (expand via telecom?) monthly Predictability Meetings*
**Optimize NCO operational job stream with user input**
- For improved integrated forecast decision support
*Periodically reevaluate job stream from user and science perspectives*
**Reestablish Annual NCEP SOO Workshop**
- Rotate focus of workshop among various topics
*Hold Ensemble User Workshop every 3-4 years*

## ENSEMBLE CONFIGURATION - CURRENT STATUS

***CONFIGURATION******	******Global******	******	******	******Regional***
- Cycles per day		4			2
- Membership		10			15
- Resolution		T126L28 till 7.5 days 	48km
- 				T62L28 beyond
- Model version(s)		Single GFS		ETA (2 conv. schemes), RSM
- Initial perturbation		Breeding		Breeding
- Boundary perturbations	N/A			Global ensemble
***ISSUES  -******  ***
*COLLABORATIVE PROJECTS MUST ENABLE OPERATIONAL IMPLEMENTATIONS*
**Global**
- North American Ensemble Forecast System –  with Met. Service of Canada
- Post-processing & product development – Aimed at operational applications
- THORPEX – NOAA, NA, & international collaborators
- Projects on initial and model related perturbations – Path to operations
**Regional**
- 	Northeast Energy Project – OAR & Industry collaborators
- Heat wave forecast related research – Should transition into operations
- WRF – FSL, NCAR, USAF and other collaborators
- Potential for rapid development of next generation operational system

## ENSEMBLE CONFIGURATION - RECOMMENDATIONS

**Global ensemble**
- Implement hurricane relocation for perturbed initial conditions
- Experiment with techniques used successfully with GFS system
- Continue efforts to build multi-center ensemble
- Combine NCEP, ECMWF, MSC, JMA, FNMOC ensembles
- Best possible multi-model approach (with added benefits of initial condition variability)
**Regional ensemble (SREF)**
- Run ensemble & hires ETA (WRF) control from same initial time
- Utility of off-cycle ensemble (9 & 21Z) is limited when used with 12Z & 00Z controls
- Differences between ensemble & hires control from different cycles hard to interpret
- Closer coupling between ensemble & hires control allows proper interpretation of both
- Alternative suggestions for computer resource allocation:
- Increase less the resolution in future implementation for both ensemble & hires control
- Run ~5 initial/model perturbation members along with hires control, finish rest of ensemble later
- Run 5 members from early, hires from final analysis, finish ensemble, run hires window for dominant clusters
- Introduce 4 cycles per day – Run ensemble at 00, 06, 12, & 18Z
- Will allow comparison of hires control and lores ensemble, enhancing utility of both
**Additional suggestions for both systems**
**Membership **
- Evaluate effect of increased membership in combination with post-processing gains
**Spatial resolution** – *As computational resources increase*,
- Increase ensemble resolution (~50-50% resources for hires control & lores ensemble)
**Initial perturbations**
- Continue research aimed at better quantification of initial uncertainty
**Model error representation**
- Continue research on stochastic model perturbations & model diversity

## ENSEMBLE DATA ACCESS - CURRENT STATUS

**Global ensemble** – 1x1 grid, pgrib, enspost, Sager file types
- NCEP Service Centers	All data available - Limited NAWIPS access
- AWIPS			Limited data out to 84 hrs – need 180 hrs (WAFS?)
- NCEP ftp servers		All data available
- NWS server		2 cycles only – Need to add 06 & 18Z cycles?
**Regional ensemble** (SREF) – GRIB212 (40km)
- NCEP SCs		All data available – Limited NAWIPS access
- AWIPS			No access to data – Need selected variables
- NCEP ftp servers		Selected variables only – All data needed?
- NWS server		None – Need to post data
**Issues:**
- Disc space usage – Inefficient due to use of multiple file formats
- Same data packaged in various formats for convenient access and historical reasons
- Bandwidth limitations – Ftp overload due to data access limited to prepared files
- Typical user needs only fraction of downloaded data
- Increase in data volume – Need advance planning to facilitate future data access
- Additional ensembles; Increased resolution, membership

## ENSEMBLE DATA ACCESS - RECOMMENDATIONS

- Provide access to all ensemble data (including members)
- Allows optimal use of ensemble information by diverse user base
- Should be feasible given low cost of disc storage space
- Lower resolution ensemble has similar data volume to hires control
- Temporary disc space limitations should be mitigated by
-  Freezing output resolution (or list of available variables)
- Facilitate user controlled access to data (e. g., NOMAD)
- Allow users to choose what they want to download by
- Selecting members, variable, level, time and spatial domain of interest
- Providing basic functionalities to manipulate data (eg, download derived statistics only – see Products Working Group recommendations)
- Consider for NAWIPS, AWIPS, and ftp dissemination
- Eliminates need for duplicate data files
- Significantly reduces bandwidth requirements
- Prototype system exists (NOMAD, all global ensemble data available)
- As interim solution until system operational, introduce split pgrib files?
- Shift to use of GRIB2 format
- WMO sanctioned uniform format for ensemble data
- Need for international ensemble data exchange (see Configuration WG)
- x3 to x5 reduction in file size

## ENSEMBLE STATISTICAL POSTPROCESSING - CURRENT STATUS

**NWP models, ensemble formation are imperfect**
- Known model/ensemble problems addressed at their source
- No “perfect” solution exists, or is expected to emerge
- Systematic errors remain and cause biases in
- 1st, 2nd moments of ensemble distribution
- Spatio-temporal variations in 2nd moment
- Tails of distributions
**No comprehensive operational post-processing in place**
- MOS applied on individual members (global ensemble, MDL)
- QPF calibration of 1st moment (global ensemble, EMC & CPC)
- Week 2 calibration with frozen system (global ensemble, CDC)
**Issues:**
- Users need bias-free ensemble guidance products
- Bias-corrected ensemble members must be consistent with verification data
- Algorithms must be relatively cheap & flexible for operational applications
- Post-process on model grid first, then “downscale” to NDFD grid / observs?
- Level of “correctible” details depends on
- Bias signal vs. random error noise ratio
- Sample size of available forecast/observation training data pairs
- Relatively small sample for short-med. ranges – Capture regime dependent bias?
- Much larger for extended ranges – Capture climatological bias via frozen system?

## Develop techniques for two-stage statistical post-processing:
1) Assess and mitigate biases on model grid with respect to analysis fields
Feedback to model / ensemble development
1st moment correction based on: Time mean error; Cumulative distributions 
2nd moment correction based on: Time mean ratio of ens mean error & spread
Post-processed forecasts bias corrected with respect to reanalysis fields
Generate anomaly forecasts using global/regional reanalysis climatology
2) Downscale bias-corrected fcsts from model grid to NDFD/observatn locations
“Smart” interpolator for bias correction and variance generation on fine scales
Multiple regression (MOS); Bayesian methods; Kalman Filtering; Neural nets
Apply downscaling methods on bias-corrected fields (no lead time dependence)
Use large reanalysis and corresponding observational data base (&/or NDFD analysis fields)
To describe ensemble-based pdf forecasts, use 3-parameter distributions
Test two methods, find best fitting analytic distribution (out of ~25 candidates)
Simple method:	Fit actual ensemble data
Kernel approach: 	Find best fit to climate data, then apply it on each member w/weight
Operationally implement post-processing techniques
Apply basic bias-correction techniques centrally (NCO) to serve wide user base
Post-process all variables used from the ensemble (first model, then derived variables)
Disseminate bias-corrected forecasts on lowres ensemble model grid
Save disc and bandwidth resources
Keep raw forecast fields also accessible for special user processing needs
Use additional post-processing (if any) locally to address special needs, eg:
Hurricane forecasting

- 1) Assess and mitigate biases on model grid with respect to analysis fields
- Feedback to model / ensemble development
- 1st moment correction based on: Time mean error; Cumulative distributions
- 2nd moment correction based on: Time mean ratio of ens mean error & spread
- Post-processed forecasts bias corrected with respect to reanalysis fields
- Generate anomaly forecasts using global/regional reanalysis climatology
- 2) Downscale bias-corrected fcsts from model grid to NDFD/observatn locations
- “Smart” interpolator for bias correction and variance generation on fine scales
- Multiple regression (MOS); Bayesian methods; Kalman Filtering; Neural nets
- Apply downscaling methods on bias-corrected fields (no lead time dependence)
- Use large reanalysis and corresponding observational data base (&/or NDFD analysis fields)
- To describe ensemble-based pdf forecasts, use 3-parameter distributions
- Test two methods, find best fitting analytic distribution (out of ~25 candidates)
- Simple method:	Fit actual ensemble data
- Kernel approach: 	Find best fit to climate data, then apply it on each member w/weight
- Operationally implement post-processing techniques
- Apply basic bias-correction techniques centrally (NCO) to serve wide user base
- Post-process all variables used from the ensemble (first model, then derived variables)
- Disseminate bias-corrected forecasts on lowres ensemble model grid
- Save disc and bandwidth resources
- Keep raw forecast fields also accessible for special user processing needs
- Use additional post-processing (if any) locally to address special needs, eg:
- Hurricane forecasting

**ENSEMBLE STATISTICAL POSTPROCESSING - RECOMMENDATION**

## ENSEMBLE PRODUCTS - CURRENT STATUS

**Product development software**
- Some functionalities exist
- Scattered around different developers/platforms/users
- NCO operations
- NAWIPS official build
- NAWIPS development by NCEP SOOs
- AWIPS
- Other platforms
**Products generated centrally by**
- NCO	Limited number of gridded products (operational)
- EMC	Additional set of gridded and web-based products (non-operational)
**Issues:**
- Lack of standard/common software toolbox for ensembles
- Missing functionalities
- Multiple software versions of existing functionalities
- Duplication of efforts
- Lack of comprehensive, well designed set of products
- Non-standard set of products/displays (global vs. regional ensembles, etc)
- NAWIPS, AWIPS requires access to products (web not enough)
- Need for operationally generated and supported web product suite

## ENSEMBLE PRODUCTS - RECOMMENDATIONS

- Develop a software toolbox for interrogating ensemble data
- Establish development team - NCO, EMC, NCEP Service Center experts
- Compile list of required functionalities – See attached list
- Develop standard software package (subroutines) for each functionality
- Work in NAWIPS framework
- Ensure software (subroutines) are portable to different platforms
- Ensure batch and on demand processing capabilities
- Provide interactive processing/display capability where needed
- Offer subroutines for use by AWIPS and broader inter/national community
- Consider WRF, NAEFS, THORPEX applications
- Establish operational/local product generation suites
- Use standard software toolbox for product generation
- Identify list of products – See template on next page
- Type of product generation based on typical usage:
- Every day	- Generate centrally (NCO), produce multiple file formats
- Occasionally	- On demand (NCEP Service Centers)
- Interactively	- On screen manipulation (NAWIPS)
- Distribute centrally generated products within NAWIPS, AWIPS
- Set up and maintain operational NCEP ensemble product web page
- Post products on web page for use by broader community
- Provide limited interactive query tools if desired (example within NOMAD)

## ENSEMBLE PRODUCTS - FUNCTIONALITIES

|  | FUNCTIONALITY | CENTRALLY GENERATED | LOCALLY GENERATED |
| --- | --- | --- | --- |
| 1 | Mean of selected members |  |  |
| 2 | Spread of selected members |  |  |
| 3 | Median of selected values |  |  |
| 4 | Lowest value in selected members |  |  |
| 5 | Highest value in selected members |  |  |
| 6 | Range between lowest and highest values |  |  |
| 7 | Univariate exceedance probabilities for a selectable threshold value  |  |  |
| 8 | Multivariate (up to 5) exceedance probabilities for a selectable threshold value  |  |  |
| 9 | Forecast value associated with selected univariate percentile value  |  |  |
| 10 | Tracking center of maxima or minima in a gridded field (eg – low pressure centers)  |  |  |
| 11 | Objective grouping of members |  |  |
| 12 | Plot Frequency / Fitted probability density function at selected location/time (lower priority)  |  |  |
| 13 | Plot Frequency / Fitted probability density as a function of forecast lead time, at selected location (lower priority)  |  |  |

- For each functionality, NCEP Service Centers provide list of variables/levels for which central/local generation of products is needed: *MSLP, Z,T,U,V,RH, etc, at 925,850,700,500, 400, 300, 250, 100, etc hPa*

## ENSEMBLE VERIFICATION – CURRENT STATUSFor lack of time, this topic was not discussed at the workshop

- Global ensemble verification package used since 1995
- Comprehensive verification stats computed against analysis fields
- Inter-comparison with other NWP centers
- Regional (SREF) verification package
- Basic measures computed routinely since 1998
- Probabilistic measures being developed independently from global ensemble
- Issues
- Need to unify computation of global – regional ensemble verification measures
- Unified framework must facilitate wide-scale national/international collaboration:
- North American Ensemble Forecast System (collaboration with Met. Service Canada)
- THORPEX International Research Program
- WRF meso-scale ensemble developmental and operational activities
-  Facilitate wider community input in further development/enhancements
- How to establish basis for collaboration with NCAR, statistical community, etc

## ENSEMBLE VERIFICATION - RECOMMENDATIONS

- Design unified and modular ensemble/probabilistic verification framework
- Data handling/storage
- Use standard WMO file formats as ensemble data input
- Allow non-standardized user/site specific procedures
- Computation of statistics
- Establish required software functionalities (scripts) and verification statistics (codes)
- Jointly develop and share scripts/subroutines with standard input/output fields
- Improvements to common infrastructure benefit all
- Comparable scientific results, independent of investigators
- Access/display of output statistics
- Explore if standard output file format(s) feasible? Use text or FVSB-type files?
- Develop/adapt display software for interactive interrogation of output statistics
- Examples: FVS display system; FSL approach to WRF verification
- Develop and implement new verification framework
- Utilize existing software and infrastructure where possible
- Direct all internal ensemble-related verification efforts toward new framework
- Share work with interested collaborators
- Meteorological Service of Canada (subroutines, L. Wilson and colleagues)
- FSL (display tools, A. Laugh)
- Make new software available to national/international community
- Coordinate further development with wider community (WMO, etc input)

## ENSEMBLE VERIFICATION – DESIGN SPECIFICATIONS

**Compute statistics selected from list of available**
- Point-wise measures, including:
- RMS, PAC for individual members, mean, median
- Measures of reliability (Talagrand, spread vs. error, reliability components of Brier, RPSS, etc)
- Measures of resolution (ROC, info content, resol. comps. of BSS, RPSS, potential econ.value, etc)
- Combined measures of reliability/resolution (BSS, RPSS, etc)
- Multivariate statistics (e.g., PECA, etc)
- Variables & lead times –make all available that are used from ensemble
**Aggregate statistics as chosen in time and space**
- Select time periods
- Select spatial domain (pre-designed or user specified areas)
**Verify against observational data or analysis fields**
- Scripts running verification codes should handle verification data issues
- Use same subroutines to compute statistics in either case
- Account for effect of observational/analysis uncertainty?
**Define forecast/verification events by either**
- Observed/analyzed climatology, e.g., 10 percentile thresholds in climate distribution
- Automatically compute thresholds for forecast values
- User specified thresholds – automatically compute corresponding climate percentiles
- Ensemble members (like in Talagrand stats) – compute climate percentiles
**Facilitate the use of benchmarks**:
- Climatology, persistence, or specified prior forecast data set
**Prioritize and find balance between **
- Flexibility vs. complexity; operational vs. research use, etc

## ENSEMBLE TRAINING

- CURRENT STATUS:
- NCEP Training Material available since 2002 (P. Manousos)
- Webcast (D. Bright, add detail)
- COMET professional training module to be released soon (B. Bua)
- Teletraining (provide detail)
- ISSUES:
- Official NWS training opportunities/requirements not established
- Training for professional national, international, and lay users needed
-  Share training resources nationally/internationally
- RECOMMENDATIONS:
- Establish NWS formal ensemble training requirements
- Consider organizing AMS Ensemble Training Workshops
- Practicing broadcast etc meteorologists
- Emergency managers
- Share training material on national/international level
- Establish NWS OS FAQ on ensembles
- NCEP/HPC International Desk – Spanish/Portugese translations of existing material
- Exchange ideas/material with WMO Ensemble Training initiative
- Contribute to Socio-Economic Applications part of intl. THORPEX research program