# Presentation: 196441

## Northern American Ensemble Forecast System (NAEFS)-Bias Correction

- Yuejian Zhu,
-  Bo Cui and Zoltan Toth
- Environmental Modeling Center
- NOAA/NWS/NCEP
- Acknowledgements:
- DingChen Hou  EMC

## NAEFS Background Information

- First of a kind project
- Operational multi-center ensemble system
- Bias correction, climate percentiles never computed on such a scale operationally
- Timetable
- Mar 2003	Project started
- Oct 2003	Draft Research, Development and Implementation Plan
- Sep 2004	Initial Operational Capability – Operational data exchange
- May 2006	First Operational Implementation
- Mar 2007 NAEFS upgrade
- Challenges
- Developed joint plan with MSC personnel
- Arranged operational data exchange
- Coordinated GEFS development with international NAEFS developments
- Coordinated software development & operational implementation with MSC
- Worked with less THORPEX resources than planned originally
- Future expansion
- Develop sustainable plans
- Coordinate with partners
- Rename NAEFS and position it as prototype GIFS system
- Resource concerns
- Computational (telecommunication, disc, etc)
- Human

## First Implementation of NAEFS – Summary

- Bias corrected members of joint MSC-NCEP ensemble
- Decaying accumulated bias (~past 50 days) for each var. for each grid point
- For selected 35 of 50 NAEFS variables
- 32(00Z), 15(06Z), 32(12Z) and 15(18Z) joint ensemble members
- Bias correction against each center’s own operational analysis
- Weights for each member for creating joint ensemble (equal weights now – unequal weights to be added later)
- Weights don’t depend on the variables
- Weights depend on geographical location (low precision packing)
- Weights depend on the lead time
- Climate anomaly percentiles for each member
- Based on NCEP/NCAR 40-year reanalysis
- Used first 4 Fourier modes for daily mean,
- Estimated climate pdf distribution (standard deviation) from daily mean
- For selected 19 of 50 NAEFS variables
- 32(00Z), 15(06Z), 32(12Z) and 15(18Z) joint ensemble members
- Adjustment made to account for difference between oper. & re-analysis
- Provides basis for downscaling if local climatology available
- Non-dimensional unit

## Bias Correction Method & Application    

-  decaying averaging mean error = (1-*w*) * prior t.m.e + *w* * (f – a)

-   Bias Correction: application to NCEP operational ensemble 15 members

- Bias Correction Method & Application**   **

- 6.6%

- 3.3%

- 1.6%

*For separated cycles**, **each lead time and individual grid point, t.m.e = time mean error*

-  Test different decaying weights.
-    0.25%, 0.5%, 1%, 2%, 5% and
-    10%, respectively
-   Decide to use 2% (~ 50 days)
-    decaying accumulation bias
-    estimation

*Toth, Z., and Y. Zhu, 2001*

## List of Variables for Bias Correction, Weightsand Forecast Anomalies for CMC & NCEP Ensemble

## Summary of NAEFS First Implementation

- Period:
- 04/10/2006 – Current (NCO real time parallel)
- Maps comparison for bias (before and after)
- 500hPa height, 2m temperature
- Statistics for
- Bias reduction in percentage
- Height, temperature, winds
- RMS errors
- Probabilistic verifications (ROC)
- NH, SH and tropic
- Conclusions
- Bias reduced (approximately 50% at early lead time)
- RMS errors improved by 9% for d0-d3
- Probabilistic forecast
- Improved for all area, all lead time
- Typically for NH, 20-24 hours improvement from d7

## 500hPa height: 120 hours forecast (ini: 2006043000)

- Shaded:      left – raw bias                          right – bias after correction

## 2 meter temperature: 120 hours forecast (ini: 2006043000)

- Shaded:     left – raw bias                          right – bias after correction

## Bias Improvement (absolute value) after Bias correction

- Overall bias reduction:
- (globally)
- D0-3: 50%
- D3-8: 40%
- D8-15: 30%

- _500hPa height_

- _850hPa temperature_

- _Sea level pressure_

- _2m Temperature_

- There is daily variation after bias correction, more bias reduced for valid 12Z cycle

## Bias Improvement (absolute value) after Bias correction

- Overall bias reduction:
- (Tropic)
- D0-3: 50%
- D3-8: 45%
- D8-15: 40%

- _10m V-component_

- _2m temperature_

- _Sea level pressure_

- _10m U-component_

## Evaluation after bias correction (16 cases)

- _Northern Hemisphere_

- _Tropics_

- _Southern Hemisphere_

**Black-operational ensemble (10m)**
**Red-real time parallel ensemble (14m)**
**Green-real time parallel ensemble after**
**bias correction (14m)**

- _RMS errors for ensemble mean reduced for 48-h forecast (~9%)_

- _Probabilistic skill_
- _Extended 20-h for d-7_

## NAEFS verification

- Web-site:
- http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/opr/naefs.html
- Reference: NCEP/NCAR 40y reanalysis (next slide)
- Variables:
- 1000hPa, 500hPa heights, 850hPa, 2m temperature, 10m u and v
- Verified for ensemble mean:
- RMS errors, spread, mean error (bias) and absolute error
- Verified for ensemble distribution:
- Histogram (Talagrand)
- Verified for ensemble probabilistic forecast
- ROC, RPSS, CRPS, BSS (Resolution and Reliability), EV
- Regions:
- NH, SH, Tropical, Asia, Europe and Northern American
- Statistics from seasonal average

## Climatological Data

- NCEP/NCAR 40 years (1958-1997) reanalysis
- Monthly Sampling
- For example: 40*30=1200
- Generating10 equally-a-likely, based on monthly sampling
- Projected to verify date
- All forecast skills will base on 10 equally-a-likely climatological bins.

## Example of web-page setting: http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/opr/naefs.html

| 500 hPa Height Scores NCEP .vs NCEPb |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| SH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| TROP | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |

| 1000 hPa Height Scores (NCEP .vs NCEPb) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| SH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| TROP | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |

| 850 hPa Temperature Scores (NCEP .vs NCEPb) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| SH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| TROP | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |

| 2 Meters Temperature Scores (NCEP .vs NCEPb) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| SH | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |
| TROP | ROC | EV | RPSS | BSS | CRP | CRPS | RMS/SPRD | ERR/ABSE | HISTOGRAM |

## ISSUES ADDRESSED

- Effect of bias-correction
- Different variables
- Comparing of NCEP and CMC’s forecasts
- Before & after bias correction
- Impact of combined ensemble (NAEFS)
- Before & after bias correction
- Gains from bias correction + combination =
- NAEFS advantage

## HISTOGRAM

_**HISTOGRAM**_

- _1-day_

- _16-day_

- _12-day_

- _8-day_

- _5-day_

- _3-day_

## HISTOGRAM

_**HISTOGRAM**_

- _1-day_

- _16-day_

- _12-day_

- _8-day_

- _5-day_

- _3-day_

- _Good spread, but more biased_

## There is less room to improve the skill by bias-correction only 

- _RMSE and Spread_

- _Mean and absolute errors_

- _CRPSS_

- 10 meter wind (u-component)
- Less biased,
- There is less room to improve the skill by bias-correction only

## ISSUES ADDRESSED

- Effect of bias-correction
- Different variables
- Comparing of NCEP and CMC’s forecasts
- Before & after bias correction
- Impact of combined ensemble (NAEFS)
- Before & after bias correction
- Gains from bias correction + combination =
- NAEFS advantage

## Continuous Rank Probability Score 

- 0%

- 100%

- 50%

- p07

- p09

- p08

- p06

- p03

- p02

- p01

- p04

- p05

- p10

- _Obs (truth)_

_**Continuous Rank Probability Score**__** **_

- X

- Xo

- Heaviside Function H

- Order of 10 ensemble members (p01, p02,...,p10)

_*CRP Skill Score is*_

## Ranked Probabilistic Score

- Ranked (ordered) Probability Score (RPS) is to verify multi-category probability forecasts, to measure both reliability and resolution which based on climatologically equally likely bins

- and

- x

- _Verify Analysis_

- _Ensemble Forecast_

- OBS*              On*
- FCST PROB* Pn*

- 0

- 0

- 0

- 0

- 1

- 0

- 0

- 0

- 0

- 0

- 0%

- 20%

- 10%

- 0%

- 10%

- 30%

- 20%

- 0%

- 0%

- 10%

- i=1
- i=2
- i=3
- i=4
- i=5
- i=6
- i=7
- i=8
- i=9
- i=10 = **k : number of categories**

- _Example of 10 climatologically equally likely bins, 10 ensembles_

## 500hPa height

- _500hPa height_

- _1000hPa height_

- _850hPa temperature_

- _2 meter temperature_

- _Black-NCEP bias-corrected __Red-CMC bias-corrected__ __Green-NAEFS combined_

## 500hPa height

- _500hPa height_

- _1000hPa height_

- _850hPa temperature_

- _2 meter temperature_

- _Black-NCEP bias-corrected __Red-CMC bias-corrected__ __Green-NAEFS combined_

## ISSUES ADDRESSED

- Effect of bias-correction
- Different variables
- Comparing of NCEP and CMC’s forecasts
- Before & after bias correction
- Impact of combined ensemble (NAEFS)
- Before & after bias correction
- Gains from bias correction + combination =
- NAEFS advantage

## Solid: RMS error

- _Solid: RMS error_
- _Dash: Spread_

- _Solid: Mean error (bias)_
- _Dash: Mean absolute error_

- _36h improvement by NAEFS_

## RPSS .vs CRPSS

_**RPSS .vs CRPSS**_

_**ROC score**_

- _Winter 2006-2007_
- _NH 2m temperature_
- _For_
- _NCEP raw forecast (black)_
- _NCEP bias corrected forecast (red)_
- _NAEFS forecast (pink)_

- _24h improvement by NAEFS_

## Background !!!!!

## Relative Operating Characteristics area (ROC area)

- _Relative Operating Characteristics area (ROC area)_

_**f(noise)**_

_**f(signal)**_

- False alarm rate

- 0

- 1

- 1

- Decision threshold

- Hit rate

- _Near perfect forecast_

- _No skill forecast_

- _Real forecast_

## NAEFS Performance Review

**KEY PERFORMANCE MEASURES**

** **
_** **_

## NAEFS Configuration Review (NCEP)

**MINIMAL (PREFERRED) CONFIGURATION FOR THE GLOBAL ENSEMBLE FORECAST SYSTEMS OPERATIONAL AT CMC AND NCEP**
** **