# Presentation: 517003

## POSSIBLE GLOBAL ENSEMBLE UPGRADES, 2007-2009

**Zoltan Toth**
**Environmental Modeling Center**
**NOAA/NWS/NCEP**
- Ackn.: Yuejian Zhu, Dick Wobus, Dingchen Hou, Bo Cui, Malaquias Pena, Joo-Hyung Son
*http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html*

## CENTRAL COMPUTING SYSTEM

- Current Central Computing System (CCS) became operational early 2007
***Up to ~4-fold increase in CPU usage possible until***
- Next upgrade in 2009/10
- Potential improvements
- Some require major increase in cpu
- Even if tests are successful, implementation for some of them will need to be delayed until next CCS upgrade (2010)
- Current configuration:
- T126, L28, 20 members, ET, no stochastic physics

## Increased resolution
T126 => T190						x 2.3 cpu incr.
L28 => L64						x 2.3
Reduced impact after bias correction and downscaling?
Improved physics						x 1.5
Preferably keep using same physics for hires GFS & ensemble
Case dependent estimate of analysis uncertainty in ET  	-
Estimate GSI analysis uncertainty, use to constrain ET
Stochastic perturbations					-
Account for increased error growth due to imperfect models 
Hind-casting	(28 yrs, once per day or couple of days)		x 1.33 - 2.3
Improve bias correction beyond 5 days
Sub-seasonal forecasting
Ocean / Sea Ice coupling					x 1.4
Extend forecasts to 30 - 45 days				x 1.25 – 1.8
Major expansion of applications; TPARC-IPY
River flow forecasting					-
Enhanced post-processing
Improved bias correction / combination			-
Implement downscaling					-
Aggregate increase without change in resolution 		x 3.5

- T126 => T190						x 2.3 cpu incr.
- L28 => L64						x 2.3
- Reduced impact after bias correction and downscaling?
- Improved physics						x 1.5
- Preferably keep using same physics for hires GFS & ensemble
- Case dependent estimate of analysis uncertainty in ET  	-
- Estimate GSI analysis uncertainty, use to constrain ET
- Stochastic perturbations					-
- Account for increased error growth due to imperfect models
- Hind-casting	(28 yrs, once per day or couple of days)		x 1.33 - 2.3
- Improve bias correction beyond 5 days
- Sub-seasonal forecasting
- Ocean / Sea Ice coupling					x 1.4
- Extend forecasts to 30 - 45 days				x 1.25 – 1.8
- Major expansion of applications; TPARC-IPY
- River flow forecasting					-
- Enhanced post-processing
- Improved bias correction / combination			-
- Implement downscaling					-
***Aggregate increase without change in resolution ******	******	******x 3.5***

**POTENTIAL UPGRADES DURING 2007-09 PERIOD**

## BIAS ESTIMATION

- Definition
- General: Bias in ensemble forecast distribution
- Specific: First & second moments of distribution
- Bias estimates in first moment related to
- Difference between time mean forecast (f) and analysis (a, proxy for truth)
- Quality of bias estimate
- Determines quality of bias correction (example for first moment)
- Depends on signal (true bias) and noise (random forecast error) ratio
- Short lead time – small random forecast error, low noise
- Longer lead time – large random forecast error, large noise
- Increase in sample size leads to reduced error in bias estimate

## SAMPLE SIZE REQUIREMENTS

- Short lead forecasts
- Most recent f-a sample (30-90 days) adequately large
- EXPERIENCE:
- Advantage from regime dependent bias estimate
- Longer lead forecasts (5+ days)
- Hind-casts can significantly improve bias estimates
- ASSUMPTION:
- Size and nature of analysis error (DA specifics) is irrelevant beyond 5 days
- Practical approach
- Linearly combine bias estimates from
- Recent data sample	1-5 days
- Hind-cast		5+ days

## HIND-CAST GENERATION

- Alternatives
- Frozen DA / forecast system
- Generate once
- Labor intensive
- EXPERIENCE:
- No benefit from DA/forecast improvements
- Operational DA / forecast system
- Real time generation
- Automatic process, not one-time effort
- All upgrades to DA / forecast system can be used
- ~2-month parallel testing required
- Proposed configuration for global ensemble
- Single forecast only, using reanalysis for hind-cast initiation
- EXPERIENCE:
- Significant benefit (Hamill & Whitaker)
- ASSUMPTIONS:
- Forecast uncertainty can be reasonably estimated from recent sample
- One-member hind-cast provides reasonable estimate for hind-cast uncertainty
- Hind-casts from reanalysis with less data less relevant for uncertainty estimation sample adequate to
- TO BE TESTED:
- Additional potential benefits from
- Ensemble hind-casts
- Real-time reanalysis (with operational DA/forecast systems)

## REAL-TIME GENERATION OF HIND-CAST DATASET?

- Today’s Julian Date
- TJD

- TJD + 30

- TJD - 30

- Actual ensemble **generated today**

- Hind-casts for TJD+30 **generated today**

- Hind-casts (or its statistics) for TJD+/- 30 **saved on disc**

- Time

- 2005

- 2004

- 2003

- 1968

- 1967

**REAL-TIME GENERATION OF HIND-CAST DATASET?**

- 2006

## Fig. 7. The sample size required for the absolute bias error to be less than 0.05 (solid line denotes traditional approach and the dashed line Bayesian approach) and 1.0 (the dot-dashed line denotes traditional approach and the dotted line the Bayesian approach) as a function of correlation. 

- Fig. 7. The sample size required for the absolute bias error to be less than 0.05 (solid line denotes traditional approach and the dashed line Bayesian approach) and 1.0 (the dot-dashed line denotes traditional approach and the dotted line the Bayesian approach) as a function of correlation.

## Operational

**CDAS**

**CFS (2003)**

## Slide 10

## Representing Model Related UncertaintyA Proposed Stochastic Perturbation (SP) Scheme

**General Approach: **Adding a stochastic forcing term in to the tendencies of the model equations.
**Strategy**: Generate the S terms from (random) linear combinations of the conventional perturbation tendencies.

**Desired Properties **
- 1.    Forcing applied to all variables
- 2.   Approximately balanced
- 3.   Smooth variation in space and time
- 4.    Flow dependent
- 5.    Quasi-orthogonal

**Example of Combination Coefficients**

**Reduced Outlier**

**Reduced Bias**

**Increase Spread**

**Reduced Absolute Systematic error**

**RMSE**

**Increase Spread**

**Increase Spread**

- ---- Operation
- ---- Operation +1%PP
- ---- Operation +
-        optimal pp
-        (upper limit)

## Ensemble Streamflow ForecastTwo Possible Approaches  One way and two way coupling               

- Experimental Medium-range Ensemble Streamflow Forecasts Based on Coupled GFS-Noah Ensemble Runoff Forecast

**Runoff**
**(ensemble)**

**Fluxes**

- CTL

- ENS_Mean - CTL
- Difference

- (Mean of member
- Scores)
- -(CTL score)
- Difference

- GFS-CTL Difference

**Ranges:**
** ****(m**3/s)**
**>2000m**
**1000-2000**
**500-1000**
**300-500**
**200-300**
**70-90**
**35-45**
**15-20**

**>2000m**3/s**
**1000-2000**

**500-1000**
**300-500**

**200-300**
**70-90**
**35-45**
**15-20**

**>2000m**3/s**
**1000-2000**

**500-1000**
**300-500**

**200-300**
**70-90**
**35-45**
**15-20**

- Without Bias-correction

- After Bias-correction

- 0

- 0

**Discussion: **
- Although the bias-removal can not be applied in operational forecasting, practical bia-correction algorithms may have similar effect.

**Observations:**
-  Positive skill for the large river basins.
-  Improvement due to bias-correction.
-  Positive skill for all river basins after bias correction.

**Temporal Correlation**

**CRPSS**

## OUTLINE / SUMMARY

**WHAT AFFECTS FORECAST QUALITY?**
- RESOLUTION
- RELIABILITY
**WHAT AFFECTS QUALITY OF BIAS CORRECTION? **
- BIAS CORRECTION METHOD
- SAMPLE SIZE
**GENERATION OF HIND-CASTS WITH FROZEN DA/MODEL**
- INCREASES SAMPLE SIZE
- REDUCES RESOLUTION/RELIABILITY OF RAW FORECAST
**REAL-TIME GENERATION OF HIND-CASTS**
- LATEST DA/MODEL VERSION USED
- LARGE SAMPLE FOR CORRECTING BIAS IN 1ST MOMENT

## QUALITY OF BIAS CORRECTION

**ASSUME BOTH NATURE & FORECAST ARE STATIONARY***** ***
- Quality of bias correction depends on quality of bias estimation
- Bias estimation
- Estimating difference between expected values of two processes
- Nature (or its proxy) and
- Forecast of nature
- Statistical filters can be used
- Desirable characteristics of bias estimation
- Convergence to true value with increasing sample size
- Unbiased estimate
- Influenced by choice of method
- Minimal random noise in estimate
- Rate of convergence to unbiased estimate with increasing N must be high
- State of convergence depends on
- Choice of method
- Sample size

## QUALITY OF BIAS CORRECTION

**REAL WORLD APPLICATIONS***** ***
- More complex problem, processes not stationary but influenced by
- Seasonal cycle
- Bias estimate as function of seasonal cycle
- Can be done but increases sample size requirement
- Regime changes
- Bias estimate as function of regime
- Can be done
- Use of recent data works well for short lead
- Increases sample size requirement for long lead
- Changes in forecast process (periodic upgrades of NWP DA & modeling)
- Estimation can be done but may require regeneration of large hind-cast dataset
- How to balance influence of different factors?
- Update model continuously
- Best statistical resolution
- Lack of large hind-cast dataset corrupts ability to bias correct
- Generate large hind-cast dataset
- Best reliability
- Corrupts statistical resolution for not using best available NWP system
- Can we have the good aspects of both worlds?

## ISSUES

- How much skill we lose by using an 8-10 year old forecast system?
- 2 days skill at D3, 3 days at D7.5, 4+ days at 9D
- Estimate 2% rms error reduction per year in recent yrs
- How much skill we gain due to use of larger sample?
- Reduction depends on level of bias in forecast
- CDC hind-casts have much larger bias
- Large bias reduction observed
- Still does not compensate for loss of skill until 10+ day lead time
- Operational forecast system
- Not known
- Estimate via synthetic data
- Use synthetic data to estimate bias error reduction with more data & current fcst
- What is current level of NHE time mean error (estimated bias)?
- ~6-14% of climate standard deviation (2m temp)
- Gain from expanding sample
- From 100 days (Kalman Filter method) to 5,000 days (50 yrs hind-cast)
- 5-10% (depending on method) of random rms error
- This is equivalent to 2.5-5 yrs of model development
- If DA/model has to be frozen for 3-5 yrs
- Gain in reliability due to larger sample &
- Loss in resolution and reliability due to use of frozen system
- Are of similar magnitude
- Is it worth it?

## Raw, Optimal & Actual Bias Corrected Ensembles  

-    bias correction improves RPSS for all lead time vs. raw oper. ens.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains  significant improvement for all lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-     oper. fcst is better than the bias-corrected reforecast out to 9-10 days.  Beyond 10 days, bias-corrected  reforecast becomes competitive to or better than oper. fcst

-  Sign of improving
-    larger for CDC reforecast

- Annual Mean RPSS ( 20040301 – 20050228 )
- 500 mb Height over Northern Hemisphere

## Gain per year = 

*Gain in 5 yrs = 0.06 or*
*	**	**~15 hrs*

*Gain per year = *
*2% rms error reduction*

*Gain in 5 yrs = 15 hrs or*
*	**6 m (10%)*

## Slide 19

## ISSUES - 2

- Regime dependent vs. climate mean bias correction
- Regime dependent (with small sample)
- Better at short lead
- Climate mean (with much larger sample)
- Better at long lead
- Could try regime dependent correction using hind-cast sample
- Combined method may be best?
- Is it worth the effort to generate hind-cast dataset?
- Questionable
- Are there additional factors/gains to consider?
- Bias correction for highly non-linear downstream applications
- River flow forecasting?
- Can we have the good aspects of both worlds?
- Latest model AND large hind-cast dataset?
- Real time generation of hind-cast data
- Assume biggest problem is bias in first moment
- Can be estimated using large sample of *single* forecasts
- Use recent data only to correct bias in higher moments of ensemble
- Focus on long lead bias estimation problem
- Regime dependent short lead estimation ok with most recent small sample
- Assume long lead bias depends only on model (and not quality of initial condition)

## Raw, Optimal & Actual Bias Corrected Ensembles  

- CDC reforecast
-  decaying method gives better results than climate mean bias estimation for short range (~day 5), value of regime dependent correction
-  some gain from climate mean bias correction after _5 days

- Raw, Optimal & Actual Bias Corrected Ensembles

- RPSS of 500 mb Height
- Northern Hemisphere, 2004 Summer

## Raw, Optimal & Actual Bias Corrected Ensembles  

- CDC reforecast
-  decaying method gives better results than climate mean bias estimation for short range (~day 5), value of regime dependent correction
-  some gain from climate mean bias correction after _5 days

- Raw, Optimal & Actual Bias Corrected Ensembles

- RPSS of 500 mb Height
- Northern Hemisphere, 2004 Summer

## Slide 23

## REAL-TIME GENERATION OF HIND-CAST DATASET?

- When computing facilities are upgraded (2007 at NCEP)
- Forgo increasing resolution/membership
- Instead generate hind-cast “ensemble”
- Arrangement
- Generate control forecast from re-analysis initial conditions
- Initialized from 30 days ahead of current date
- For each year in re-analysis dataset (30-50 yrs)
- Additional cost is 2-3 times current ensemble configuration - *DOABLE*
- Bias correction methodology
- Same as currently, except
- Estimate climate mean bias as mean of difference between
- Re-analysis & hind-cast forecast (+/-30 days x 30 yrs = 1,800 sample)
- Use centrally weighted recursive filter to save on disc storage requirements
- Combine regime dependent Kalman filter + climate mean bias estimates
- More weight on Kalman filter at short
- More weight on climate mean estimate at long leads
- Explore possible regime dependent corrections at long lead
- Save & use entire sample of 1,800 using hind-casts in research
- Discard 30+ day old hind-cast data
- Implement DA/model upgrades at wish
- Generate regime dependent & climate mean bias statistics during
- ~30-day parallel testing prior to operational implementation

## BACKGROUND

## Current method, Real data

*Before bias correction*

*After bias correction*

*Estimated*

- Current method, Simulated data

*Estimated*

*Before bias correction*

*After bias correction*

- Current & new methods, Simul. data

*Before bc.*

*After bc, new method*

*Actual*

- New method, Simulated data

*Estimated*

*Before bias correction*

*After bias correction*

*After bc., current method*

##  Bias correction methods  

-  Bias correction methods
- Decaying averaging bias correction ( ~ 46 day independent training data)
- Operational NCEP ensemble
- CDC reforecast data
- 25-yr climatological mean forecast error (Hamill & Whitaker)
- CDC reforecast
- 31-day centered running mean forecast error ( dependent data), used as “optimal” benchmark
- Operational NCEP ensemble
- CDC reforecast data
-  “Hybrid” system, 2 step adjustment

-    FCSTclibriated = FCSTOPR – BIAS Decaying OPR-RFC – BIASRFC-REA (25-YRS)

-  Data sets
-  NCEP operational global ens. Data (2004/05 analysis/modeling system)
-  CDC reforecast data set (1998 modeling system, 1978 – 2005)
- 500 mb height, 850 mb temp, 2m temp, 10m U and V, Mar. 2004-Feb. 2005

## Raw, Optimal & Actual Bias Corrected Ensembles  

-    bias correction improves RPSS for all lead time vs. raw oper. ens.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains  significant improvement for all lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-     oper. fcst is better than the bias-corrected reforecast out to 9-10 days.  Beyond 10 days, bias-corrected  reforecast becomes competitive to or better than oper. fcst

-  Sign of improving
-    larger for CDC reforecast

- Annual Mean RPSS ( 20040301 – 20050228 )
- 500 mb Height over Northern Hemisphere

## Raw, Optimal & Actual Bias Corrected Ensembles  

-  Decaying average
-    oper. ens. with bias correction has better performance than the raw fcst.

- Raw, Optimal & Actual Bias Corrected Ensembles

- 3 OPRR ENS.

- 3 RFC ENS.

-  Climate error removed
-     bias corrected reforecast gains significant improvement for most lead time vs. raw reforecast

-  Operational vs. reforecast ens.
-    both the raw and post-processed oper. fcst. are better than the bias-corrected reforecast

- RPSS of 850 mb Temperature
- Northern Hemisphere, 2004 Summer

-   RMS error of OPR_DAV2% reduced for first week
-  Climate error removed
-   RMS error of RFC_COR   improvement for all lead  times vs. RFC_RAW

- Raw, Optimal & Actual Bias Corrected Ensembles

## Bias-correction algorithms

**Current method (method 1)**

- : Kalman Filter weight

**New method (method 2)**
- Bias   ~  weighted average of
- Bias Estimation
- Equal weight
- Kalman Filter
- Bias correction

## Slide 32

## Slide 33

## Slide 34