# Presentation: 489846

## Performance of Experimental Ensemble Forecasts with Multiple Versions of NCEP's GFS ModelDingchen Hou*, Zoltan Toth,Yuejian Zhu and Richard Wobus*Environmental Modelling Centre, NCEP/NWSAcknowledgement: Hualu Pan and Shrinivas Moorthi* SAIC at Environmental Modelling Centre, NCEP/NWS

## Outline

**Introduction**
**Experimental Design**
**Ensemble Forecast Verification**
**Outlier and Error Correlation Analysis**
**Summary of Results**

## Introduction 

**Two Major Sources of Forecast Errors:**
-        Imperfect INITIAL CONDITIONS
-        Imperfect MODELS (including boundary information)
**Possible Approaches for Representing Model Related Uncertainty in Ensemble Forecasting:**
-       Multi-model Ensembles (e.g. Houtekamer et al., 1996)
-       Stochastic Physics (e.g. Buizza et al., 1999)
**Research Effort with the GFS Ensemble Forecast**
-      Multiple model version ensembles ---this Presentation
-      Inclusion of stochastic forcing ---- in progress

## Experimental design

**Global Ensemble Forecast System at NCEP**
-       NCEP Global Forecast System (GFS/MRF)
-      Breeding Technique (Toth and Kalnay, 1993, 1997)
-      5 Pairs of Perturbations
**Design of Experiment**
-      T126L28 to 24hr, then T62L28 to 384h, output every
-         12 hours;
-     August 1st -- September 30th 2002, 00Z integrations.
-     Each version of the model generates a **single-version** **ensemble** of 10 members (5 pairs).
-     **Dual-version ensembles** are generated from the single version ensembles, each consists of 5 pairs of members selected from two single-version ensembles.

## Experimental Design

** ****SAS and RAS are both based on Arakawa-Schubert **
-     1. Simplified Arakawa-Schubert scheme, **SAS** (Pan and Wu, 1994). Simplification of AS following Grell (1993); Operational in GFS and GEFS; Considering only the Deepest Clouds.
-      2. Relaxed Arakawa-Schubert scheme, **RAS**** **(Moorthi and Auares, 1992, 1999). A fairly complete implementation of AS updraft cloud model;  Adjustment achieved in finite time through iteration (relaxation).
-      3. No Cumulus Parameterisation Scheme is used  (**NAS**). Included for comparison.

**Experimental Design**

## Verification of  Precipitation   Verification Domain: North America   Verification Scores:          EQ threat score            TSS score:                 True Skill Statistics (Hanssen-Kuipers Discriminant)            BIAS, Ratio of forecast to observation   Results:   SAS and RAS ensembles are similar in performance scores, while NAS is much poorer.   Dual-Version Ensembles have no improvement over Single-Version Ensembles SAS/RAS.

**Ensemble Forecasts Verification**

## Precipitation Forecast SkillsSAS  RAS and NASDay 1                     Day 3

- EQ Threat

- TSS

- BIAS

- EQ Threat

- TSS

- BIAS

- Precipitation Forecast SkillsSAS  RAS and NASDay 1                     Day 3

## Precipitation Forecast Scores Day 3SAS/RAS/S+R.   SAS/NAS/S+N.

## Ensemble Forecasts VerificationVerification of 500hPa Height (ensemble mean)   Verification Domains: NH and SH (20-80 degrees)   Verification Data set: GFS Final Analysis     Verification Scores       AC=Anomaly Correlation Coefficient      BIAS=Mean Error of Ensemble Mean       RMS=Root Mean Square Error of Ensemble Mean       Spread of Ensemble     Results   The difference in performance scores between SAS and RAS ensembles is very small.   NAS ensemble has larger negative bias and RMS, but the difference is not significant.    Dual-Version Ensembles don't have significant improvement in terms of AC and RMS.

## Forecast Scores 500hPa height (SAS,RAS,S+R)Time series of day 5                     As a function      day 5 forecast                        of lead time (days)

- correlation

- Correlation

- RMS

- Spread

- RMS

- Spread

**SAS****, NAS and ****S+N**

## Outlier and Error Correlation Analysis

**Excessive Outliers.** Count the percentage of members of ensemble 1 falling outside the range of ensemble 2 forecasts at each grid point and take the average over the domain and cases of consideration, one get percentage outliers. Subtracting the expected value  from two completely overlapping ensembles (2/11) leads to percentage excessive outliers (PEO).

- outliers

- Ensemble 1

- Ensemble 2

- SAS-RAS
- PEO>0

- RAS-SAS
- PEO<0

- Partial Overlapping

- Complete Overlapping

- PEO >0

- outliers

**Excessive outliers**
**--- Overlapping of **
**two SAS and RAS**

- Before Removing
- mean error

- After removing
- mean error

- R-S

- R-S, S-R

- N-S, N-R

## Outlier and Error Correlation Analysis

- Error Correlation Coefficient between two forecasts  **R=Cov(E1,E2) / (var(E1,E1)var(E2,E2))**(1/2)** where **E **is forecast error vector.** ** R is related to the the similarity between two forecasts. Smaller R means larger difference and higher possibility for RMS reduction by taking (weighted) mean of the two forecasts.

- Ensemble 1

- Ensemble 2

- R close to -1 --- Lower RMSE
- In mean of dual-version ens.

- R close to 1, insignificant
- reduction of RMSE

- SAS and RAS

- Imaginary case

## Error Correlation Coefficients

- NH

- SH

- --- Mean for members
- --- Control to Control
- --- Ens. Mean to mean
- --- Mean for mem-mean

- Between SAS and RAS

- SAS and RAS leads to
- difference in control fcst.
- But the difference between
- corresponding members
- decreases with time,
- results in relatively small
- difference between the
- mean.
- The relatively high R
- between SAS member and
- RAS mean suggests little
- improvement to RAS mean
- by adding a SAS member.

## Summary of the Results

-   1. SAS and RAS performance scores and spread for both height and precipitation are relatively close to each other on average.
-   2. When cumulus parameterisation scheme is excluded (NAS), poorer performance in precipitation forecast is expected. However, the difference in the 500hPa forecast is trivial for up to one week of lead time and insignificant for week 2.
-   3. The dual-version ensemble with SAS and RAS, shows no significant improvement in the forecast of both 500hPa height and precipitation, compared with the single-version ensembles.

## Summary of Results

-    4. When NAS is combined with SAS or RAS, the performance in both precipitation and 500hPa height shows no significant difference from SAS/RAS.
-    5. Outlier analysis and Error Correlation analysis revealed that the SAS and RAS ensembles are largely overlapping. With the change of cumulus parameterisation scheme the ensemble cloud in the phase space (and hence the ensemble mean) does not change significantly (although individual forecasts do).
-    6. With the trivial effect of including SAS and RAS and the insignificant effect of NAS, the multi-model version approach to improve the NCEP’s global ensemble forecast system seems not promising. The effort has been shifted to stochastic forcing.

**SAS****, ****RAS**** and S+R**

## Forecast Scores 500hPa Height (SAS,NAS,S+N) Time Series                      As a Function of Day-5 Forecast              of Lead Time           

- RMS

- Spread

- AC

- AC

- RMS

- Spread