# Presentation: 825886

## Model Verification and Diagnostic Method

- Yuejian Zhu
- Environmental Modeling Center
- NOAA/NWS/NCEP
- (September 12th 2006)

## Contents

- Introduction
- Accuracy (forecast against analysis)
- Consistency (forecast against forecast)
- Precipitation verification
- Diagnostics (forecast against analysis)
- Zonal mean – crossing section
- Area mean – vertical plots
- Instantaneous – global map plots
- Zonal mean – model physics
- Surface variables
- Climatology from 20 years observation
- Conclusions
- Expectations

## MWR Jan. 2005

- Pattern Anomaly Correlation

- Root Mean Square

- Simple Measurement
- For Ensemble mean

- Northern Hemisphere
- 500hPa geopotential height

_*MWR Jan. 2005*_

## NCEP ensemble mean performance for past 5-year

## Based on re-analysis monthly climatology

_*MWR Jan 2006*_

## Verification – Accuracyforecast against analysis

- Verification:
- Based on 2.5 by 2.5 degree resolution global data
- Variables include
- Height – 1000hPa and 500hPa for ex-tropic (20-80)
- U and V – 850hPa and 200hPa for tropic region
- RMSs for
- Analysis and forecast
- Forecast and climatology
- Mean errors for
- Analysis and forecast
- Forecast and climatology
- PACs for
- Analysis anomaly and forecast anomaly
- Climatology (NH: CPC 15 years, SH: CPC 8 years, TR: NCEP/NCAR 17 years reanalysis (New climatology is available now)
- Based on zonal wave groups
- Wave 1-3, 4-9, 10-20, 1-20
- Output example (next slide)

## File=/global/vrfy/SCORESs.2006091000

- N Hem AC by Waves (  Z  20N-80N )  -       1-3     4-9    10-20    1-20
-     0 hrs after = 0Z   9 10  6  500mb     .9997   .9997   .9978   .9996
-    24 hrs after = 0Z   9  9  6  500mb     .9910   .9954   .9675   .9926
-    48 hrs after = 0Z   9  8  6  500mb     .9801   .9846   .9012   .9786
-    72 hrs after = 0Z   9  7  6  500mb     .9527   .9725   .8423   .9577
-    96 hrs after = 0Z   9  6  6  500mb     .8734   .9391   .7382   .9035
-   120 hrs after = 0Z   9  5  6  500mb     .7282   .7083   .5722   .7084
-   144 hrs after = 0Z   9  4  6  500mb     .5204   .6188   .4591   .5663
-   168 hrs after = 0Z   9  3  6  500mb     .3941   .5021   .1802   .4286
-   192 hrs after = 0Z   9  2  6  500mb    -.0196   .1411   .0744   .0702
-   216 hrs after = 0Z   9  1  6  500mb    -.2254   .3384  -.1796   .1190
-   240 hrs after = 0Z   8 31  6  500mb     .1714   .3108   .2148   .2416
-   264 hrs after = 0Z   8 30  6  500mb     .2841   .1233  -.3993   .1642
-   288 hrs after = 0Z   8 29  6  500mb     .3842   .4616  -.0117   .4111
-   312 hrs after = 0Z   8 28  6  500mb    -.1729   .0584   .0993  -.0391
-   336 hrs after = 0Z   8 27  6  500mb     .3505   .5301   .0215   .4217
-   360 hrs after = 0Z   8 26  6  500mb     .6005  -.0643   .1385   .2888
-   384 hrs after = 0Z   8 25  6  500mb     .4517   .2523   .2839   .3370
- N Hem Differences (  Z  20N-80N )  -   (F-A)RMS   Mean (F-C)RMS   Mean
-     0 hrs after = 0Z   9 10  6  500mb      2.40    -.72   80.68   19.06
-    24 hrs after = 0Z   9  9  6  500mb     10.71   -4.35   79.79   15.43
-    48 hrs after = 0Z   9  8  6  500mb     17.25   -5.44   78.71   14.34
-    72 hrs after = 0Z   9  7  6  500mb     23.20   -4.38   79.48   15.40
-    96 hrs after = 0Z   9  6  6  500mb     34.55   -5.98   79.06   13.80
-   120 hrs after = 0Z   9  5  6  500mb     57.84   -6.58   75.40   13.20
-   144 hrs after = 0Z   9  4  6  500mb     70.65   -5.48   78.31   14.30
-   168 hrs after = 0Z   9  3  6  500mb     81.17   -7.71   78.64   12.07
-   192 hrs after = 0Z   9  2  6  500mb    107.55   -4.70   82.72   15.08
-   216 hrs after = 0Z   9  1  6  500mb    104.91  -11.99   82.07    7.79
-   240 hrs after = 0Z   8 31  6  500mb     90.64   -8.82   77.07   10.96
-   264 hrs after = 0Z   8 30  6  500mb     99.50   -4.62   85.07   15.16
-   288 hrs after = 0Z   8 29  6  500mb     86.16  -10.88   81.58    8.90
-   312 hrs after = 0Z   8 28  6  500mb    102.74   -4.69   67.51   15.09
-   336 hrs after = 0Z   8 27  6  500mb     82.02  -15.86   66.33    3.92
-   360 hrs after = 0Z   8 26  6  500mb     89.51   -7.02   78.74   12.76
-   384 hrs after = 0Z   8 25  6  500mb     83.16   -4.53   73.21   15.25

## Plotting (GrADs)

- Computer Platform:
- IBM-SP
- Time Series:
- RMS errors and PAC for different lead time
- Variables include
- Height – 1000hPa and 500hPa for ex-tropic (20-80)
- U and V – 850hPa and 200hPa for tropic region
- Die-off plots:
- RMS errors and PAC for different variables
- Variables include
- Height – 1000hPa and 500hPa for ex-tropic (20-80)
- U and V – 850hPa and 200hPa for tropic region
- Plotting example (next slide)

## Slide 9

## Slide 10

## Slide 11

## What does the numbers mean to scientist?

- _What does the numbers mean to scientist?_

## Slide 13

## Useful information

- _[http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/vrfy_grads.html](http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/html/vrfy_grads.html)_
- Verification scripts, program
- Plotting scripts, program
- _[http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/](http://wwwt.emc.ncep.noaa.gov/gmb/yzhu/)_
- My web-site contains
- GFS verification plots
- GEFS deterministic and probabilistic plots
- Forecast maps.
- NAEFS products

## Synoptic example of 500hPa height forecast

- Ini: 2003021200 Valid: 2003021700

**NCEP analysis**

**NCEP forecast**

**DAO forecast**

**MSC forecast**

**NCEP=.7978**
**DAO=.8257**
**MSC=.6068**

**Subjective evaluation**

## Good Examples

## Bad example

- _Bad example_

## Verification – consistence

- Expect to have the same/similar solution from different initial conditions – “reliability”.
- Consistency is different from accuracy.
- User consideration (requirement), user likes consistency forecasts.
- The similar properties: deterministic (quantitative) or ensemble (probabilistic) forecasts.
- Consistency forecasts should increasing/decreasing predictability gradually from different cycles (6-hour or 24-hour)
- High consistency forecast is corresponding to high forecast accuracy and reliability

## Measurement of Deterministic Forecast

- Any model forecast variables
- F(t) against F(t-1) (for same validation time)
- Example: t=t00z, t-1=t18z(yesterday) or t00z (yesterday)
- Pattern Anomaly Correlation (PAC)
- To compare the PAC from different model forecasts.
- High correlation is better
- Root Mean Square (RMS) Error
- Low RMS error is better
- Comparison
- Could be time series
- Could be period average

## Example for deterministic forecast

- NH 500hPa geopotential height
- PAC scores
- 1d means 24 hrs forecast .vs 48 hrs
- 2d means 48 hrs forecast against 72 hrs forecast valid at the same time, and so on

## DiagnosticsZonal Mean- Crossing Section

- Variables should include
- Height, temperature, zonal wind, meridional wind, relative humidity, vertical motion and horizontal divergence.
- Forecast verified against global analysis
- Analysis
- Instantaneous, 6-hr, 1-d, and 5-d average
- Forecast
- Instantaneous for different forecast lead time
- Period average for fixed lead time
- Difference
- For different variables
- Different lead times
- Examples (next slides)

## 200hPa Jet

_***200hPa Jet***_

## ITCZ

_***ITCZ***_

## DiagnosticsArea Mean- Vertical plots

- Variables should include
- Mainly temperature, and precipitation
- Forecast verified against global analysis
- Lead time
- 6 hrs, 1-, 2- ... 16-d forecast lead time
- areas
- Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
- Over land and over ocean
- Difference
- For different variables
- Different lead times
- Examples (next slides)

## Global .vs Mid-Latitude

_*Global .vs Mid-Latitude*_

## Land .vs Ocean

_***Land .vs Ocean***_

## DiagnosticsInstantaneous - global map plots

- Variables should include
- Height, temperature, winds, relative humidity, vertical motion and divergence/vorticity
- Forecast verified against global analysis
- Lead time
- 6 hrs, 1-, 2- ... 16-d forecast lead time
- areas
- Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
- Over land and over ocean
- Difference
- For different variables
- Different lead times
- Examples (next slides)

## 500hPa geopotential height

_***500hPa geopotential height***_

## Analysis error?

_***Analysis error?***_

_***Model error?***_

## DiagnosticsZonal Mean – Model Physics

- Variables should include
- Precipitation, evaporation, sensible/latent heating, 2m and skin temperature, precipitable water, total cloud cover, surface stress, radiations and etc...
- Forecast verified against global analysis
- Lead time
- 6 hrs, 1-, 2- ... 16-d forecast lead time
- areas
- Global, 30-90N, 30-90S, 20-80N, 20-80S, 20N-20S
- Over land and over ocean
- Difference
- For different variables
- Different lead times
- Examples (next slides)

## Slide 31

## DiagnosticsSurface Variables - Climatology

- Needs reliable resource to compare
- Reference: _[http://wwwt.emc.ncep.noaa.gov/gmb/noor/indest/indest.htm](http://wwwt.emc.ncep.noaa.gov/gmb/noor/indest/indest.htm)_
- COADS and other observations
- Zonal surface stress
- Meridional surface stress
- Sensible heat
- Latent heat
- Surface net short wave radiation
- Outgoing long wave radiation

## Precipitation – 20 years statistics

- Visit: http://http://wwwt.emc.ncep.noaa.gov/gmb/noor/indest/precip/xiearkin/xaclim.htm