# Comparison **of CENRAP, VISTAS, and**  **WRAP 36 km MM5 Model Runs for 2002 Task 3: Meteorological**  **Gatekeeper Report** 
Susan Kemball

- Cook, Yiqin Jia, Chris Emery and Ralph Morris

ENVIRON International Corporation Zion Wang and Gail Tonnesen UCR CE
- CERT 

# Western Regional Air Partnership (WRAP) **Regional Modeling Center (RMC)**

# Model Setup for CENRAP, **WRAP, and VISTAS**  **Runs**

• 

# All use National RPO 36 km Grid, **164 x 128 x 34**

• 

# Lambert Conformal Projection

• 

# Pleim **- Xiu/ACM LSM/PBL** 

• 

# RRTM Longwave **and Dudhia Shortwave**

• 

# No shallow convection

# Model Setup for CENRAP, **Initial WRAP,**  **and**

**VISTAS Runs, continued.**

**CENRAP VIST** 

**AS WRAP**  **1**


## Cumulus **Convection Kain**  ****

**Fritsch 2 Kain**  ****

**Fritsch 2 Kain**  ****

**Fritsch Moist**  **Physics Reisner**  **1 Reisner** 

**1 Reisner** 

**2 Analysis**  **Nudging Surface**  **u/v u/v**  **u/v/t/q Analysis**  **Nudging Aloft**  **u/v/t/q u/v/t/q**  **u/v/t/q**



# Physics Options Differ **ences**

1 Final WRAP

MM5 used Betts

- Miller Cumulus and u/v surface analysis nudging

# Method for Comparing **the CENRAP,**

**WRAP, and VISTAS MM5 Runs**

• Surface statistics for wind, temperature, and humidity

• Upper air soundings of temperature, dew point, and winds

• Precipitation For CENRAP

, WRAP , and VIST

AS runs, we examine:

and compare with observations. Focus on CENRAP

dom ain, and also give overview of performance in other regions.

# Surface Evaluation **Method**

• Use Metstat statistical package to compare model surface output to surface station data

• Metstat calculates statistics which describe how well the model output fields agree with observations

• Metstat analysis divides U.S. into subdomains to show how MM5 performs region by region

• Metstat statistics summarized in scatter plots which contain a model performance benchmark box as well as results of other MM5/RAMS runs used for air quality modeling

# METSTAT Evaluation **Package**

• 

# Average observed and predicted

• 

# Absolute Bias and Error

• 

# RMSE

• 

# Index of Agreement (IOA)

• 

# Daily and, where appropriate, Hourly **Evaluation**

• 

# Statistical **Performance Benchmarks**  > 

# Based on an analysis of > 30 MM5 and RAMS runs > 

# Not meant as a pass/fail test, but to put modeling results in the proper perspective

Wind Speed Wind Dir ection Te mper atur e Hu midity RM SE  2 m /s Me an Bi as   0. 5m/ s   10    0. 5K   1g/ kg Index of Agr eem en t  0 .6  0 .8  0 .6 Gr oss Erro r  3 0   2 K  2 g/ kg

# Subdomains for Model Evaluation
1 = Pacific NW 2 = SW 3 = North 4 = Desert SW 5 = CenrapN 6 = CenrapS 7 = Great Lakes 8 = Ohio Va lley 9 = SE 10 = NE 11 = MidAtlantic

# Datasets **for Met Evaluation**

• 

# NCAR **dataset ds472 airport surface met observations** 

• 

# Twice **- Daily Upper**  **- Air Profile**  **Obs (~120 in US)** > 

# Temperature > 

# Moisture surface temperature surface pressure sea level pressure visibility cloud cover height of cloud base max imum temperature mi nimum

temperature dew poi nt precipitation sunshine snowfall snow cover win d direction win d force (speed) u w ind component v wind component win d gust descriptive weather

### • WRAP

has smallest error

• cenrapN subdomain outside goal for CENRAP run **CENRA P**  **/ V IST**  **AS / WR**  **AP J**  **anuar y**  **Te mp**  **erat ur**  **e Per**  **fo rm**  **ance Com**  **par is**  **on Ov**  **er C ENRA**  **P Domain** 

0 0.5 1 1.5 2 2.5 3 3.5

-2 -1 .5 -1 -0 .5 0 0.5 1 **Temper ature**  **Bi as**  **(K )** **T e** **m p** **e r** **a t** **u r** **e E** **r r** **o r** **( K** **)**

Be nc hm ark CE NRA P VI ST AS WR AP i CE NRA P N CE NRA P S 

### • WRAP has smallest error

• cenrapN subdomain outside goal for CENRAP run

### CE **NRAP / VISTA**  **S / WR**  **AP Jan**  **ua ry**  **Hu mi**  **dity**

**P erfo**  **rman ce**  **Co mp**  **ari so**  **n O ver**  **CE NRAP**  **Do mai**  **n**

0 0. 5 1 1. 5 2 2. 5 3

-1. 5 -1 -0. 5 0 0. 5 1 1. 5 2 2. 5 **Hu mi**  **di ty**  **B ia**  **s (g**  **/k g)** **H u** **m i** **d i** **t y** **E r** **r o** **r (** **g /** **k g** **)**

Be nc hmark CE NRA P VI ST AS WR

AP i CE NRA P N

CE NRA P S 

### • Humidity performance is comparable for all three runs

### CE **NRAP / V**  **IS TA**  **S / WR AP**  **J an**  **ua ry**  **Wind P erfo**  **rman ce**  **Co mp**  **ari so**  **n ov er**  **CE NRAP**  **Do mai**  **n**

0 10 20 30 40 50 60 70 80

0 0. 5 1 1. 5 2 2. 5 3 **Wi nd**  **Speed RM**  **SE (m/s**  **) W** **i n** **d D** **i r** **e c** **t i** **o n** **E r** **r o** **r (** **m /** **s )** 

Be nc hm ark CE NRA P VI ST AS WR

AP i CE NRA P N CE NRA P S 

### • Wi nd performance is comparable for all three runs

### CENRAP **/ V ISTA**  **S / WR**  **AP Ma**  **rch Te**  **mp eratu**  **re Perfo**  **rman ce**  **Co mp**  **ari so**  **n O ver**  **CE NRAP**  **Do mai**  **n**

0 0. 5 1 1. 5 2 2. 5 3 3. 5

-2 -1. 5 -1 -0. 5 0 0. 5 1 **Te mp**  **erat ure**  **Bi as**  **(K )** **T e** **m p** **e r** **a t** **u r** **e E** **r r** **o r** **( K** **)**

Be nc hm ark CE NRA P VI ST AS WR

AP i CE NRA P N CE NRA P S 

### • WRAP has best overall performance

• cenrapN subdomain outside goal for CENRAP run

### CENRAP **/ V ISTA**  **S / WR**  **AP Ma**  **rch H**  **um idity**  **Pe rfo**  **rman ce**  **Co mp**  **ari so**  **n O ver**  **CE NRAP**  **Do mai**  **n**

0 0. 5 1 1. 5 2 2. 5 3

-1. 5 -1 -0. 5 0 0. 5 1 1. 5 2 2. 5 **Hu mi**  **di ty**  **B ia**  **s (g**  **/k g)** **H u** **m i** **d i** **t y** **E r** **r o** **r (** **g /** **k g** **)**

Be nc hm ark CE NRA P VI ST AS WR

AP i CE NRA P N

CE NRA P S 

### • Performance is comparable for all three runs**CENRA P**  **/ V IST**  **AS / WR**  **AP Ma**  **rch Wi**  **nd Per**  **fo rm**  **ance Compar**  **is on**  **Ov er**  **CENRA P**  **Dom ai**  **n**

0 10 20 30 40 50 60 70 80

0 0.5 1 1.5 2 2.5 3 **Wi nd**  **Speed RM**  **SE (m/s**  **) W** **i n** **d D** **i r** **e c** **t i** **o n** **E r** **r o** **r (** **m /** **s )** 

Benc hm ar k CE NRA P VI ST AS WR

AP i CENRA P N CENRA P S**CENRA P**  **/ V IST**  **AS / W**  **RA P**  **Jul y**  **Te mp**  **erat ur**  **e Per**  **fo rm**  **ance Com**  **par is**  **on Ov**  **er CENRA**  **P Dom**  **ai n** 

0 0.5 1 1.5 2 2.5 3 3.5

-2 -1 .5 -1 -0 .5 0 0.5 1 **Tem per**  **at ur** 

**e Bi**  **as (K**  **) T** **e m** **e p** **r a** **t u** **r e** **E r** **r o** **r (** **K )** 

Be nc hmark VI ST AS WR AP i CE NRA P CE NRA P N CE NRA P S 

### • Te mperature error is higher in all runs relative to January

### CENRAP **/ V ISTA**  **S / WR**  **AP Ju**  **ly H**  **um idit**  **y Pe**  **rf or**  **man ce**  **Co mp**  **ari so**  **n O ver**  **CE NRAP**  **Do mai**  **n**

0 0. 5 1 1. 5 2 2. 5 3

-1. 5 -1 -0. 5 0 0. 5 1 1. 5 2 2. 5 **Hu mi**  **di ty**  **B ia**  **s (g**  **/k g)** **H u** **m i** **d i** **t y** **E r** **r o** **r (** **g /** **k g** **)**

Be nc hm ark VI ST AS WR AP CE NRA P CE NRA P N CE NRA P S 

### • Humidity error is higher in all runs relative to January**CENRA P**  **/ V IST**  **AS / W**  **RA P**  **Jul y**  **Wi nd**  **Per fo**  **rm ance**  **Com par**  **is on**  **Ov er**  **CENRA P**  **Dom ai**  **n**

0 10 20 30 40 50 60 70 80

0 0.5 1 1.5 2 2.5 3 **Win d**  **Speed RM**  **SE (m/s**  **) W** **i n** **d D** **i e** **r c** **t i** **o n** **E r** **r o** **r (** **m /** **s )** 

Be nc hmark VI ST AS WR AP i CE NRA P CE NRA P N CE NRA P S 

### • Wi nd performance is comparable for all three runs

### CE **NRAP / VISTA**  **S / WR AP**  **O cto**  **be r**  **Te mp**  **erat ur**  **e Pe**  **rf or**  **man ce**  **Co mp**  **ari so**  **n O ver**  **CE NRAP**  **Do mai**  **n**

0 0. 5 1 1. 5 2 2. 5 3 3. 5

-2 -1. 5 -1 -0. 5 0 0. 5 1 **Te mp**  **erat ure**  **Bi as**  **(K )** **T e** **m p** **e r** **a t** **u r** **e E** **r r** **o r** **( K** **)**

Be nc hm ark CE NRA P VI ST AS WR

AP i CE NRA P N CE NRA P S 

### • WRAP has smallest error

### CEN **RAP / V**  **IS TAS / W**  **RAP Oc**  **tober H**  **umi di**  **ty**

**P er**  **form ance**  **Com par**  **is on O**  **ve r**  **CEN RAP**  **Dom ai**  **n**

0 0. 5 1 1. 5 2 2. 5 3

-1. 5 -1 -0 .5 0 0. 5 1 1. 5 2 2. 5 **Hum id**  **ity B**  **ia s**  **(g /k**  **g) H** **u m** **i d** **i t** **y E** **r r** **o r** **( g** **/ k** **g )** 

Be nc hm ar k CENRA P VI STA S WR AP i CENRA P N CENRA P S 

### • Performance is comparable for all three runs

### CENRAP **/ V ISTA**  **S / WR**  **AP Oct**  **obe r**  **Wind P erf**  **or man**  **ce Co**  **mp ari**  **so n O**  **ver CE**  **NRAP Do**  **mai n** 

0 10 20 30 40 50 60 70 80

0 0. 5 1 1. 5 2 2. 5 3 **Wi nd**  **Speed RM**  **SE (m**  **/s )** **W i** **n d** **D i** **r e** **c t** **i o** **n E** **r r** **o r** **( m** **/ s** **)**

Be nc hm ark CE NRA P VI ST AS WR

AP i CE NRA P N

CE NRA P S 

### • Wi nd performance is comparable for all three runs

## Su mma ry o f CEN RAP / V IS TAS / W RAP Me ts ta t S ta tis tic s Ja nuary Ma rc h Ma y Ju ly Oc to ber Dec em ber ce nrap N te mp erat ure WR AP WR

AP WR

AP WR

AP WR

AP WR

AP hum id ity WR AP WR

AP None VI ST AS None WR

AP wi nd No ne V/C VI ST AS CE NRA P VI ST AS CE NRA P
ce nrap S te mp erat ure No ne WR AP WR

AP CE NRA P None WR AP hum id ity VI ST AS VI ST

AS None No ne WR AP WR

AP wi nd No ne V/W VI ST AS CE NRA P WR

AP None
ov eral l te mp erat ure WR AP WR

AP WR

AP No ne WR AP WR

AP hum id ity No ne VI ST AS None No ne WR AP WR

AP wi nd No ne VI ST AS VI ST

AS CE NRA P None WR AP 

### Entry in table indicates which run performed best in that subdomain for that field. Note that initial WRAP

r un used surface t/q nudging that can improve performance statistics but introduce instabilities

# Overview **of CENRAP/WRAP/VISTAS**  **Surface Performance in CENRAP Region** 

• Overall, performance over CENRAP region was within benchmarks.

• All runs had similar wind and humidity performance.

• CENRAP

and VIST

AS had lar ger temperature error than WRAP

• For CENRAP

run, the cenrapN subdomain fell outside the benchmark in January and December

# Overview **of CENRAP/WRAP/VISTAS**  **Surface Performance in CENRAP Region** 

• 

# In general, surface performance is better in winter than in summer for all three runs.

• Humidity error is higher in the summer than in the winter for all three runs; there is wet bias in the winter . Less clear cut in other seasons.

• Wi nd speed RMSE and wind direction error increase going from winter to summer , then decrease again as winter approaches.

• For all runs, cenrapN subdomain has cold bias in winter and a warm bias in summer . Less clear cut for cenrapS.

# Performance **Outside CENRAP region**

• 

# Surface performance had strikingly dif ferent characteristics in the eastern vs. western U.S. subdomains.

• The following soccer plots emphasize performance issues over western U.S common to all three runs.

• We stern U.S. subdomains are shown with filled icons. All other subdomains denoted by open circles colored according to run.

• When an eastern subdomain falls outside the benchmark, it is identified.**CENRA P**  **/ V IST**  **AS / WR**  **AP J**  **anuar y**  **Te mp**  **erat ur**  **e Per**  **fo rm**  **ance Com**  **par is**  **on**

0 0.5 1 1.5 2 2.5 3 3.5

-2 -1 .5 -1 -0 .5 0 0.5 1 **Temper ature**  **Bi as**  **(K )** **T e** **m p** **e r** **a t** **u r** **e E** **r r** **o r** **( K** **)**

Be nc hm ark VI ST AS WR AP CE NRA P SE NE ce nr apN

ohiov al De se rt SW Pa cN W SW No rt h 

### • WRAP has sm allest error
• We stern subdom ains have lar ge cold bias**CENRA P**  **/ V IS**  **TA S**  **/ WR AP**  **Januar y**  **Hum id**  **it y**  **Per fo**  **rm ance**  **Com par**  **is on** 

0 0.5 1 1.5 2 2.5 3

-1 .5 -1 -0 .5 0 0.5 1 1.5 2 2.5 **Hum id**  **ity B**  **ia s**  **(g /k**  **g) H** **u m** **i d** **i t** **y E** **r r** **o r** **( g** **/ k** **g )** 

Benc hm ar k VI ST

AS WR

AP CE NRA P De se rt SW Pa cN W SW No rt h 

### • All subdomains lie within benchmark**CENRA P**  **/ V IST**  **AS / WR**  **AP J**  **anuar y**  **Wi nd**  **Per fo**  **rm ance**  **Compar ison** 

0 10 20 30 40 50 60 70 80

0 0.5 1 1.5 2 2.5 3 **Wi nd**  **Speed RM**  **SE (m/s**  **) W** **i n** **d D** **i r** **e c** **t i** **o n** **E r** **r o** **r (** **m /** **s )** 

Benc hm ar k VI ST

AS WR

AP CENRA P NE De se rt SW Pa cN W SW No rt h 

### • Wi nd performance does not meet benchmark in any western U.S. subdomain

We stern subdom ains**CENRA P**  **/ V IST**  **AS / WR**  **AP J**  **ul y**  **Te mp**  **erat ur**  **e Per**  **fo rm**  **ance Com**  **par is**  **on**

0 0.5 1 1.5 2 2.5 3 3.5

-2 -1 .5 -1 -0 .5 0 0.5 1 **Tem per**  **at ur** 

**e Bi**  **as (K**  **) T** **e m** **e p** **r a** **t u** **r e** **E r** **r o** **r (** **K )** 

Be nc hm ark VI ST AS WR AP CE NRA P De se rt SW Pa cN W SW No rt h 

### Lar ge cold bias in western U.S.for all three runs**CENRA P**  **/ V IS**  **TA S**  **/ WR AP**  **J ul**  **y Hum**  **id it**  **y Per**  **fo rm**  **ance Com**  **par is**  **on**

0 0.5 1 1.5 2 2.5 3

-1 .5 -1 -0 .5 0 0.5 1 1.5 2 2.5 **Hum id**  **ity B**  **ia s**  **(g /k**  **g) H** **u m** **i d** **i t** **y E** **r r** **o r** **( g** **/ k** **g )** 

Be nc hm ark VI ST AS WR AP CE NRA P De se rt SW Pa cN W SW No rt h 

### • Lar ge wet bias in WRAP desert SW

and North**CENRA P**  **/ V IST**  **AS / WR**  **AP J**  **ul y**  **Wi nd**  **Per fo**  **rm ance**  **Compar ison** 

0 10 20 30 40 50 60 70 80

0 0.5 1 1.5 2 2.5 3 **Win d**  **Speed RM**  **SE (m/s**  **) W** **i n** **d D** **i e** **r c** **t i** **o n** **E r** **r o** **r (** **m /** **s )** 

Be nc hm ark VI ST AS WR AP CE NRA P NE SE
Oh io V alle y De se rt SW Pa cN W SW No rt h 

### Wi nd speed RMSE and direction error increased relative to January

We stern subdomains

# Overview **of Performance**  **Outside**

**CENRAP Domain**

• 

## For wind, all three runs perform ed similarly . Note that western U.S. subdom ains were outside the benchmark for all seasons for all three runs.

• For temperature WRAP r un had best overall perform ance. Exception is in summer in the west, where it is the outlier
. • however this is partly due to surface t/q nudging which is not desirable

• For hum idity

, all three runs are comparable. Exception is WRAP during sum mer in western U.S., which has a lar ge wet bias associated with excessive rainfall.

• CENRAP has lar gest cold bias in the west during winter , but smallest cold bias in the west during the summer
.

• For hum idity

, most subdomains have a wet bias all year
. We stern subdom ains SW and PacNW are the exceptions and have a dry bias.

# Method for Evaluating **Upper Air**

**Performance**

• Use Matthew Johnson’ s (IDNR) RAOB program to compare observed 0Z and 12Z soundings with MM5 soundings.

• Based on Metstat evaluation, compare soundings for January and July only
. These two months are representative of MM5’ s best and worst performance during the 2002 36 km run. Examined soundings from 2 stations in CENRAP dom ain.

• Analysis is based on inspection of the soundings, and is necessarily subjective

• Tw o sets of soundings will be shown here to summarize results of January and July analysis.

# Comparison of Soundings for Midland AFB, TX for **January 7, 12Z CENRAP**  **VIST AS**  **WRAP**

WRAP T colder than VIST

AS and CENRAP

# Comparison of Soundings for Midland AFB, TX. for **July 16, 0Z .**  **CENRAP VIST**  **AS WRAP** 

PBL top inversion underestimated

# Summary of Upper Air Evaluation

• For July soundings with a deep, convecting boundary layer , CENRAP and VIST

AS frequently better reproduced the observed temperature inversion at the top of the PBL, while WRAP s howed a smoother decrease of temperature with increasing altitude.

• In January and July

, CENRAP

a nd VIST

AS were better able to simulate PBL

temperature inversions than was WRAP
.

• WRAP s urface temperature was generally closer than CENRAP or VIST AS to the observed surface temperature, but profiles above surface were less accurate in WRAP t han in CENRAP

o r VIST AS.

# Summary of Upper Air Evaluation

• CENRAP

and VIST

AS dew point temperature profiles were generally closer than that of WRAP to observed profile, and were better able to handle extreme excursions in temperature and dew point profiles in the lower troposphere.

• Ve rtical wind profiles were very similar , particularly above the boundary layer (this is reasonable since all runs were analysis nudged above the PBL).

• Even in cases where the PBL temperature profiles were dif ferent, wind structure was similar
. This suggests that local thermal processes are driving the temperature dif ferences, rather than advection.

# Method for Evaluating **Precipitation**

**Performance**

• For each model run, calculate the total MM5 precipitation for each hourly output time: To tal Precip = Convective Precip + Lar ge - Scale Precip and sum over a month to obtain a total monthly precipitation amount.

• Compare with CPC 0.25x0.25 degree gridded precipitation analysis.

# January **Precipitation Comparison** 

• Rainfall predicted with good skill in all three runs.

• Rainfall overpredicted in all three runs over cenrapN subdomain
.

# Mar **ch Pr ecipitation**  **Comparison**

• Rainfall predicted with good skill in all three runs.

• Rainfall overpredicted in all three runs over desertSW subdomain
.

• All three runs underpredict the maximum over Arkansas.

# July Pr **ecipitation Comparison** 

• Rainfall overpredicted in all three runs over most of the U.S
.

• Convective rainfall is over

predicted

• Problem most severe in the WRAP r un

• Overprediction of rainfall associated with wet humidity bias seen in surface plots

# October **Precipitation Comparison** 

• Rainfall predicted with fair skill in all three runs.

• Rainfall underpredicted in all three runs over CENRAP region
.

• All three runs underpredict the maximum over Louisiana.

• Metstat shows dry bias in cenrapS subdomain humidity for this month

# December Precipitation **Comparison**

• Rainfall predicted with good skill in all three runs.

• Rainfall overpredicted in all three runs over cenrapN subdomain
.

# Summary of Precipitation **Comparison**

• In winter , all three runs showed considerable skill in reproducing the observed precipitation field.

• In summer all runs had a wet bias, particularly in the southern U.S. This bias was strongest in the WRAP r un.

• This precip bias is consistent with the wet bias in the metstat surface humidity statistics for SW , DesertSW

, and North.

• All model runs produced too much rain over CenrapN region in winter , and underpredicted maxima in the CenrapS region in fall and spring.

# Preliminary Conclusions from 2002 MM5 **Comparison of CENRAP,VISTAS, and WRAP** 

• 

# Overall, **VISTAS performed**  **best. Unclear whether this is due to the large**  **- scale**  **moist physics or convection**  **schemes or to the interaction between them, or to**  **effects of differences**  **in FDDA.**

• 

# CENRAP **performance fell**  **between VISTAS and**  **WRAP.**

• 

# In all **runs, MM5 performed better in winter than in**  **summer.**

• 

# All three runs had excessive **convective rainfall**  **in summer** 

# Preliminary Conclusions from 2002 MM5 **Comparison of CENRAP,VISTAS, and WRAP** 

• 

# Although WRAP **had the best surface statistics for**  **temperature, upper air performance**  **was worst, which**  **suggests that nudging surface temperature is not a**  **good idea.**

• 

# A new **WRAP 2002 36/12 km run has been made to address issues of performance**  **in the western U.S., particularly**  **in summer, and does not include surface analysis nudging of temperature**  **and mixing ratio and**  **the Betts -**  **Miller cumulus scheme.**