Comparison of CENRAP, VISTAS, and 

WRAP 36 km MM5 Model Runs for 2002

Task 3: Meteorological Gatekeeper Report

Susan Kemball-Cook, Yiqin Jia, Chris Emery and Ralph Morris

ENVIRON International Corporation

Zion Wang and Gail Tonnesen

UCR CE-CERT

Western Regional Air Partnership (WRAP)

Regional Modeling Center (RMC)
Model Setup for CENRAP, WRAP, and VISTAS 

Runs

• All use National RPO 36 km Grid, 164 x 128 x 34

• Lambert Conformal Projection

• Pleim-Xiu/ACM LSM/PBL

• RRTM Longwave and Dudhia Shortwave

• No shallow convection
Model Setup for CENRAP, Initial WRAP, and 

VISTAS Runs, continued.

CENRAP VISTAS WRAP1

Cumulus 

Convection

KainFritsch 2KainFritsch 2KainFritsch

Physics Options Differences

Moist Physics Reisner 1 Reisner 1 Reisner 2

Analysis Nudging

Surface

u/v u/v u/v/t/q

Analysis Nudging

Aloft

u/v/t/q u/v/t/q u/v/t/q

1 Final WRAP MM5 used Betts-Miller Cumulus and u/v surface 

analysis nudging
Method for Comparing the CENRAP, 

WRAP, and VISTAS MM5 Runs

• Surface statistics for wind, temperature, and humidity

For CENRAP, WRAP, and VISTAS runs, we examine:

• Surface statistics for wind, temperature, and humidity

• Upper air soundings of temperature, dew point, and winds

• Precipitation

and compare with observations.

Focus on CENRAP domain, and also give overview of performance 

in other regions.
Surface Evaluation Method

• Use Metstat statistical package to compare model surface 

output to surface station data

• Metstat Metstat calculates calculates statistics statistics which describe which describe how well the how well the model 

output fields agree with observations

• Metstat analysis divides U.S. into subdomains to show 

how MM5 performs region by region

• Metstat statistics summarized in scatter plots which contain a

model performance benchmark box as well as results of other

MM5/RAMS runs used for air quality modeling
METSTAT Evaluation Package

• Average observed and predicted

• Absolute Bias and Error

• RMSE

• Index of Agreement (IOA)

• Daily and, where appropriate, Hourly Evaluation

• Statistical Performance Benchmarks

> Based on an analysis of > 30 MM5 and RAMS runs 

> Not meant as a pass/fail test, but to put modeling results 

in the proper perspective

Wind Speed Wind Direction Temperature Humidity

RMSE  2 m/s

Mean Bias  0.5m/s  10  0.5K  1g/kg

Index of Agreement  0.6  0.8  0.6

Gross Error  30  2K  2g/kg
Subdomains for Model Evaluation

1 = Pacific NW

2 = SW

3 = North

4 = Desert SW

5 = CenrapN

6 = CenrapS

7 = Great Lakes

8 = Ohio Valley

9 = SE

10 = NE

11 = MidAtlantic
Datasets for Met Evaluation

• NCAR dataset ds472 airport surface met observations

surface temperature surface pressure sea level pressure visibility

cloud cover height of cloud base maximum 

temperature

minimum 

temperature

dewpoint precipitation sunshine snowfall

snow cover wind direction wind force (speed) u wind component

v wind component wind gust descriptive weather

• Twice-Daily Upper-Air Profile Obs (~120 in US)

> Temperature

> Moisture

v wind component wind gust descriptive weather
• WRAP has smallest error

• cenrapN subdomain

outside goal for CENRAP

run

CENRAP / VISTAS / WRAP January Temperature Performance Comparison 

Over CENRAP Domain

2

2.5

3

3.5

T

e

m

p

erature

Error (K)

• WRAP has smallest error

• cenrapN subdomain

outside goal for CENRAP

run

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

p

erature

Error (K)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S

run
CENRAP / VISTAS / WRAPJanuary Humidity Performance 

Comparison Over CENRAP Domain

2

2.5

3

H

u

midity

E

r

r

o

r (g/k

g)

• Humidity performance is comparable

for all three runs

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

E

r

r

o

r (g/k

g)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP January Wind Performance 

Comparison over CENRAP Domain

50

60

70

80

Win

d

Dir

e

ctio

n

E

r

r

o

r (m/s)

• Wind performance is comparable

for all three runs

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dir

e

ctio

n

E

r

r

o

r (m/s)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP March Temperature Performance 

Comparison Over CENRAP Domain

2

2.5

3

3.5

T

e

m

p

e

r

atu

r

e

E

r

r

o

r (K)

• WRAP has best overall

performance

• cenrapN subdomain

outside goal for CENRAP

run

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

p

e

r

atu

r

e

E

r

r

o

r (K)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP March Humidity Performance 

Comparison Over CENRAP Domain

2

2.5

3

H

u

midity

E

r

r

o

r (g/k

g)

• Performance is comparable

for all three runs

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

E

r

r

o

r (g/k

g)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP March Wind Performance Comparison Over 

CENRAP Domain

40

50

60

70

80

Win

d

Dire

ctio

n

Error (m/s)

CENRAP N

CENRAP S

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dire

ctio

n

Error (m/s)

Benchmark CENRAP VISTAS WRAP i
CENRAP / VISTAS / WRAP July Temperature Performance Comparison Over 

CENRAP Domain

2

2.5

3

3.5

T

e

m

e

prature

Error (K)

• Temperature error is higher

in all runs relative to January

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

e

prature

Error (K)

Benchmark VISTAS WRAP i CENRAP 

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP July Humidity Performance 

Comparison Over CENRAP Domain

2

2.5

3

H

u

midity

E

r

r

o

r (g/k

g)

• Humidity error is higher in

all runs relative to January

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

E

r

r

o

r (g/k

g)

Benchmark VISTAS WRAP CENRAP 

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP July Wind Performance Comparison Over 

CENRAP Domain

40

50

60

70

80

Win

d

Dierctio

n

Error (m/s)

• Wind performance is comparable

for all three runs

0

10

20

30

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dierctio

n

Error (m/s)

Benchmark VISTAS WRAP i CENRAP 

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP October Temperature Performance 

Comparison Over CENRAP Domain

2

2.5

3

3.5

T

e

m

p

e

r

atu

r

e

E

r

r

o

r (K)

• WRAP has smallest error

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

p

e

r

atu

r

e

E

r

r

o

r (K)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP October Humidity Performance 

Comparison Over CENRAP Domain

1.5

2

2.5

3

H

u

midity

Error (g/k

g)

• Performance is comparable

for all three runs

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

Error (g/k

g)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
CENRAP / VISTAS / WRAP October Wind Performance 

Comparison Over CENRAP Domain

50

60

70

80

Win

d

Dir

e

ctio

n

E

r

r

o

r (m/s)

• Wind performance is comparable

for all three runs

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dir

e

ctio

n

E

r

r

o

r (m/s)

Benchmark CENRAP VISTAS WRAP i

CENRAP N

CENRAP S
Summary of CENRAP / VISTAS / WRAP Metstat Statistics

January March May July October December

cenrap N

temperature WRAP WRAP WRAP WRAP WRAP WRAP

humidity WRAP WRAP None VISTAS None WRAP

wind None V/C VISTAS CENRAP VISTAS CENRAP

cenrap S

temperature None WRAP WRAP CENRAP None WRAP

humidity VISTAS VISTAS None None WRAP WRAP

wind None V/W VISTAS CENRAP WRAP None

overall

temperature WRAP WRAP WRAP None WRAP WRAP

humidity None VISTAS None None WRAP WRAP

wind None VISTAS VISTAS CENRAP None WRAP

Entry in table indicates which run performed best in that subdomain

for that field.

Note that initial WRAP run used surface t/q nudging that can improve 

performance statistics but introduce instabilities
Overview of CENRAP/WRAP/VISTAS 

Surface Performance in CENRAP Region

• Overall, performance over CENRAP region was within benchmarks.

• All runs had similar wind and humidity performance.

• CENRAP and VISTAS had larger temperature error than WRAP

• For CENRAP run, the cenrapN subdomain fell outside the 

benchmark in January and December
Overview of CENRAP/WRAP/VISTAS 

Surface Performance in CENRAP Region

• In general, surface performance is better in winter than in 

summer for all three runs.

• Humidity error is higher in the summer than in the winter for all 

three runs; there is wet bias in the winter. Less clear cut in other 

seasons.

• Wind speed RMSE and wind direction error increase going 

from winter to summer, then decrease again as winter 

approaches.

• For all runs, cenrapN subdomain has cold bias in winter and a 

warm bias in summer. Less clear cut for cenrapS.
Performance Outside CENRAP region

• Surface performance had strikingly different characteristics in the 

eastern vs. western U.S. subdomains.

• The following soccer plots emphasize performance issues over 

western U.S common to all three runs.

• Western U.S. subdomains are shown with filled icons. All other 

subdomains denoted by open circles colored according to run.

• When an eastern subdomain falls outside the benchmark, it 

is identified.
CENRAP / VISTAS / WRAP January Temperature Performance Comparison

2

2.5

3

3.5

T

e

m

p

erature

Error (K)

SE

NE cenrapN

•WRAP has smallest error

•Western subdomains have

large cold bias

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

p

erature

Error (K)

Benchmark VISTAS WRAP CENRAP 

ohioval

DesertSW 

PacNW

SW

North
CENRAP / VISTAS / WRAPJanuary Humidity Performance Comparison

1.5

2

2.5

3

H

u

midity

Error (g/k

g)

DesertSW 

• All subdomains lie within benchmark

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

Error (g/k

g)

Benchmark VISTAS WRAP CENRAP 

DesertSW 

PacNW

SW

North
CENRAP / VISTAS / WRAP January Wind Performance Comparison

40

50

60

70

80

Win

d

Dire

ctio

n

Error (m/s)

DesertSW 

PacNW

SW

North

• Wind performance does not meet

benchmark in any western U.S. subdomain

Western subdomains

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dire

ctio

n

Error (m/s)

Benchmark VISTAS WRAP CENRAP 

NE

Western subdomains
CENRAP / VISTAS / WRAP July Temperature Performance Comparison

2

2.5

3

3.5

T

e

m

e

prature

Error (K)

Large cold bias in western U.S.for all three runs

0

0.5

1

1.5

-2 -1.5 -1 -0.5 0 0.5 1

Temperature Bias (K)

T

e

m

e

prature

Error (K)

Benchmark VISTAS WRAP CENRAP 

DesertSW 

PacNW

SW

North
CENRAP / VISTAS / WRAP July Humidity Performance Comparison

1.5

2

2.5

3

H

u

midity

Error (g/k

g)

DesertSW 

• Large wet bias in WRAP

desert SW and North

0

0.5

1

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

Humidity Bias (g/kg)

H

u

midity

Error (g/k

g)

Benchmark VISTAS WRAP CENRAP 

DesertSW 

PacNW

SW

North
CENRAP / VISTAS / WRAP July Wind Performance Comparison

40

50

60

70

80

Win

d

Dierctio

n

Error (m/s)

SE

DesertSW 

PacNW

SW

North

Wind speed RMSE and direction error

increased relative to January

0

10

20

30

0 0.5 1 1.5 2 2.5 3

Wind Speed RMSE (m/s)

Win

d

Dierctio

n

Error (m/s)

Benchmark VISTAS WRAP CENRAP 

NE Ohio Valley

Western subdomains
Overview of Performance Outside 

CENRAP Domain

• For wind, all three runs performed similarly. Note that western U.S. subdomains 

were outside the benchmark for all seasons for all three runs.

• For temperature WRAP run had best overall performance. Exception is 

in summer in the west, where it is the outlier. 

• however this is partly due to surface t/q nudging which is not desirable

• For humidity, all three runs are comparable. Exception is WRAP

during summer in western U.S., which has a large wet bias associated with

excessive rainfall.

• CENRAP has largest cold bias in the west during winter, but smallest

cold bias in the west during the summer.

• For humidity, most subdomains have a wet bias all year. Western subdomains 

SW and PacNW are the exceptions and have a dry bias.
Method for Evaluating Upper Air 

Performance

• Use Matthew Johnson’s (IDNR) RAOB program to compare

observed 0Z and 12Z soundings with MM5 soundings.

• Based on Metstat evaluation, compare soundings for January

and July only. These two months are representative of MM5’s

best and worst performance during the 2002 36 km run. 

Examined soundings from 2 stations in CENRAP domain.

• Analysis is based on inspection of the soundings, and is 

necessarily subjective

• Two sets of soundings will be shown here to summarize 

results of January and July analysis.
Comparison of Soundings for Midland AFB, TX for 

January 7, 12Z

CENRAP VISTAS WRAP

WRAP T colder than VISTAS

and CENRAP
Comparison of Soundings for Midland AFB, TX. for

July 16, 0Z.

CENRAP VISTAS WRAP

PBL top inversion

underestimated
Summary of Upper Air Evaluation

• For July soundings with a deep, convecting boundary layer,

CENRAP and VISTAS frequently better reproduced the observed 

temperature inversion at the top of the PBL, while WRAP showed

a smoother decrease of temperature with increasing altitude.

• In January and July, CENRAP and VISTAS were better able to 

simulate PBL temperature inversions than was WRAP. 

• WRAP surface temperature was generally closer than CENRAP

or VISTAS to the observed surface temperature, but profiles above 

surface were less accurate in WRAP than in CENRAP or VISTAS.
Summary of Upper Air Evaluation

• CENRAP and VISTAS dew point temperature profiles were 

generally closer than that of WRAP to observed profile, and were 

better able to handle extreme excursions in temperature and dew

point profiles in the lower troposphere.

• Vertical wind profiles were very similar, particularly above the

boundary layer (this is reasonable since all runs were analysis

nudged above the PBL).

• Even in cases where the PBL temperature profiles were different,

wind structure was similar. This suggests that local thermal 

processes are driving the temperature differences, rather than 

advection.
Method for Evaluating Precipitation 

Performance

• For each model run, calculate the total MM5 precipitation for

each hourly output time:

Total Precip = Convective Precip + Large-Scale Precip

and sum over a month to obtain a total monthly precipitation

amount.

• Compare with CPC 0.25x0.25 degree gridded precipitation

analysis.
January Precipitation Comparison

• Rainfall predicted with

good skill in all three runs.

• Rainfall overpredicted

in all three runs over

cenrapN subdomain.
March Precipitation Comparison

• Rainfall predicted with

good skill in all three runs.

• Rainfall overpredicted

in all three runs over

desertSW subdomain.

• All three runs underpredict

the maximum over 

Arkansas.
July Precipitation Comparison

• Rainfall overpredicted in all

three runs over most of the

U.S.

• Convective rainfall is overpredicted

• Problem most severe in the

WRAP run

• Overprediction of rainfall

associated with wet humidity

bias seen in surface plots
October Precipitation Comparison

• Rainfall predicted with

fair skill in all three runs.

• Rainfall underpredicted

in all three runs over

CENRAP region.

• All three runs underpredict

the maximum over 

Louisiana.

• Metstat shows dry bias in

cenrapS subdomain

humidity for this month
December Precipitation Comparison

• Rainfall predicted with

good skill in all three runs.

• Rainfall overpredicted

in all three runs over

cenrapN subdomain.
Summary of Precipitation Comparison

• In winter, all three runs showed considerable skill in 

reproducing the observed precipitation field.

• In summer all runs had a wet bias, particularly in the southern

U.S. This bias was strongest in the WRAP run.

• This precip bias is consistent with the wet bias in the metstat 

surface humidity statistics for SW, DesertSW, and North.

• All model runs produced too much rain over CenrapN

region in winter, and underpredicted maxima in the CenrapS

region in fall and spring.
Preliminary Conclusions from 2002 MM5 

Comparison of CENRAP,VISTAS, and WRAP

• Overall, VISTAS performed best. Unclear whether this 

is due to the large-scale moist physics or convection 

schemes or to the interaction between them, or to 

effects of differences in FDDA.

• CENRAP performance fell between VISTAS and 

WRAP.

• In all runs, MM5 performed better in winter than in 

summer.

• All three runs had excessive convective rainfall in 

summer
Preliminary Conclusions from 2002 MM5 

Comparison of CENRAP,VISTAS, and WRAP

• Although WRAP had the best surface statistics for 

temperature, upper air performance was worst, which 

suggests that nudging surface temperature is not a 

good idea.

• A new WRAP 2002 36/12 km run has been made to 

address issues of performance in the western U.S., 

particularly in summer, and does not include surface 

analysis nudging of temperature and mixing ratio and 

the Betts-Miller cumulus scheme.