NASAlCR-20 1 0-216287 

Verify MesoNAM Performance 

William H Bauman III 

ENSCO, Inc. , Cocoa Beach, Florida 

NASA Applied Meteorology Unit, Kennedy Space Center, Florida 

September 2010 
NASA STI Program ... in Profile 

Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space 

science. The NASA scientific and technical 

information (ST!) program plays a key part in 

helping NASA maintain this important role. 

The NASA STI program operates under the 

auspices of the Agency Chief Information 

Officer. It collects, organizes, provides for 

archiving, and disseminates NASA's STI. The 

NASA STI program provides access to the NASA 

Aeronautics and Space Database and its public 

interface, the NASA Technical Report Server, 

thus providing one of the largest collections of 

aeronautical and space science STI in the world. 

Results are published in both non-NASA channels 

and by NASA in the NASA STI Report Series, 

which includes the following report types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant 

phase of research that present the results of 

NASA Programs and include extensive data 

or theoretical analysis. Includes compilations 

of significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers but has 

less stringent limitations on manuscript length 

and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 

and technical [mdings that are preliminary or 

of specialized interest, e.g., quick release 

reports, working papers, and bibliographies 

that contain minimal annotation. Does not 

contain extensive analysis. 

• CONTRACTOR REPORT. Scientific and 

technical [mdings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored 

by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, 

often concerned with subjects having 

substantial public interest. 

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific 

and technical material pertinent to 

NASA's mission. 

Specialized services also include organizing and 

publishing research results, distributing 

specialized research announcements and feeds, 

providing help desk and personal search support, 

and enabling data exchange services. 

For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home page 

at http://www.sti.nasa.gov 

• E-mail your question via the Internet to 

help@stLnasa.gov 

• Fax your question to the NASA STI Help 

Desk at 443-757-5802 

• Phone the NASA STI Help Desk at 

443-757-5802 

• Write to: 

NASA STI Help Desk 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 
NASAICR -201 0-216287 

Verify MesoNAM Performance 

William H Bauman III 

ENSCO, Inc., Cocoa Beach, Florida 

NASA Applied Meteorology Unit, Kennedy Space Center, Florida 

National Aeronautics and 

Space Administration 

Kennedy Space Center 

Kennedy Space Center, FL 32899-0001 

September 2010 
Acknowledgements 

The author thanks Mr. Randy Nyman of ACTA, Inc. for providing archived MesoNAM forecast files and 

Mr. William Roeder of the 45th Weather Squadron for providing his statistical expertise to this project. 

Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

This report is also available in electronic form at 

http://science.ksc.nasa.gov/amu/ 
Executive Summary 

The 45th Weather Squadron (45 WS) Launch Weather Officers (LWOs) use the 12-km resolution 

North American Mesoscale (NAM) model (MesoNAM) forecasts extensively to support launch weather 

operations. However, the actual performance of the model at Kennedy Space Center (KSC) and Cape 

Canaveral Air Force Station (CCAFS) has not been measured object~vely. In order to have tangible 

evidence of model performance, the 45 WS tasked the Applied Meteorology Unit (AMU) to conduct a 

detailed statistical analysis of model output compared to observed values. The analysis compared the 

MesoNAM forecast winds, temperature and dew point to the observed values from the sensors on 

specified KSC/CCAFS wind towers. 

The AMU calculated the following statistics to verify the performance of the model: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = O. 

The AMU calculated the verification statistics once the files were properly formatted and stratified. 

First the difference between all hourly model forecasts and tower observations was calculated and then 

the mean monthly bias, the standard deviation of the mean monthly bias and then the RMSE. Finally, the 

hypothesis zero test was applied to the mean monthly bias statistics. 

The model's performance statistics were calculated for each wind tower based on sensor height and 

model initialization time. The data were stratified by month and onshore/offshore wind direction based on 

the orientation of the coastline to each tower's location prior to the statistics calculations. 

The 45 WS requested the AMU only calculate statistics for the current MesoNAM which became 

operational in mid-August 2006 and was based on the Weather Research and Forecasting model. 

Therefore the period of record for the data used in this task began with the first full month of data, 

September 2006, through May 2010. 

The model performance indicated: 

• The accuracy decreased as the forecast valid time from the model initialization increased, 

• There was a diurnal signal in temperature with a cool bias during the late night and a warm bias 

during the afternoon, 

• There was a diurnal signal in dew point temperature with a low bias during the afternoon and a 

high bias during the late night, and 

• The model parameters at each vertical level most closely matched the observed parameters at 

heights closest to those vertical levels. 

The AMU developed a graphical user interface (GUI) that consists of a multi-level drop-down menu 

written in JavaScript embedded within Hyper-Text Markup Language code that can be used with most 

popular web browsers. This GUI allows the L WO to easily and efficiently navigate among the charts and 

spreadsheet files. 

The objective statistics provide the forecasters knowledge of the model's strengths and weaknesses 

and the GUI allows quick access to the data which will result in improved forecasts for operations. 
Table of Contents 

Executive Summary ...................................................................................................................................... 1 

List of Figures ............................................................................................................................................... 3 

List of Tables ................................................................................................................................................. 5 

1. Introduction ............................................................................................................................................ 6 

2. Data .............................................................................................. ........................................................... 7 

2.1 Tower Observations ...................................................................................................................... 7 

2.2 MesoNAM Forecast Products ....................................................................................................... 8 

2.3 Data Stratification ......................................................................................................................... 8 

3. File Formats ............................................................................................................................................ 9 

3.1 Tower Observation Files ....... ........................................................................................................ 9 

3.2 MesoNAM Forecast Files ........................................................................................................... 10 

3.3 Merged Files ............................................................................................................................... 10 

4. Verification Statistics .............................................................. .............................................................. 11 

4.1 Statistics ...................................................................................................................................... 11 

4.2 Difference Calculations .............................................................................................................. 11 

4.3 Mean Monthly Model Bias Calculations .................................................................................... 16 

4.4 Standard Deviation of Mean Monthly Bias Calculations ........................................................... 16 

4.5 Root Mean Square Error Calculations ........................................................................................ 17 

4.6 Hypothesis Zero Test .................................................................................................................. 17 

4.7 Statistics Values .......................................................................................................................... 18 

5. Model Performance .............................................................................................................................. 21 

5.1 Accuracy ..................................................................................................................................... 21 

5.2 Diurnal Signal ............................................................................................................................. 21 

5.3 Model Parameter Vertical Level and Sensor Height .................................................................. 22 

6. MesoNAM Verification Tool Graphical User Interface ....................................................................... 24 

7. Summary and Conclusions ................................................................................................................... 28 

References ................................................................................................................................................... 29 

2 
List of Figures 

Figure 1. Map of KSC/CCAFS showing the locations of the wind towers used to verify 

MesoNAM forecasts (red pentagons labeled with tower number and the supported 

launch activity), the locations of the MesoNAM model grid points (green circles with 

black dot) and CCAFS weather station (magenta square labeled KXMR). The 

MesoNAM land/sea mask is shown by the magenta line. The model assumes only land 

is to the left of the land/sea mask . ............................................................................................... 7 

Figure 2. Depiction of column headers for tower observation data contained in the raw data files ........... 9 

Figure 3. As in Figure 2 but for the tower observation data as formatted by the AMU QC program ........ 9 

Figure 4. Column format for observation data from an individual tower sensor after hourly mean 

values were calculated and the files were exported to Excel format. .......................................... 9 

Figure 5. Depiction of column headers for MesoNAM forecast data contained in the raw data 

files .......................................................................................................................................... 10 

Figure 6. As in Figure 5 but for the MesoNAM forecast data after fmal reformatting ............................ 10 

Figure 7. Depiction of column headers for files containing merged tower observation data and 

MesoNAM forecast data. The observation data was located in columns A-H and the 

MesoNAM data was located in columns J-N ............................................................................ 10 

Figure 8. Dual-sensor configuration on Towers 002, 006 and 110. Sensors are mounted on the 

NW and SE side of each tower. The NW sensor is always selected when the winds are 

between 249° and 22° (light red arc) and the SE sensor is always selected when the 

winds are between 69° and 203° (light green arc). Either sensor can be used in the two 

gray regions when the winds are between 23° and 68° or 204° and 248° ................................. 12 

Figure 9. Map of KSC/CCAFS showing the locations of the wind towers used to verify 

MesoNAM forecasts (red pentagons labeled with tower number and the supported 

launch activity) and the dual-sensor towers (green ellipses). The onshore and offshore 

stratifications were determined by the coastline alignment relative to the Atlantic Ocean 

for each tower ............................................................................................................................ 13 

Figure 10. Example of a merged Excel workbook file showing the tower observations from Tower 

002 (columns A-H), MesoNAM forecasts (columns J-N) initialized at 00 UTC (OOZ tab 

at bottom left), and difference calculations (columns P-S) for an offshore stratification ......... 14 

Figure 11. Representation of the sensor configuration for onshore and offshore stratifications on the 

dual-sensor Tower 002. The NW sensor is used for offshore flow stratification while the 

SE sensor is used for onshore flow stratification ...................................................................... 15 

Figure 12. As in Figure 11 but for the dual-sensor Towers 006 and 110. Both the NW and SE 

sensors must be used for onshore and offshore flow stratifications. The gray dashed line 

represents the divide between the data used from the NW sensor and from SE sensor ............ 15 

Figure 13. As in Figure 11 but for the single-sensor towers. A single sensor was used for both 

onshore and offshore stratifications .......................................................................................... 16 

Figure 16. An example of a verification chart showing model bias of T (blue line) and hypothesis 

zero test = true (light green shaded regions) from a 12 UTC model initialization using 

observations from Tower 2 at 54 ft for November with an onshore stratification. The red 

and purple ellipses highlight the region ofthe chart in the example discussed in the text ........ 18 

Figure 14. Example of a master Excel workbook file used to calculate the mean monthly statistics. 

Column A contains tower, sensor identification and model initialization information, 

3 
column B shows the model forecast hour, columns C-P include the calculated statistics 

and the rest ofthe spreadsheet holds the hourly difference calculations ................................... 19 

Figure 15. Example of a master Excel workbook file used to calculate the mean monthly statistics. 

The graphs were automatically populated by the statistics calculated in the spreadsheet 

and show, from top to bottom, model bias, model standard deviation of bias and model 

root mean square error ..................................... .......................................................................... 20 

Figure 17. An example of verification charts illustrating the model's increasing error (solid black 

trend lines) for (a) the model standard deviation of wind speed bias (green line) and (b) 

standard deviation of wind direction bias (orange line) using observations from Tower 

002 at 54 ft for October with an offshore stratification ............................................................. 21 

Figure 18. An example of verification charts illustrating the model's diurnal signal for the model 

bias ofT (blue line) and hypothesis zero test = true (light green shaded regions) (a) from 

a 00 UTC model initialization and (b) from a 12 UTC model initialization using 

observations from Tower 006 at 6 ft for May with an onshore stratification ............................ 22 

Figure 19. As in Figure 18 but for Td ......................................................................................................... 22 

Figure 20. An example of verification charts illustrating how the model's bias varies with respect to 

tower sensor height for the model bias of T (blue line) and hypothesis zero test = true 

(light green shaded regions) from (a) the 6 ft sensor height and (b) the 54 ft sensor 

height using observations from Tower 002 for June with an onshore stratification ................. 23 

Figure 21. An example of verification charts illustrating how the model's bias varies with respect to 

tower sensor height for (a) the model bias of wind speed (green line) from the 54 ft 

sensor height and (b) the 12 ft sensor height using observations from Tower 2 for June 

with an onshore stratification . ................................................................................................... 23 

Figure 22. Main page of the GUI in a web browser showing the basic layout of the GUI.. ....................... 24 

Figure 23. Main page of the GUI illustrating the functionality of the drop-down menu (heavy blue 

polygon region) ......................................................................................................................... 25 

Figure 24. A page from the GUI displaying four charts of model T bias and standard deviation of 

bias and model Td bias and standard deviation of bias for Tower 002 at a sensor height 

of 54 ft for a 12 UTC model initialization for November. The blue lines characterize T 

and the red lines T d. The green shaded regions correspond to a "true" hypothesis zero 

test. The icon representing a link to access the Excel file used to create the charts is 

shown within the red ellipse under the blue menu bar .............................................................. 26 

Figure 25. The "Help/About" page from the GUI. ..................................................................................... 27 

4 
List of Tables 

Table 1. Towers, launch activities and sensor heights at KSC and CCAFS used in the objective 

analysis to verify the MesoNAM forecasts ................................................................................... 6 

5 
,

----

1. Introduction 

The 45th Weather Squadron (45 WS) Launch Weather Officers (LWOs) use the 12-km resolution 

North American Mesoscale (NAM) model (MesoNAM) text and graphical product forecasts extensively 

to support launch weather operations. However, the actual performance of the model at Kennedy Space 

Center (KSC) and Cape Canaveral Air Force Station (CCAFS) has not been measured objectively. In 

order to have tangible evidence of model performance, the 45 WS tasked the Applied Meteorology Unit 

(AMU; Bauman et al. 2004) to conduct a detailed statistical analysis of model output compared to 

observed values. 

The model products are provided to the 45 WS by ACTA, Inc. and include hourly forecasts from 0 to 

84 hours based on model initialization times of 00, 06, 12 and 18 UTC. The objective analysis compared 

the MesoNAM forecast winds, temperature (T) and dew point (Td), as well as the changes in these 

parameters over time, to the observed values from the sensors in the KSC/CCAFS wind tower network 

shown in Table 1. The L WOs support launch and landing activities using the sensors in the column 

labeled "Required Sensor Heights ". The other sensors present on the towers are shown in the column 

labeled "Additional Sensor Heights " and were included in this work for completeness. The objective 

statistics calculated will give the LWOs knowledge of the model's strengths and weaknesses based on the 

observations at individual wind towers, which will result in improved forecasts for operations. 

6 

Table 1. Towers. launch activities and sensor heights at KSC and CCAFS used in the objective 

analysis to verify the MesoNAM forecasts. 

Tower Number Supported Actlvlty Required Sensor A ddltlona' Sensor 

and Facility Heights Heights 

002 Delta II (LC-17) 6 ft. 54 ft, 90 ft 145 ft, 204 ft 

006 Delta IV (LC-37)lFalcon 9 (LC-40) 54ft 6 ft, 12 ft, 162 ft, 204 ft 

108 Delta IV (LC-40)lFalcon 9 (LC-40) 54ft 6 ft, 12 ft 

110 Atlas VlFalcon 9 (LC-40) 54 ft, 162 ft, 204 ft 6 ft, 12 ft 

041 Atlas V (LC-41) 230 ft 

393/394 Shuttle (LC-39A) 60ft 

3971398 Shuttle (LC-39B) 60ft 

511/5121513 Shuttle Landing Facility 6 ft, 30 ft 
.

------------------------------

2. Data 

The period of record (POR) for the data set was September 2006 - January 2010. The AMU chose 

September 2006 as the beginning of the POR because the 12-km Weather Research and Forecasting 

(WRF) model became the operational MesoNAM model in mid-August 2006 and the 45 WS required the 

AMU conduct the model verification using the "current operational model". 

2.1 Tower Observations 

The AMU used the KSC/CCAFS wind tower data from the AMU archive and the AMU wind tower 

quality control (QC) software to remove erroneous observations from the dataset. Scripts were written in 

S-PLUS® (Insightful Corporation 2007) to import and modify the QC'd wind tower observation files to 

remove unneeded time periods, observed parameters and sensor heights from the dataset for each tower; 

and to reformat the files to prepare them for merger with the MesoNAM forecast files. The locations of 

the towers used for the verification are shown on the map of KSC/CCAFS in Figure 1 . 

• 

• 

AtJ.ntlc Oc •• n 

• 

Figure 1. Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM 

forecasts (red pentagons labeled with tower number and the supported launch activity), the locations of 

the MesoNAM model grid points (green circles with black dot) and CCAFS weather station (magenta 

square labeled KXMR). The MesoNAM land/sea mask is shown by the magenta line. The model assumes 

only land is to the left of the land/sea mask. 

Since the tower data were reported every 5 minutes and the MesoNAM forecasts were hourly, the 

45 WS requested the AMU calculate the mean value for each observed parameter from the tower data at 

the top of every hour using the observations from 30 minutes prior and 30 minutes after the hour. The 

S-PLUS scripts were written to reformat the tower data and calculate the mean values in this manner. 

7 
2.2 MesoNAM Forecast Products 

The AMU requested and obtained the archived MesoNAM forecasts from ACTA, Inc. The 

MesoNAM forecast files were provided to the AMU as space-delimited text files. Each file contained the 

initialization and hourly forecasts to 84 hours at a single model initial time of 00, 06, 12 or 18 UTC. The 

data were valid at a single point extracted from the model forecasts, identified as "KXMR" in each file. 

This represents the location of the CCAFS weather station, which is located near the center of CCAFS 

and is identified in Figure 1 by the magenta square below the CCAFS label. The closest model grid point, 

which represents the point data used by the 45 WS, is located 3.1 NM southwest of KXMR over the 

Banana River. It is shown in Figure 1 as a green circle with a black dot and labeled "NAM". Because the 

grid point is to the left of the model land/sea mask, the model assumes the point is over land. 

An inventory of the MesoNAM files in the POR revealed 128 missing files, or model runs, out of a 

possible 4996 files for the 1249 days. Some days were missing less than four model runs while others 

were missing all four model runs. The result was a total of 1216 days containing at least one model run. 

2.3 Data Stratification 

The 45 WS requested the results be stratified by 45°, 90°, 180° and 360° wind sectors, year, warm 

season (May-September), cool season (October-April), month and model initialization time. They also 

requested the sector directions be oriented to discriminate between onshore/offshore flow for each tower 

and to analyze the model forecasts only for the current operational version of the MesoNAM. 

After discussions with the 45 WS, the requirement for yearly, warm season and cool season 

stratifications was rescinded since they believed the monthly stratification would be a better time period 

to support launch and landing operations. Based on this determination, starting with the smallest wind 

sector stratification of 45°, the AMU calculated bias statistics for three towers to determine if the sample 

size was sufficient to be statistically significant, and found it was not. The same was done for the 90° 

wind sector stratification and the sample size was also too small. Testing a wind sector stratification of 

180° indicated the results would produce sufficient sample sizes to calculate monthly statistics. After the 

AMU provided these test results to the 45 WS, they decided the best 180° stratification for each tower 

would be calculated as onshore and offshore based on the orientation of the coastline to each tower's 

location. 

8 
3. File Formats 

A significant amount of file format manipulation was required in order to compute the required 

statistics between the observation and model forecast files. In both the observation and forecast data, there 

were more parameters in the files than required to compute the verification statistics. In order to more 

efficiently compute the statistics for this large data set, the AMU removed the unneeded parameters, 

which reduced file sizes, and then reformatted the data within the spreadsheets to reduce the complexity 

of the task. The goal was to combine the observed and forecast data into Excel format files to compute the 

statistics and provide the results in a familiar format to the L WOs. 

3.1 Tower Observation Files 

The raw tower observation files were available in space-delimited text file format. Each file 

represented one tower and included all sensor heights at all locations (NW and SE side of tower) on that 

tower. As shown in Figure 2, the format of the files included columns containing, in order left to right, 

date (year, month, day), time (hour, minute, second), tower identifier (3-digit number), sensor location, 

sensor height (ft), mean wind direction, mean wind speed (ms"\ peak wind direction, peak wind speed 

(ms"\ T (0C), Td (0C) and relative humidity (%). 

~ ABC 0 E F G H I K l 

1 Year/Month/Day nme (HMSI Tower 10 Sensor loe Sensor Hgt Direction Spd (m/sl PDlredion PSpd (m/sl T (e) Td (e) RH ("I 

Figure 2. Depiction of column headers for tower observation data contained in the raw data files. 

After running the AMU QC program, the tower files were reformatted and output as space-delimited 

text files and each file contained all sensor heights on a tower on one side of the tower (NW or SE). 

Figure 3 shows the format of the new QC files that included columns containing, in order left to right, 

tower identifier (3-digit number), year, day, hour, minute (5-minute intervals), sensor height (ft), T (OF), 

Td (OF), mean wind speed (kt), mean wind direction, peak wind speed (kt), peak wind direction and 

relative humidity (%). 

B C E F 

Year Month Hour Minute Sensor Hat 

The S-PLUS scripts mentioned in Section 2.1 were run on the QC'd files to calculate the mean values 

of wind speed, wind direction, T and Td at the top of every hour using the observations from 30 minutes 

prior and 30 minutes after the hour and to output the resulting files in Excel format - one file for every 

month for each year in the POR. Each file contained data for one sensor height at one sensor location per 

tower. Figure 4 shows the format of the files with columns containing, in order left to right, year, month, 

day, hour, mean wind speed (kt), mean wind direction, mean T (OF) and mean Td (OF). In preparation for 

merger with the MesoNAM forecast files, the AMU wrote Microsoft Excel Visual Basic for Applications 

(VBA) scripts to process the hourly mean files into Excel workbook files containing four spreadsheets per 

workbook. Each spreadsheet represented the MesoNAM model initialization times of 00, 06, 12 and 18 

UTC and contained 84 hourly mean observations to match the 84 hours of model forecasts per model 

initialization. 

It A 8 C ""-'_',;84;48.:..4,.;. 

Figure 4. Column format for observation data from an individual 

tower sensor after hourly mean values were calculated and the 

files were exported to Excel format. 

9 
3.2 MesoNAM Forecast Files 

The AMU wrote VBA scripts to import the MesoNAM files into Excel spreadsheets and reformatted 

them to merge with the wind tower observation spreadsheets. As shown in Figure 5, the format of the raw 

MesoNAM files included columns containing, in order left to right, date (year, month, day/hour, minute), 

T (0C), Td (0C), wind speed (kt), wind direction, mean sea level pressure (mb), low cloud (% coverage), 

mid cloud (% coverage) and high cloud (% coverage). 

~ AB C 0 E F G HI 

1 VYMMDD/HHMM TIe) Td Ie) Spd 11et) Direction PMSI. Lan MelD HeLD 

Figure 5. Depiction of column headers for MesoNAM forecast data 

contained in the raw data files. 

Reformatting the raw MesoNAM files included converting the temperature and dew point from 

Celsius to Fahrenheit and moving rows and columns in the MesoNAM spreadsheets to merge with the 

wind tower spreadsheets. Figure 6 shows the final format of the column headers depicting the model 

forecast data that was used in the statistical verification containing, in order left to right, date (year, 

month, day, hour), wind speed (kt), wind direction, T COF), Td (OF) 

3.3 Merged Files 

A B C o E 

YYMMIlIlHH WRI ~pn WRIIlIl WRI I WRI Tn 

Figure 6. As in Figure 5 but for the 

MesoNAM forecast data after final 

reformatting. 

The AMU wrote VBA scripts to create an Excel workbook for each of the 1216 days with at least one 

model run. The scripts opened the tower observation Excel files and MesoNAM Excel files and merged 

them to create workbooks that included up to four spreadsheets per workbook, one for each available 

model run, containing the combined tower observations and MesoNAM forecasts for each sensor on 

every tower. This resulted in a total of 57,124 workbooks. As shown in Figure 7, the format of the merged 

files included columns containing, in order left to right, year, month, day, hour, mean wind speed (kt), 

mean wind direction, mean T (OF) and mean Td (OF) of the tower observations and date (year, month, day, 

hour), wind speed (kt), wind direction, T COF), Td (OF) of the MesoNAM forecasts. 

.oil A 8 c o F G H J K L M 

Figure 7. Depiction of column headers for files containing merged tower observation data and 

MesoNAM forecast data. The observation data was located in columns A-H and the MesoNAM data was 

located in columns J-N. 

10 
,---------------------- ----

4. Verification Statistics 

The AMU calculated the verification statistics once the files were properly formatted and stratified. 

First the difference between all hourly model forecasts and tower observations was calculated and then 

the mean monthly bias, the standard deviation of the mean monthly bias and then the root mean square 

error (RMSE). Finally, the hypothesis zero test was applied to the mean monthly bias statistics. 

4.1 Statistics 

The 45 WS requested the AMU calculate the following statistics to verify the performance of the 

model against the observations: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = O. 

4.2 Difference Calculations 

The model "bias" was calculated for each model forecast against every observation, which 

represented the difference between the MesoNAM forecast parameters and the tower observation 

parameters. This difference was calculated by subtracting the observed parameter from the model forecast 

for each hour of every day in the POR stratified by tower, sensor height and onshore/offshore flow as 

shown by the equation below. 

DifMesoNAM = Model Forecast - Tower Observation 

4.2.1 NW and SE Sensors 

Before making the difference calculations, the AMU needed to stratify the data into onshore and 

offshore components. Special care was required when stratifying the observations at Towers 002, 006 and 

110 because they have dual sensors located on the NW and SE side of each tower at the same height on 

the tower. Figure 8 depicts which sensor is chosen to report real-time observations to the forecaster 

display based on the observed wind direction. The NW sensor is always used when the winds are between 

249° and 22° (thick light red arc) and the SE sensor is always used when the winds are between 69° and 

203° (thick light green arc). However, both sensors can display observations in the gray arcs, 23° to 68° 

or 204° to 248°. The sensor used in the gray arc is based on the last sensor used before the winds were in 

the gray arc. For example, if the SE sensor was the previously selected sensor based on wind direction, 

then that sensor will continue to be the selected sensor until winds are observed between 248° and 23°. 

Conversely, if the NW sensor was the previously selected sensor, it will remain so until the winds are 

observed between 68° and 204°. Data from both sensors are saved for archive regardless of wind 

direction. For this task, the NW sensor was used for winds between 226° and 45° (thin red arc in Figure 

8) and the SE sensor was used for winds between 46° and 225°(thin green arc in Figure 8). 

11 
NWSensor 

SE Sensor 

2040 

Figure 8. Dual-sensor configuration on Towers 002, 006 and 110. 

Sensors are mounted on the NW and SE side of each tower. The NW 

sensor is always selected when the winds are between 249° and 22° 

(light red arc) and the SE sensor is always selected when the winds are 

between 69° and 203° (light green arc). Either sensor can be used in the 

two gray regions when the winds are between 23° and 68° or 204° and 

248°. 

4.2.2 Onshore/Offshore Directions 

Figure 9 shows the locations of Towers 002, 006 and 110 (identified by the green ellipses). The 

coastline orientation in the vicinity of Tower 002 is northeast to southwest. Therefore, the onshore 

stratification only used data from the SE sensor and the offshore stratification only used data from the 

NW sensor. Towers 006 and 110 are affected by a coastline orientation from NW to SE. Therefore, their 

onshore stratification required using the NW sensor for winds from 316° to 45° and the SE sensor for 

winds from 46° to 135°. Conversely, their offshore stratification required using the NW sensor for winds 

from 226° to 315° and the SE sensor for winds from 136° to 225°. To summarize the sensor 

configurations for onshore and offshore flow for the dual-sensor towers: 

For Tower 002: 

• Onshore: SE sensor 46° to 225° 

• Offshore: NW sensor 226° to 45° 

For Towers 006 and 110: 

• Onshore: NW sensor 316° to 45° and SE sensor,46° to 135° 

• Offshore: NW sensor 226° to 315° and SE sensor 136° to 225° 

The AMU developed VBA scripts that select the appropriate sensors for the onshore and offshore 

calculations for the dual-sensor towers. 

12 
Indian River 

Figure 9. Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM 

forecasts (red pentagons labeled with tower number and the supported launch activity) and the dualsensor towers (green ellipses). The onshore and offshore stratifications were determined by the coastline 

alignment relative to the Atlantic Ocean for each tower. 

The wind tower configurations at the Shuttle pads presented a similar challenge to computing onshore 

and offshore statistics as for the dual-sensor towers. As Figure 9 shows, Tower 393 (Pad 39A) and Tower 

397 (Pad 39B) are on the NW side of each pad and have sensors mounted on the NW side of each tower. 

Tower 394 (Pad 39A) and Tower 398 (Pad 39B) are on the SE side of each pad and have sensors mounted 

on the SE side of each tower. This sensor configuration is similar to that described above for the dualsensor towers except the sensors are not on the same tower. The AMU also developed VBA scripts to 

extract the appropriate data from each of these towers and then combine the data at each pad to compute 

statistics for the onshore and offshore components. 

4.2.3 Onshore/Offshore Differences 

Using the Excel files with the merged tower observations and MesoNAM forecasts, the AMUdeveloped VBA scripts calculated the difference of wind speed, wind direction, T and Td based on the 

onshore and offshore wind direction stratifications. The scripts opened a merged Excel workbook file, 

calculated the difference and entered the results in columns P-S. Figure 10 shows an example of an 

offshore stratification worksheet from Tower 002. The difference was not calculated in rows 23-25 and 

47-51 because the mean wind direction was onshore at those times, as shown in the green ellipses in 

columnF. 

13 
A B C 0 E G H K M N 0 P Q R S 

2 2007 .. 0 3.3 5 66.2 45.5 010407I0000 5.9 J04 65.5 30.6 2.6 -60.9 .j).7 -14.9 

3 2001 4 1 U 351 65.1 46.1 01040110100 9.3 287 62.6 34.2 U 63.5 -2.5 -12.0 

4 2007 .. 2 6.1 316 63.4 47.9 01040110200 12.1 282 60.3 32.7 6.0 34.1 -3.2 -15.2 

5 2007 4 3 11.1 288 62.8 44.0 01040110300 16.1 282 60.3 34.2 5.6 6.3 -2.5 -9.9 

6 2007 4 8.9 291 60.6 45.5 01040710400 16.1 285 59.0 35.6 7.8 6.2 -1.6 -9.9 

1 2001 5 8.6 299 59.7 43.6 01040110500 16.2 296 58.1 36.5 7.6 2.6 -1.6 -7.1 

8 2001 6 U 302 58.6 43.0 010401/0600 15.9 291 56.5 35.1 1.5 5.1 -2.2 -8.0 

9 2007 7 9.4 308 58.4 42.3 01040110700 16.2 J04 55.2 33.1 U 3.9 -].2 -9.2 

10 2007 8 9.1 321 57.6 40.6 010401/0800 17.9 ]1] 55.0 31.3 8.2 8.1 -2.6 -9.3 

11 2007 9 9.8 326 55.8 40.0 010401/0900 11.3 317 54.3 29.5 8.5 9.5 -15 -10.5 

12 2007 10 10.2 324 54.8 38.8 01040111000 18.3 ]19 5].2 2B.2 B.l 5.0 -1.5 -10.6 

13 2001 11 10.0 320 54.1 38.7 01040111100 17.9 319 52.3 28.0 1.9 1.2 -1.8 -10.1 

14 2001 12 10.9 320 54.0 31.1 01040111200 18.3 ]18 5].6 29.8 1.4 2.3 .j).4 -7.9 

15 2007 13 11.6 321 54.1 36.1 01040111300 19.B 325 51.0 32.4 B.2 -].8 2.3 -4.3 

16 2001 14 11.B 32B 56.2 35.1 010407/1400 lB.4 330 59.2 33.6 6.6 -2.4 ].0 -1.5 

17 2001 15 10.9 321 58.0 35.8 07040111500 11.2 333 61.2 35.4 6.3 -12.2 3.2 .j).4 

18 2001 16 8.1 313 59.2 36.3 01040111600 16.5 336 62.4 36.0 U -23.4 3.2 .j).3 

19 2001 11 1.0 311 60.9 35.8 01040111700 14.1 341 63.5 36.7 7.8 -30.0 2.6 0.9 

20 2007 18 6_9 313 62.1 35_0 01040111800 13.8 340 64.0 36_1 6.9 -21.5 1.3 1.1 

21 2001 19 10.0 29 61.0 41.9 01040111900 11.4 339 64.6 35.1 1.4 -49.5 3.6 -6.8 

23 2001 21 U 60.1 42.6 01040112100 6.4 340 65.3 33.6 f\O'" 24 2001 22 1_1 60.3 42.6 01040712200 6.3 351 64.8 33.1 .,. O~ 

2001 0 4.1 58.5 44.1 010408I0000 6.5 9 51.1 33.B 2.4 32.B .j).B -10.3 

2001 .. 1 3_0 10 58.3 44.2 01040810100 6.2 5 56.1 34.9 3.2 5.3 -2.2 -9.3 

2001 4 2 3.5 339 59.0 43.9 070408/0200 6.4 357 54.9 34.9 2.9 -11.3 -4.2 -9.0 

2007 4 3 4.1 331 59.0 44.2 07040810300 6.6 339 53.4 32.1 2.5 -7.7 -5.6 -11.5 

2007 4 4 6.7 291 56.5 43.B 070408/0400 8.6 322 52.5 28.2 1.9 -31.0 -4.0 -15.5 

2007 4 5 1_5 298 55.0 42.1 07040810500 10.7 318 51.1 25.9 3.2 -19.7 -4.0 -16.3 

2007 6 1.7 306 54.0 41.4 070408l0600 12.0 320 SO.4 23.4 4.3 -14.3 -3_6 -lB.l 

2007 7 1.B 309 63.2 41_3 07040810700 12.4 321 SO.O 21.1 4.6 -12.5 -3.2 -19.5 

2007 B 8.4 324 52.9 36_5 010408/0800 12.6 323 49.5 21.4 4.2 1.4 -3.5 -15.1 

2007 9 1.5 323 52.8 33.9 070408/0900 12.0 329 48.9 21.7 4.5 -5.9 -3.9 -12.2 

2007 10 6.5 317 53.5 34.2 01040811000 11.8 329 4B.6 22.6 5.3 -11 .6 -4.9 -11.6 

2007 11 1.0 318 53.9 34.9 07040811100 9.9 332 47.8 22.B 2.9 -13.5 -8.1 -12.0 

2007 12 1.3 32B 53.1 36.0 01040811200 9.3 334 52.0 27.3 2.0 -8.4 -1.7 -8.6 

2007 13 7.1 32B 55.3 36.4 07040811300 9.7 345 58.6 35.8 2.6 -1 6.6 3.3 .j).6 

2007 14 U 326 58.4 34.0 01040811400 10.2 346 61.3 38.1 1.8 -19.9 3.0 4.1 

2007 15 6.9 32B 60.3 33.6 07040111500 1.8 347 62.2 38.7 1.9 -19,4 1.9 5.0 

2007 16 6.0 332 61.6 34.2 01040811600 8.2 345 6O.B 38.7 2.2 -12.6 .j).8 4.4 

2007 17 3.3 333 63.1 36_8 07040811700 8 .• 351 60_4 39.2 5.5 -11_3 -2.1 2.4 

2007 lB 6.5 39 63.4 43.0 01040811800 7.4 357 60.8 40.5 0.9 -42.5 -2.6 -2.5 

2007 19 5.7 16 64_1 42.5 07040811900 11 .9 323 58.6 40.B 6.2 -52.6 -5.4 -1.6 

2007 20 1.B 63.6 43_6 07040812000 14.1 314 SO.9 44.8 6.3 -80.0 -12.1 t2 7 21 6.6 63.3 44.4 07 8/21 13.6 317 49.5 44.1 

2007 22 5.6 63.0 44.1 070408/2200 9.7 323 48.7 44.6 f\O'" 2007 23 3.1 62.5 45.0 07040812300 6 .• 309 49.1 45_1 OI\·I\O~ 2007 0 3.9 61.4 46.0 07040!llOOllO 6.1 281 48.6 45.9 

46.6 4.1 75.0 -13.3 -1.0 

<16.9 4.0 -71.0 -13.4 .j).6 

47.3 4.2 44.B -13.6 .j).5 

~" . . ,... .. 

Figure 10. Example of a merged Excel workbook file showing the tower observations from Tower 002 

(columns A-H), MesoNAM forecasts (columns J-N) initialized at 00 UTC (OOZ tab at bottom left), and 

difference calculations (columns P-S) for an offshore stratification 

The procedures to calculate the difference included quality checks and stratification calculations. In the 

VBA scripts, the first step implemented was a QC test that ensured the tower observations did not exceed 

unrealistic values and/or climatological maxima and minima based on a 30-year Operational Climatic 

Data Summary provided by the 14th Weather Squadron for the Shuttle Landing Facility (KITS). Even 

though an initial QC was performed on the raw observational data, through visual inspection, the AMU 

discovered some out-of-bounds observations were still present after the initial QC. The VBA script QC 

test checked to ensure the observations were within the following ranges: 

• Mean wind direction ~ 0° and ~ 360°, 

• Mean wind speed ~ 0 kt and ~ 65 kt, 

• Mean T ~ 20 OF and ~ 100 OF, and 

• Mean T d ~ 5 OF and ~ 85 of. 

4.2.4 Coastline Orientation 

To stratify the onshore and offshore flow, the AMU scripts accounted for the orientation of the 

coastline differences between Tower 002 and the rest of the towers. For T and Td sensors without 

corresponding wind sensors at the same height, the wind sensors closest to the T and T d sensors were used 

14 
to determine the stratification for those observations. For example, the 12 ft wind sensors were used for 

the 6 ft T and T d sensors. 

To calculate the proper stratification for Tower 002, the AMU scripts checked the mean wind direction 

associated with each parameter prior to every difference calculation. Separate calculations were made for 

the sensors on the NW and SE sides of the tower. For onshore flow, the scripts verified the mean wind 

direction observed was ~ 45° and < 225°, represented by the green arrows and arc in Figure 11. In this 

case, data from the SE sensor were used. For offshore flow, the NW sensor was used if the scripts verified 

the mean wind direction observed was ~ 225° and ::s 360° or < 45°, represented by the red arrows and arc 

in Figure 11. 

Figure 11. Representation of the sensor configuration for 

onshore and offshore stratifications on the dual-sensor Tower 

002. The NW sensor is used for offshore flow stratification while 

the SE sensor is used for onshore flow stratification. 

To calculate the proper stratification for the other two dual-sensor towers, Towers 006 and 110, the 

AMU scripts also checked the mean wind direction associated with each parameter prior to every 

difference calculation and made separate calculations for the sensors on the NW and SE sides of the 

tower. Because of the towers' orientation to the coastline, sensors on either side could be used for both 

onshore and offshore stratifications, which required using NW and SE sensors to determine the 

stratification. For onshore flow, the NW sensor was used if the mean wind direction was ~ 315° or < 45°, 

represented by a green arrow and thick light green arc in Figure 12. The SE sensor was used if the mean 

wind direction was ~ 45° and < 135°, represented a green arrow and thick dark green arc in Figure 12. For 

offshore flow, the NW sensor was used if the mean wind direction was ~ 225° and < 315°, represented by 

a red arrow and thick light red arc in Figure 12. The SE sensor was used if the mean wind direction was 

~ 135° and < 225°, represented by the red arrow and thick dark red arc in Figure 12. 

Figure 12. As in Figure 11 but for the dual-sensor Towers 006 

and 11 o. Both the NW and SE sensors must be used for onshore 

and offshore flow stratifications. The gray dashed line represents 

the divide between the data used from the NW sensor and from 

SE sensor. 

15 
r------ ------- ------- -------------- ------------

As with the dual-sensor towers, the AMU scripts checked the mean wind direction associated with 

each parameter from the single-sensor towers prior to every difference calculation for the towers with one 

sensor at a specified height. For onshore flow, the scripts verified the mean wind direction observed at 

the sensor was 2: 315° or < 135°, represented by the green arrows and arc in Figure 13. For offshore flow, 

the scripts verified the mean wind direction observed at the sensor was 2: 135° and < 315°, represented by 

the red arrows and arc in Figure 13. 

Figure 13. As in Figure 11 but for the single-sensor towers. A 

single sensor was used for both onshore and offshore 

stratifications. 

4.3 Mean Monthly Model Bias Calculations 

The mean monthly model bias statistics were calculated using the previously computed hourly 

differences between the model forecast parameters and tower observation parameters. As discussed in 

Section 4.2, the model bias statistics were calculated in the master Excel workbook files and displayed in 

columns C-F as shown in Figure 15 by using the following equation: 

Where: 

n 

BiaSMesoNAM Monthly Mean = ~I (fi - ° i) 

i=l 

n = number of available model forecasts in any given stratification, 

f = MesoNAM forecast ofT, Td, wind speed or wind direction, and 

o = observed T, Td, wind speed or wind direction from each tower/sensor height. 

4.4 Standard Deviation of Mean Monthly Bias Calculations 

The model standard deviation of mean monthly bias statistics were calculated using the previously 

computed mean monthly bias and using the following equation: 

JL(X - X)2 

STDEVMesoNAM Bias = n 

Where: 

n = number of available model forecasts in any given stratification, 

x = model bias of each forecast, 

i = mean bias of each forecast period in any given stratification (Section 4.3). 

16 
4.5 Root Mean Square Error Calculations 

The model root mean square error (RMSE) was calculated using the following equations: 

Where: 

RMSE = -VMSE 

n 

1", 2 

MSE = nLCfi - oa 

i=l 

n = number of available model forecasts in any given stratification, 

f= MesoNAM forecast ofT, Td, wind speed or wind direction, and 

0= observed T, Td, wind speed or wind direction from each tower/sensor height. 

4.6 Hypothesis Zero Test 

Hypothesis testing uses statistics to determine the probability that a given hypothesis is true. The 

AMU worked with the 45 WS to determine the best way to setup the test. The goal was to determine ifthe 

model bias of any of the parameters assessed throughout the model forecast period was not statistically 

significantly different from zero. The 45 WS recommended the test statistic for this data set would be to 

divide the mean bias of a given parameter by its mean standard deviation divided by the square root of the 

number of observations within a given stratification (onshore or offshore), as shown by: 

Mean Bias /; 

jtvtean SrDEV 

.fii 

Where 

Mean Bias as defined in Section 4.3, 

Mean STDEVas defined in Section 4.4, and 

n = number of observations. 

If this equation produced a value 2: -1.96 or::::; 1.96 (personal communication, Mr. Roeder, 45 WS) for a 

data point, then the bias at that point was not statistically significantly different from zero and the model 

forecast for that point was considered to have no error. 

The AMU worked closely with the 45 WS L WOs to determine the best way to display the results of 

the hypothesis zero test graphically so it would be useful for operations. The group agreed to have the 

hypothesis zero test displayed on the model bias charts as shaded regions only when the test was true 

(-1 .96 ::::; test value::::; 1.96). When the test was true, it indicated that even though the bias may not have 

been zero at any given data point, statistically the bias was zero. 

In Figure 14, there are several time periods when the hypothesis zero test was true as indicated by the 

green shading. The shape of the green shading is defined by the test value when the test was true. For an 

operational example, consider the forecast valid time 73-83 hr indicated by a red ellipse in Figure 14. 

Even though the model temperature bias (blue line) was not zero, the hypothesis zero test indicated that 

statistically, the temperature bias was zero and the L WO would not have to adjust the model temperature 

forecast for this time period. When the hypothesis zero test was false, the graph was not shaded and the 

L WO would have to consider adjusting the model forecast temperature. An example of a false hypothesis 

zero test is shown by a purple ellipse in Figure 14 during the 61-72 hr forecast valid time. 

17 
Model T Bias (Onshore) 

12Z Initialization, Tower 2, 54 tt, November 

_ Hypothesis Zero - T (OF) 

3.0 

2.0 

-LL 1.0 0 

-II 0.0 ... 

:J 

1ii -1.0 ... 

II 

Q. -2.0 

E 

~ -3.0 

-4.0 

-5.0 

Forecast Valid TIme from Model Initialization 

Figure 14. An example of a verification chart showing model bias of T (blue line) and hypothesis zero 

test = true (light green shaded regions) from a 12 UTC model initialization using observations from 

Tower 2 at 54 ft for November with an onshore stratification. The red and purple ellipses highlight the 

region of the chart in the example discussed in the text. 

4.7 Statistics Values 

The AMU developed master files in Excel workbook format stratified by tower, sensor height, 

onshore/offshore flow and month in order to calculate the statistics. Figure 15 shows a portion of one of 

the master files for Tower 006 at 54 ft during offshore flow in June 2007-2009. Each master file contains 

four spreadsheets representing the model initialization times of 00, 06, 12 and 18 UTC. In each 

spreadsheet, column A shows the month, tower/sensor identification, sensor height and model 

initialization time and column B shows the model forecast hour, 0-84. Columns C-F contain the model 

bias, columns H-K contain the model standard deviation of the bias, and columns M-P contain the model 

RMSE for the month for each model forecast hour based on the formulas in each cell. The columns 

beginning with R contain the differences for each parameter that were previously calculated for each day 

of every month in the POR and each model forecast hour. 

18 
.. u ... ., IS ... ·u ... .. ,. ... ... ... ... ... IS .... U .. .. ... ... ,. .. u u ... .., ....... ., It .... U .. LI LI ., " ... 11 ... .. U .L1 

" ... , ... , ... ... .. .... .... ... " L> ... ... ... ... .. ... U " .,.. ·U J.l . L> LI ... .., ... LI .., ....... LI . .., U Lt U 

.. .... .... ... .. LI .. ... .. ... 1.1 ... J.l .. ., .... .... U .. ... .. .. .. U ... ... .. ... .. ... ... .. ... U .. LO ... .... U ... J.l .. .. U U ,~ 

I II ... ... U .. .. ., ., .. ., .. LI . ., ., ... .. .. ... ... ... ., U .. ... .... .. .. ... .. .. .. ... U 

U · .. ... U .... 

U . ,.. .. ... II .. ... ... ., U 

U ... .. .. ... .. • • U ., ... U 

LO .... ., .. ... . .., U . U ... .. U ... .., ... " U .... U .. U ... " U U ... ... .... ... .. .... · La 

.. OA .... ... .. ... ... ,. .. U ,. ... ...... .. ... ..., ... .. U .... U .. .... .... ... ... ... .., .. U ... U ... 

... U .. ., ., ., . ,... ..... ... ·U ... " ... ... " .... . .. ., ... · .. ,... U ... ... .... ., .. ., U .. ... ... .. .... ... .. ... .. .. .. , . .... . ., ... ., ,. U .... ·U ... .. .. ... .. .. .. .. .... ... .. ... U ... ·U l.I .. ... ... ... U U . J.) U .. U ., L> 

... ... U .... 

· . .., U " . , ., ,. .. .... .. U .. " .... ... " .. .. " ... U ..., ., ... ,. ... ., .. .. L> U ., ... U ... .. fA .. .... "1' -1..l .., ., ., ... ,.. .. ... ,.. ... U .,.. 

.'J.1 ... ..., 

U II ..sA .. , ... 

... · " ... .. .. U 

... " ... .. ,. .. ... .... . L1 ., .. ., .... ... ., II 2..1 .1.1 U . J.) 

... U .. . ... .. U .. .. ,. OJ ... 

Figure 15. Example of a master Excel workbook file used to calculate the mean monthly statistics. 

Column A contains tower, sensor identification and model initialization information, column B shows the 

model forecast hour, columns C-P include the calculated statistics and the rest of the spreadsheet holds 

the hourly difference calculations. 

The AMU created charts in each spreadsheet that were automatically populated with the statistics by a 

VBA script that extracted the difference calculations from the merged spreadsheets and saved them in the 

master workbook files. Figure 16 shows the same spreadsheet as in Figure 15 but it is displaying several 

of the resulting charts based on the statistics calculated within the spreadsheet. The charts in the master 

workbooks were ultimately used to populate the graphical user interface (GUI) described in Section 6. 

19 
-•. ~rl·.£y·t::r U'''AJ:r-P-H~'"''l ~ - . 

- g 

~ . • c 0 • • . , I · , . . • , q . • T U Y W . , • .. AI "- AD .. II .. AM 

.. . . . . . . , . . .. u u u .. .. .. rr .. _, __ , --_-_, ~ · .. U 

.. · ~ ~ ... D 

" -- -....... - ooz_ .... _ ....... _ -. -. 4 . .. _~z.r. T("fl -....,..n.tIIt.fO-lpGlktI ·U .rr · 

., . M ~ ·u .. ·U U 

.. U ,. .. ... , . .. .. .. .. ro -. · " " -Ill 

" U A, /I I\" \. '\ u ·u -. .. os .. .1.1\ I\J n I\J I: A n ·u I ID III 

.. Eu u -. .. " " '" to 

lOt I .. l .... 1 111&. IU "'\ "- III u · ro ... n 

ro. ,1. j, .... I. I. " .. w. .. n , ,. 1 I : 'l Alii r> · " ... .. ." ! -u ~ III \I u .. u ." to . " .. ro. \. .J fI .. · · · -.. III ... ... . ", - 'V I 111 • .. u u .. ... ... .... .. .. u .. . ., ... . · " .. .. .. .. OJ .. " .. .. n ,. .. · " " ... .", ro, .... ........ "--- --~ ~ ................. ---- · u 

... · · , .. ... · U -I 

u. ............ a.v .. '_Icwww.. -_ ..... _--,-, , · -. UI · ... • UI ........................ 54tt. ... 0Il ................... f!t. .... 

· , ., ... .. .. .. · ~ ~ 

'" .. , ., ~ 

'" .. A I. .. " -. .. _r> .. U. E'" A M M " -ro · .. ur r'I IVI A.. I" " . " · , .. u. r .r-.. tlh J "'-J V\ .. ·ro ~ · .. ... I \.. I" N,\ -.. " -u · " ... ... u r ' / "\../ "- J 'V\ I .. 'v -~ '.J .. -r> -, U ... . ... u. .. .. . ., .. .. .... III .. • rr · " Ie .1» 

u. .. .. .. ~ n .. .161 ·121 

... n I I -D 

III o. .. . " · -I -. . U. . . " II .. .. to OJ .. .. .. .. n ,. .. . · .. II .. .. .. OJ .. .. .. .. n ,. .. ... , 

· · ... . .., . U. .................... ............... F..-Iv.lW ......... ............... -I U .U • ... W " n _u ... . .. , U1 " U ~ -» ... _._-, ----,-, .. .. , ·11 .... U. II II n .. '" _ ...................... 14 ...... DOl ................... ,.' ... .. .. os U. ,. .. .. to .. lO' o. ,. ... II .. W " III A f'\ A .. · .. ur ... E" .. A ,,/ \ .AI "- .tV \ .. ·1 .. ur t'\ '" 1\ I" .. .. ·n " wI" "\ V " V \. V .. ... . " .. no / \... /"\ /If \/\ I : ". V' .. .. . ,. " .. to 

... ... ./v -v V I ... .. .. " .. ~n,u oar " ·1 ... " 

Figure 16. Example of a master Excel workbook file used to calculate the mean monthly statistics. The 

graphs were automatically populated by the statistics calculated in the spreadsheet and show, from top to 

bottom, model bias, model standard deviation of bias and model root mean square error. 

20 
5. Model Performance 

The goal of this work was to provide the L WOs with a tool to evaluate model performance compared 

to observations at the tower(s) used to support a specific launch vehicle and not to evaluate the overall 

model performance at KSC/CCAFS. However, it is worth mentioning several characteristics of the model 

that were noted during this work. 

5.1 Accuracy 

As one would expect, the model's accuracy decreased as the forecast valid time from the model 

initialization increased. Figure 17a and Figure 17b show examples of the model's standard deviations of 

wind speed bias and wind direction bias, respectively. The trend line in both figures illustrates the 

increasing error in the model forecast compared to the wind observations. This model characteristic was 

noted in most of the data. 

Model St.nard Dew of WInd Speed 81 •• (Offahore) 

OOZ 1n1t1.IIz.don, T_ 2, 54 ft, October 

.-------------------------------

t=~~~~;1~~1f~~~~~r-~ I L---------~L--------- ---- ... 

• +------------------------------- 1 1.0 +-------------------------------

a ForeC8lt v.lld nme from Modellnltlellzetlon 

Model Stend.rd Dew of WInd Dlr 81 •• (Offahore) 

OOZ Inltl.llz.don, To_ 2, 54 ft, October 

r-------------------------------

70 +-------,..-- ---1f---"~__\__: ~'"6 

_ +_~~----~T* ~-r~ T_r_-­

a • +_ ~~~L-~~~------;pL---

6 ~~ --¥-~---------------------­ i ~----------------------------

a +-------------------------------

r_------------------------------

b Forec:.lt YeIId nme from Modellnltlellzlltton 

Figure 17. An example of verification charts illustrating the model's increasing error (solid black trend 

lines) for (a) the model standard deviation of wind speed bias (green line) and (b) standard deviation of 

wind direction bias (orange line) using observations from Tower 002 at 54 ft for October with an offshore 

stratification. 

5.2 Diurnal Signal 

A diurnal signal was present in the T and Td model bias as shown by the examples in Figure 18 and 

Figure 19. The diurnal signal was pervasive among the towers at varying degrees of amplitude. In 

general, the model had a cool T bias during the late night (- 05-07 UTC) and a warm T bias during the 

afternoon (- 17-19 UTC). The T d showed the opposite - the high bias during the late night and low bias 

during the afternoon. 

Figure 18a illustrates an example of the T bias at Tower 006 for a model initialization time of 00 

UTC, while Figure 18b is an example of the T bias for a model initialization time of 12 UTC for the same 

tower, sensor height, month and onshore stratification. These charts show that the model's warm bias 

peaked at the 18, 42 and 66 hr forecast valid times from the 00 UTC initialization, and at the 6, 30, 54 and 

78 hr forecast valid times from the 12 UTC initialization. For both initialization times, the warm bias 

maxima occurred at about 18 UTC throughout the forecast. Conversely, the cool bias minima were at the 

29, 53 and 77 hr forecast valid times from the 00 UTC initialization, and at the 17, 41 and 65 hr forecast 

valid times from the 12 UTC initialization. For both initialization times, the cool bias minima occurred at 

about 06 UTC throughout the forecast. 

21 
Model T alas (Onshore) Model T alas (Onshore) 

OOl Initialization, Tower 6, 6 ft, May 12Z Initialization, Tower 6, 6 tt. May 

_ Hypothesis Zero - T('f) _ HypotheslsZero - T('F) 

3.o ,..---------------- 3.0 ,..----------------

~0 +_--1r-------- --- +_ ----~~----------~-

t: 1.0 E 1.0 • • !; !; 

1i 0.0 ~ 0.0 

~ D- • ·1.0 D- -1.0 E E ~ ~o +-____ .1.1<.. ___ .....=..;~ __ ..,..,..-'--

~ -2.0 +_------------'---------'=-----

-3.0 -'---------------------- -~ -'-----------------------

a Forecast v.lld TIme from Modellnitlellzetlon b Forecast v.lld TIme from Modellnltlellz.tlon 

Figure 18. An example of verification charts illustrating the model's diurnal signal for the model bias of 

T (blue line) and hypothesis zero test = true (light green shaded regions) (a) from a 00 UTe model 

initialization and (b) from a 12 UTe model initialization using observations from Tower 006 at 6 ft for 

May with an onshore stratification. 

Examples of the model's diurnal T d signal is shown in Figure 19a and Figure 19b and are valid for the 

same tower, sensor height, month and onshore stratification as in Figure 18. While showing a similar 

pattern to T, the Td high and low biases occurred during the late night and afternoon, respectively, which 

was opposite of T. Figure 19a illustrates an example of the Td bias for a model initialization time of 00 

UTe, while Figure 19b is an example ofthe Td bias for a model initialization time of 12 UTe. 

Model Td alas (Onshore) 

OOl Initialization, Tower 6, 6 ft, May 

_ Hypothe slsZero - Td('F) 

3.0 ,..------------------

+_---------------------

E 1.0 +_---i-----j---------------

a 

-~o +_-------->.,J-----lt-=::I----

-4.0 -'------.---.------:.------:----

Forecut v.lld TIme from Modelinitllllizetion 

Figure 19. As in Figure 18 but for Td. 

Model Td alas (Onshore) 

12l Initialization, Tower 6, 6 tt. May 

_ HypotheslsZero - Td('f) 

3.0 ,..------------------ ---

+_----~--------------

E 1.0 +---

; 0.0 +-......--./" 

1i 

~ -1.0 f'\--..:.......,...-=---=t-....::.:...~----=____'l.......:..:c........:;:""""O 

DE ~~---+_~---- ~~~.__+ ~ 

o +_-------b;c----\-,J------+--f::..-

-4.0 -'-----------------------

b Forecut v.lld TIme from Modellnltlellzetlon 

5,3 Model Parameter Vertical Level and Sensor Height 

The MesoNAM outputs forecast temperature and winds at heights of 2 m ~ 7 ft) and 10m (~ 33 ft), 

respectively. In general, the model bias of T was smaller at a tower sensor height of 6 ft and larger at 

higher levels. The charts in Figure 20 illustrate this showing the model bias of T :s 12.4 °FI at 6 ft (Figure 

20a) and the bias ofT:S 14.0 °FI at 54 ft (Figure 20b) at Tower 002. 

22 
Model T 81as (Onshore) 

122 InItIaIlDtIon. lbwer 2, 6 tt. JIM'MI 

_ Hypothesis Zero - T('F) 

5.0 5.0 

4.0 4.0 

t: 3.0 E 3.0 

2.0 2.0 

• LO • 1.0 !; !; 1i 0.0 1i 0.0 

• ·1.0 ~ -1.0 r -2.0 r -2.0 

{!. -3.0 {!. -3.0 

O O 

-5.0 4 0 

a ForeCMt V111k111me from Modellnitleilletion b 

Model T 81as (Onshore) 

12llnltlallZlltlon, lbwer 2. 54 tt. JIM'MI 

/\. 

I \ A A. 

/ \ ,/ "" 1 \ 

/'\. 

I '\ 

/ \ 

\ I 1 I 1 j , 

:1 ~ 1 j J 

" "\. "'\ I 

'-.} \. \. I ~ " 

ForeCMt Wldnme from Modellnitleilletion 

Figure 20_ An example of verification charts illustrating how the model's bias varies with respect to 

tower sensor height for the model bias of T (blue line) and hypothesis zero test = true (light green shaded 

regions) from (a) the 6 ft sensor height and (b) the 54 ft sensor height using observations from Tower 002 

for June with an onshore stratification. 

Figure 21 illustrates how the model bias of wind speed differs at two different levels on Tower 002. 

At the higher sensor height of 54 ft (Figure 21a) the wind speed bias was < 5 kt while at the lower sensor 

height of 12 ft (Figure 21b) the bias was < 8 kt. In general, the wind speed bias was smaller at tower 

sensor heights above 54 ft and larger at 12 ft. 

9.0 

' .0 

.... 7.0 

1; 6.0 

g, 5.0 

"0 4.0 

• I 3.0 

,)t 2.0 

a 

1.0 

0.0 

Model Wind Speed 81as (Onshore) 

OOZ InItIMIDtlon, Tower 2, 54 tt. June 

_ HypotheslsZoro -Spd(Ict) 

J'\. 

Jo/ .... ./ \. -- " \.. 

r "--' 'Y\.;J "\. 

1/ -

/".. 

/ 

j' 

F_t Wid nme from Modellnitleilletion 

\.. 

V\ 

9.0 

B.O 

... 7.0 

1; 6.0 

g, s.o 

"0 4.0 

• ! 3.0 

,)t 2.0 

b 

1.0 

0.0 

Model Wind Speed 81as (Onshore) 

OOl InltlallZlltlon, Tower 2, 12 tt. JIM'MI 

_ HypothoslsZero - Spd (Ict) 

,....... A A 

.J-'\. J \ r \.. I \. 

I \ / \. / "\. / ~ 

\ J \. I \ I \ ..... -- ,/ 

ForeCMt Wid nme from Modellnltlellzetlon 

Figure 21. An example of verification charts illustrating how the model's bias varies with respect to 

tower sensor height for (a) the model bias of wind speed (green line) from the 54 ft sensor height and (b) 

the 12 ft sensor height using observations from Tower 2 for June with an onshore stratification. 

23 
6. MesoNAM Verification Tool Graphical User Interface 

The dataset for this work contains 2,496 Excel spreadsheets with 4,896 charts. In order to present the 

data to the L WOs in a manageable and user-friendly manner, the AMU developed the MesoNAM 

Verification Tool using a Hyper-Text Markup Language (HTML) GUI so the tool could be used in most 

popular web browsers with computers running different operating systems such as Microsoft Windows 

and Linux. 

For easy navigation, the AMU implemented a multi-level drop-down menu written in JavaScript 

embedded within the HTML code. The main page of the GUI is shown in Figure 22. The title at the top of 

the page defmes the name, version and date of the tool. The drop-down navigation menu is located under 

the title and allows the user to choose data by placing their mouse pointer over the desired tower/site to 

display a drop-down menu. A map of KSC/CCAFS showing the locations of the wind towers used to 

verify MesoNAM forecasts, the locations of the MesoNAM model grid points, the CCAFS weather 

station (KXMR) and the MesoNAM land/sea mask is shown as a reference for the L WOs. 

Indian River 

45th Weather Squadron 

MesoNAM Verification Tool V2.0, July 2010 

Developed by NASA's Applied Meteorology Unit 

• 

[108 Delta IV I Falcon 9 

-006 Delta IV I Falcon 9 

CCAFS l 

Banana River; ~MR-, 

03S. HAM 1_ , 

• «l:W 30le OGle 

017" 002 Delta II t 1001. 

• 

• 

L.tatnd _ Launc:M.a ..... 

• OIher~,. 

• 1It1OU .. Grid PoInI8 

Figure 22. Main page ofthe GUI in a web browser showing the basic layout of the GUI. 

24 
An illustration of the drop-down menu is shown in Figure 23. Assume a user placed their mouse 

pointer over the menu item for Tower 002. By doing so, this displayed the first level of the drop-down 

menu, the months January- December. The user then moved the mouse pointer over November, the 12Z 

model initialization time, 54 ft sensor height, and then clicked "Onshore" for the flow. 

45th Weather Squadron 

MesoNAM Verification Tool V2.0, July 2010 

Developed by NASA's Applied Meteorology Unit 

• 6 Delta IV I Falcon 9 

Banana River KXMR 

HAM • 03k 

• .3e 303e 003e 

017-'002 Delta II • 

001. 

• 

LIatod 

• 0IIw LI~""'''''' row.ra • IIMoIIAM GrId PoInIa 

Figure 23. Main page of the GUI illustrating the functionality of the drop-down menu (heavy blue 

polygon region). 

The resulting page in the GUI displayed from these choices is shown in Figure 24. It contains charts 

showing the model bias, standard deviation of bias and RMSE ofT, Td, wind speed and wind direction for 

onshore flow at a sensor height of 54 ft based on a 12 UTe model initialization for November at Tower 

002. The LWO can view all ofthe charts by using the vertical scroll bar on the right side of the web page. 

On each web page in the tool, the AMU added a link to the corresponding Excel workbook file that 

produced the charts on that page. This feature allows the user to view the spreadsheets from which the 

charts were derived. As shown in Figure 24, there is an Excel icon (red ellipse) just under the "Home" 

link on the navigation menu. When the user clicks the link, Excel opens the workbook file associated with 

the charts on that web page. 

25 
The AMU included a "Help/About" page as part of the GUI that briefly explains why it was 

developed, how to use it, sensor configuration and a description of the data (Figure 25). The link to this 

page is on the far right of the drop-down menu bar on the main page. The Help page includes sections 

describing the final report and how to access it, background information on the task, how to use the tool 

and the data used. 

F 

45th Weather Squadron 

MesoNAM Verification Tool V2.0, July 2010 

Developed by NASA's Applied Meteorology Unit 

Model T BII. (Onshore) 

12Z Inltll ll zitlon, Tower 2, 54 ft, November 

3.0 r-----------------

1.0 +_:_-h:---------------

-3,0 +_:_--------""'-------'lIL'lf"'---

-4.0 L-----=-FO-.. -ca--" ::-Valc::ld:-::lI:-:: m"CC. ,:-:: ro'C'Cm""M:-:: od::C' . :7' lIn""ltl"".,I::-: , ott 7.".C':" n---

Model Stlnd .. d Dev ofT Bil s (Onshore) 

12Z Inlt l.llz.tlon, Tower 2, 54 ft, November 

~o ,-----------------

3.0 +_:_-------;;:------F'r-+=----I...-::l~_t"''---''_+ 

E 

~ 

0.0 

Model Td BIIS (Onshore) 

12Zlnltl.llntlon, Tow,rZ, 54ft, November 

-1,0 -I-r-l'!1IIU-

! +_:_-----------------V~~~ -r~~ 

! 

E 

~ -3,0 +-_________________________ .L.-__ 

~O ._-----------------------

5,0 +-----------------------------~~ 

E ~O +_:_----._------_,Hr--~~~~~~-b ! 

, I 

~ ---~~ -=----- -- i! ) ,0 +_:_-II-I--¥-1t--:-P--\;;I-.......;:"-!...--------------

~ ~ 1.0 +-----------------

f 

~ 2.0 +\1"--"------------------------------

1,0 +--------------------------------

o 6 II 18 24 )0 36 42 .. 5' 60 66 72 71 I' 

foreCMt VaJldTim. from Mode' lnltlall, atlon foreCistVaUdT1rne from Model Initialization 

Figure 24. A page from the GUI displaying four charts of model T bias and standard deviation of bias 

and model Td bias and standard deviation of bias for Tower 002 at a sensor height of 54 ft for a 12 UTe 

model initialization for November. The blue lines characterize T and the red lines Td. The green shaded 

regions correspond to a ''true'' hypothesis zero test. The icon representing a link to access the Excel file 

used to create the charts is shown within the red ellipse under the blue menu bar. 

26 
45th Weather Squadron 

MesoNAM Verification Tool V2.0, July 2010 

Developed by NASA's Appl ied Meteorology Unit 

Help/About this Tool 

Final Report 

WIlle ltis H~Abru page provides an avel'liew otitis tool, it is tigtly recommended that IJSeIS read It1e final report prior to using ltis tool The final report, located 

here, COItains al of the details ottheWOl1l: conducted on ltis task leading t4l to It1e deYetopmect otitis toot. 

Background 

In order to ha.e tangible eYldence of model pefformance, the 45th Weather Squaaoo (45 WS) tasked the Appled Meteorology Urit (AMU) to conduct a detailed 

statistical analysis of model 0UIIl" cOlJ1l8red to ob5elYed values. The model products are provided to It1e 45 WS by ACTA, Inc. and Inclucle hoII1y forecasts from 0 to 8-4 

hOlIs based on model iritiaiZlltion times d 00, 06, 12 and 18 UTe. The ~ analysis ~ared the MesoNAM forecast winds, tl!f11lel1llLfe and dew poin to the 

obseM!d Y1WeS from the sensors in the KSCICCAFS wind tower network shown in Table 1. The sensor heigts nKJired for this task are in the coUnn labeled "Original 

Sensor Hei\ttS". Remaining sensor.; 00 T I1M!{S 002, OOS, 108 and 110 wen! added for CCIfC)Ieleness and those sensor hei\ttS are sIlOMI in the coUm labeled 

·Additional Sensor Heigts·. 0bjecIiye slatistics wil!iie the forecasters knowledge d the modeI's sIrengIh and weaknesses, which wil resli in i~ forecasts for 

operations. 

Table 1. Towers, launch adIvItIes and sen50f heights at KSC end CCAFS used In the objective 

_lysis to vefify the MesoNAM fortcaats. 

SUpporftd ActMty ~Irtd Sensor Adt:IIIIoMI Sensor 

TowwNumbtr and FacHIty Hei{IItts HeIghts 

002 Delta II (LC-17) 6ft, 5411, 90 ft 145 ft, 204 It 

006 DlIta N (LC-37YFalcon 9 (LC-40) 5411 6 It. 12 n, 162 II, 204 n 

108 Delta IV (LC-40YFalcon 9 (LC-40) 54n 6ft, 12 n 

110 AllIS Vlfllcon 9 (Lc-.40) 54ft, 162lt.204n 6ft,12n 

041 AtIu V (Lc-41) 230ft 

393/394 Shuttlt (LC-39A) 60ft 

397/398 Shuttlt (LC·398) SOft 

511/5121513 Shuttlt landing FICllIty 6ft, 30 II 

The gri~rt forecast in the MesoNAM txJeIin is identified as KXMR b" It1e data is actualy located at the poi'" in Figure 1 which is sotihwest of KXMR and identified 

as "NAM". AIhough the model gridpoirt is located (He( the Banana RNer, the model resoItmon is not fine enough to ditrereriiate between walef and land in the 

KSCICCAFS 'o'idnity, therefore the gridpoi'" is asslmed to be (He( land. 

.. 15 • 

Figure 25. The "Help/About" page from the GUI. 

27 
7. Summary and Conclusions 

The AMU conducted an objective analysis of the MesoNAM forecasts compared to observed values 

from sensors at specified KSC/CCAFS wind towers by calculating the following statistics to verify the 

performance of the model: 

• Bias (mean difference), 

• Standard deviation of Bias, 

• Root Mean Square Error (RMSE), and 

• Hypothesis test for Bias = O. 

The 45 WS L WOs use the MesoNAM to support launch weather operations. However, the actual 

performance ofthe model at KSC and CCAFS had not been measured objectively. The analysis compared 

the MesoNAM forecast winds, temperature and dew point to the observed values from the sensors on 

wind towers. The data were stratified by tower sensor, month and onshore/offshore wind direction based 

on the orientation of the coastline to each tower's location. The model's performance statistics were then 

calculated for each wind tower based on sensor height and model initialization time. The period of record 

for the data used in this task was based on the operational start of the current MesoNAM in mid-August 

2006 and so the task began with the first full month of data, September 2006, through May 2010. 

The analysis of model performance indicated: 

• The accuracy decreased as the forecast valid time from the model initialization increased, 

• There was a diurnal signal in T with a cool bias during the late night and a warm bias during the 

afternoon, 

• There was a diurnal signal in Td with a low bias during the afternoon and a high bias during the 

late night, and 

• The model parameters at each vertical level most closely matched the observed parameters at 

heights closest to those vertical levels. 

The AMU developed a GUI that consists of a multi-level drop-down menu written in JavaScript 

embedded within the HTML code. This tool allows the L WO to easily and efficiently navigate among the 

charts and spreadsheet files containing the model performance statistics. The objective statistics give the 

LWOs knowledge of the model's strengths and weaknesses and the GUI allows quick access to the data 

which will result in improved forecasts for operations. 

28 
References 

Bauman, W. H., W. P. Roeder, R. A. Lafosse, D. W. Sharp, and F. J. Merceret, 2004: The Applied 

Meteorology Unit - Operational Contributions to Spaceport Canaveral. Preprints, 11th 

Conference on Aviation, Range, and Aerospace Meteorology, Amer. Meteor. Soc., Hyannis, MA, 

4-8 October 2004, 24 pp. 

Insightful Corporation, 2007: S-PLUS® 8 for Windows® User's Guide, Insightful Corp., Seattle, WA, 584 

pp. 

29 
List of Acronyms 

45 WS 45th Weather Squadron MesoNAM 12-km resolution NAM 

AMU Applied Meteorology Unit NAM North American Mesoscale 

CCAFS Cape Canaveral Air Force Station MSE Mean Square Error 

GUI Graphical User Interface NM Nautical Mile 

HTML Hyper-Text Markup Language paR Period of Record 

KITS Shuttle Landing Facility 4-letter QC Quality Control 

identifier RMSE Root Mean Square Error 

KSC Kennedy Space Center WRF Weather Research and Forecasting 

KXMR CCAFS rawinsonde 4-letter identifier 

LWO Launch Weather Officer 

30 
NOTICE 

Mention of a copyrighted, trademarked or proprietary product, service, or document does not constitute 

endorsement thereof by the author, ENSCO Inc., the AMU, the National Aeronautics and Space 

Administration, or the United States Government. Any such mention is solely for the purpose of fully 

informing the reader of the resources used to conduct the work reported herein. 

31