NOAA Technical Memorandum NOS CS 25

AUTOMATED EVALUATION OF WATER LEVEL FORECAST 

GUIDANCE FROM NWS AND EXTRAMURAL REAL TIME OCEAN 

FORECAST SYSTEMS

Silver Spring, Maryland 

January 2011 

noaa National Oceanic and Atmospheric Administration 

U.S. DEPARTMENT OF COMMERCE 

National Ocean Service 

Coast Survey Development Laboratory 
Office of Coast Survey 

National Ocean Service 

National Oceanic and Atmospheric Administration 

U.S. Department of Commerce

The Office of Coast Survey (OCS) is the Nation’s only official chartmaker. As the 

oldest United States scientific organization, dating from 1807, this office has a long 

history. Today it promotes safe navigation by managing the National Oceanic and 

Atmospheric Administration’s (NOAA) nautical chart and oceanographic data 

collection and information programs. 

There are four components of OCS: 

The Coast Survey Development Laboratory develops new and efficient 

techniques to accomplish Coast Survey missions and to produce new and 

improved products and services for the maritime community and other coastal 

users. 

The Marine Chart Division acquires marine navigational data to construct and 

maintain nautical charts, Coast Pilots, and related marine products for the 

United States. 

The Hydrographic Surveys Division directs programs for ship and shore-based 

hydrographic survey units and conducts general hydrographic survey 

operations. 

The Navigational Services Division is the focal point for Coast Survey 

customer service activities, concentrating predominately on charting issues, 

fast-response hydrographic surveys, and Coast Pilot updates. 
NOAA Technical Memorandum NOS CS 25 

AUTOMATED EVALUATION OF WATER LEVEL FORECAST 

GUIDANCE FROM NWS AND EXTRAMURAL REAL TIME OCEAN 

FORECAST SYSTEMS

Richard A. Schmalz, Jr. and Philip H. Richardson

Office of Coast Survey, Coast Survey Development Laboratory, 

Silver Spring, MD 

January 2011 

noaa National Oceanic and Atmospheric Administration 

U. S. DEPARTMENT National Oceanic and National Ocean Service 

OF COMMERCE Atmospheric Administration David Kennedy 

Gary Locke, Dr. Jane Lubchenco Acting Assistant 

Secretary Under Secretary Administrator 

 

Office of Coast Survey Coast Survey Development Laboratory 

Captain John Lowell, NOAA Mary Erickson
ii

NOTICE 

Mention of a commercial company or product does not constitute an 

endorsement by NOAA. Use for publicity or advertising purposes of 

information from this publication concerning proprietary products or 

the tests of such products is not authorized.
iii

TABLE OF CONTENTS

LIST OF FIGURES ........................................................................................................... iv 

LIST OF TABLES .......................................................................................................... viii

ABSTRACT ...................................................................................................................... ix 

1. INTRODUCTION ...........................................................................................................1 

2. AUTOMATED WATER LEVEL EVALUATION ........................................................7 

3. ETSS MODEL MONTHLY ANALYSIS ....................................................................17 

4. RTOFS MODEL SEASONAL ANALYSIS ................................................................31 

 

5. ETSS, RTOFS, AND G-NCOM MODELS ANNUAL ANALYSIS ..........................41 

 

6. ADDITIONAL STATISTICAL MEASURES ............................................................53 

 

7. CONCLUSIONS AND RECOMMENDATIONS ......................................................67 

 

ACKNOWLEDGMENTS .................................................................................................68 

 

REFERENCES ..................................................................................................................68
iv

LIST OF FIGURES 

Figure 1.1. RTOFS Grid with spacing from 5 to 17 km ...................................................... 3 

Figure 1.2. ADCIRC Western North Atlantic Grid along the East Coast 

 with resolution ranging from 0.5 to 44.5 km ..................................................... 3 

Figure 1.3. ETSS East Coast Grid with order 10 km coastal resolution .............................. 4 

Figure 1.4. High Resolution ~6 km ETSS East Coast Grid in the Vicinity of 

 Chesapeake and Delaware Bays ........................................................................ 4 

Figure 1.5. ETSS Gulf of Mexico Grid with order 10 km coastal resolution ...................... 5 

Figure 1.6. G-NCOM Grid in the Atlantic Region 1/8 degree ~ 10km ............................... 5 

Figure 2.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008 ...... 9 

Figure 2.2. RTOFS Total Water Level Forecast Guidance against NOS Observations 

 for November 2008 ........................................................................................... 10 

Figure 2.3. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008 .... 11 

Figure 2.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations 

 for November 2008 ........................................................................................... 12 

Figure 2.5. G-NCOM Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008 .... 13 

Figure 2.6. G-NCOM Total Water Level Forecast Guidance against NOS Observations 

 for November 2008 ........................................................................................... 14 

Figure 2.7. ADCIRC Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008 .... 15 

Figure 2.8. ADCIRC Total Water Level Forecast Guidance against NOS Observations 

 for November 2008 ........................................................................................... 16 

Figure 3.1. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for February 2009 ...... 20 

Figure 3.2. ETSS Subtidal Water Level Forecast Guidance against NOS Observations 

 for February 2009 ............................................................................................. 21 
v

LIST OF FIGURES (Cont.)

Figure 3.3. ETSS Subtidal Water Level Guidance RMSE (meters) for East Coast 

 Stations (a) and for Gulf of Mexico Stations (b) for May 2009 ......................21 

Figure 3.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations

 for May 2009....................................................................................................24 

Figure 3.5. ETSS Subtidal Water Level Guidance RMSE (meters) for East Coast 

 Stations (a)and for Gulf of Mexico Stations (b) for August 2009 ...................26 

Figure 3.6. ETSS Subtidal Water Level Forecast Guidance against NOS Observations 

 for August 2009 ...............................................................................................27 

 

Figure 4.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for February 2009 ....32 

 

Figure 4.2. RTOFS Total Water Level Forecast Guidance against NOS Observations 

 for February 2009 ............................................................................................33 

 

Figure 4.3. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for May 2009 ............35 

 

Figure 4.4. RTOFS Total Water Level Forecast Guidance against NOS Observations 

 for May 2009....................................................................................................36 

 

Figure 4.5. RTOFS Total Water Level Guidance RMSE (meters) for East Coast 

 Stations (a) and for Gulf of Mexico Stations (b) for August 2009 .................38 

 

Figure 4.6. RTOFS Total Water Level Forecast Guidance against NOS Observations 

 for August 2009 ...............................................................................................39 

 

Figure 5.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2009 ...43 

 

Figure 5.2. RTOFS Total Water Level Forecast Guidance against NOS Observations 

 for November 2009 ..........................................................................................44 

 

Figure 5.3. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East 

 Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2009 ...46 

 

Figure 5.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations 

 for November 2009 ..........................................................................................47 

 
vi

LIST OF FIGURES (Cont.)

Figure 5.5. G-NCOM Total Water Level Forecast Guidance RMSE (meters) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2009 ..........................................................................................49 

Figure 5.6. G-NCOM Total Water Level Forecast Guidance against NOS 

 Observations for November 2009 ....................................................................50 

 

Figure 6.1. RTOFS Total Water Level Forecast Guidance Relative Error (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................55 

 

Figure 6.2. RTOFS Total Water Level Forecast Guidance Relative RMSE (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................56 

 

Figure 6.3. RTOFS Total Water Level Forecast Guidance CF (15 cm reference level) 

 for EastCoast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................57 

 

Figure 6.4. ETSS Subtidal Water Level Forecast Guidance Relative Error (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 .........................................................................................58 

 

Figure 6.5. ETSS Subtidal Water Level Forecast Guidance Relative RMSE (meters) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................59 

 

Figure 6.6. ETSS Subtidal Level Forecast Guidance CF (15 cm reference level) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................60 

 

Figure 6.7. G-NCOM Total Water Level Forecast Guidance Relative Error (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................61 

 

Figure 6.8. G-NCOM Total Water Level Forecast Guidance Relative RMSE Error (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................62 

 

Figure 6.9. G-NCOM Total Water Level Forecast Guidance CF (15 cm reference level) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ..........................................................................................63 

 
vii

LIST OF FIGURES (Cont.)

Figure 6.10. ADCIRC Total Water Level Forecast Guidance Relative Error (-) for 

 East Coast Stations (a)and for Gulf of Mexico Stations (b) 

 for November 2008 ........................................................................................64 

 

Figure 6.11. ADCIRC Total Water Level Forecast Guidance Relative RMSE Error (-) 

 for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ........................................................................................65 

 

Figure 6.12. ADCIRC Total Water Level Forecast Guidance CF (15 cm reference 

 level) for East Coast Stations (a) and for Gulf of Mexico Stations (b) 

 for November 2008 ........................................................................................66 
viii

LIST OF TABLES 

Table 2.1. ETSS, RTOFS, ADCIRC, and G-NCOM Forecast Guidance 

 Monthly Water Level Comparisons to NOS Observations 

 for November 2008 (RMSE[meters]) ................................................................7 

Table 3.1. ETSS Monthly Subtidal Water Level Forecast Guidance 

 Comparisons to NOS Observations 

 for December 2008 – November 2009 (RMSE[meters]) .................................17 

Table 3.2. ETSS December 2008 – November 2009 Monthly Subtidal 

 Water Level Forecast RMS Errors (cm): Minimum, 

 Maximum, and Range .......................................................................................29 

Table 4.1. RTOFS Seasonal Water Level Forecast Guidance 

 Comparisons to NOS Observations (RMSE[meters]) .....................................31 

Table 5.1. ETSS, RTOFS, and G-NCOM Forecast Guidance Monthly Water Level 

 Comparisons to NOS Observations 

 for November 2008 and 2009 (RMSE[meters]) ..............................................41 
ix

ABSTRACT 

The previous water level forecast guidance evaluation procedures (Richardson and 

Schmalz, 2009) were streamlined by combining into a single program the following 

separate programs: read, reformat, filter, adjust, and analysis programs. The plot program 

was modified to be run within a script to further simplify the evaluation process. The 

National Weather Service (NWS) Real Time Ocean Forecast System (RTOFS), NWS 

Extra-Tropical Storm Surge (ETSS) Model, United States Navy (USN) global Navy 

Coupled Ocean Model (G-NCOM), and University of North Carolina (UNC) Western 

North Atlantic Forecast System Advanced Circulation (ADCIRC) Model water level 

forecasts were compared with the observations at 24 National Ocean Service (NOS) 

water level stations on the U.S. East Coast and along the Gulf of Mexico shoreline using 

the new automated procedures. Because the ETSS model is subtidal, the ETSS forecast 

water levels were compared with observed subtidal water levels. The observed subtidal 

water levels were obtained using a 30 hour low pass Fourier filter. Because the RTOFS, 

ADCIRC, and G-NCOM models include tides, their forecast water levels were compared 

with the total observed water level. RTOFS, ETSS, ADCIRC, and G-NCOM model 

results were evaluated with the new procedures for November 2008 to verify that the 

results were consistent with the original evaluation. In addition, the new streamlined 

procedures were tested for monthly (ETSS), seasonal (RTOFS), and yearly (ETSS, 

RTOFS, and G-NCOM) water level evaluation. As a further enhancement, additional 

statistical measures of relative RMSE, Willmott et al. (1985) relative error, and central 

frequency were incorporated. 
x
1

1. INTRODUCTION 

Coastal ocean and estuarine nowcast/forecast systems require specification of the 

offshore boundary conditions for water level, density, and currents. These boundary 

conditions must be provided from basin scale or global numerical models, since 

observation density is insufficient in time and space to meet the necessary requirements. 

We consider here the evaluation of water level forecast guidance and to facilitate the 

discussion briefly review the characteristics of the ocean forecast models considered. 

The Real Time Ocean Forecast System (RTOFS) Atlantic application run at the National 

Centers for Environmental Prediction (NCEP) uses the Hybrid Coordinate Ocean Model 

(HYCOM) with1200 x 1684 points in the horizontal as shown in Figure 1.1 and 18 

isopyncnal and 7 z-levels in the vertical. Surface forcings, in the form of 10-m winds and 

sea-level atmospheric pressure, are from the 3-hour NCEP Global Forecast System 

(GFS), a numerical weather prediction modeling system. The open boundaries are 

relaxed to NCEP climatology. Tides are included in terms of tidal potential and 

boundary tides specified in terms of the M2, S2, N2, K2, K1, P1, O1, and Q1 tidal 

constituents. River inputs are specified from US Geological Survey (USGS) daily 

streamflow data and climatology. In the previously analyzed operational version 

(Richardson and Schmalz, 2007; Richardson and Schmalz, 2009), SST data from the 

GOES AVHRR are assimilated in RTOFS. In the November 2009 operational version, 

improvements to the tidal dynamics have been made and SSH data assimilation has been 

incorporated. Refer to Bleck et al. (2002) for further details regarding the HYCOM 

model development and computational algorithms. River inflow data for U.S. rivers are 

from the USGS and from the RivDIS climatology (http://www.rivdis.sr.unh.edu/) for foreign 

rivers. Surface forcing is provided by the NCEP Global Forecast System (GFS) 3 hourly 

model output. Each cycle produces a 24 hour nowcast, and a 120 hour forecast forced by 

the NWS NAM model. 

The University of North Carolina (UNC) ADCIRC western North Atlantic forecast 

model is a two-dimensional vertically integrated finite element model. The domain 

covers the western Atlantic Ocean from 60 degrees West longitude and contains the Gulf 

of Mexico and spans 7.9 to 45.8 degrees North latitudes. A region of the grid along the 

East Coast is shown in Figure 1.2. Tidal forcings include the M2, S2, N2, K2, K1, 2N2, O1, 

and Q1 tidal constituents (Edwards and Blain, 2002). See Luettich et al. (1992) and Kolar 

et al. (1994) for computational details. The system is run four times a day and is forced 

by the NWS North American Mesoscale (NAM) 10-meter winds. Each cycle produces a 

24 hour nowcast, and an 84 hour forecast forced by the NWS NAM model. 

NOS/Coast Survey Development Laboratory (CSDL) has utilized the subtidal water level 

forecast guidance produced by the NWS Meteorological Development Laboratory 

(MDL) Extra-Tropical Storm Surge (ETSS) Model for offshore water level boundary 

conditions for the New York Harbor /Port of New Jersey, Chesapeake Bay, and 

Galveston Bay operational forecast systems. Separate domains are run for the East Coast 

as shown in Figures 1.3 and 1.4, West Coast, Alaskan Coast, and Gulf of Mexico as via 

finite differences on an elliptical grid. See Chen et al. (1993) for additional model shown 

in Figure 1.5. The NWS GFS is used to provide the meteorological forcings. The two-
2

dimensional depth averaged shallow water equations are solved in complex variables 

details. The system is run four times a day with each forecast cycle extending to 96 

hours. 

The U.S. Navy (USN) Global Coastal Ocean Model (G-NGOM) system in Region 1 

contains the western North Atlantic and Gulf of Mexico and is run on a 1/8 by 1/8 degree 

grid as shown in Figure 1.6 based on the Princeton Ocean Model (POM) using 41 layers. 

See Blumberg and Mellor (1987) and Blumberg and Herring (1987) for computational 

details and Martin (2000) for operational implementation. The daily 00Z forecast cycle 

has a horizon out 72 hours and is forced by U.S. Navy Operational Global Atmospheric 

Prediction System (NOGAPS) winds and Multi-Channel Sea Surface Temperature 

(MCSST). Barotropic tides are added from the Oregon State global 1⁄4 by 1⁄4 degree 

TPXO6.2 tidal model (Egbert and Erofeeva, 2002) for the same eight constituents used in 

RTOFS as well as two long period constituents (Mf, Mm). 

Here, we automate the evaluation of the above ocean nowcast/forecast models to assess 

their ability to provide water level boundary conditions. The previous water level forecast 

guidance evaluation procedures (Richardson and Schmalz, 2009) were streamlined into a 

single analysis program for each of the ocean models, the previous separate read, 

reformat, filter, adjust, and analysis programs. The plot program was modified to be run 

within a script to further simplify the evaluation process. 

For November 2008, we evaluated daily forecast/nowcast results of both subtidal (ETSS) 

and total water level response (RTOFS, G-NCOM, and ADCIRC) along the East coast 

and along the Gulf of Mexico coast from hours 6-36 for all models using the new 

automated procedures. Results are compared in Chapter 2 to the previous results 

obtained by Richardson and Schmalz (2009) to verify the automated water level 

procedures. 

Chapter 3 presents a monthly evaluation of the ETSS water level forecast guidance over 

the period December 2008 – October 2009. In Chapter 4, a seasonal analysis of the 

RTOFS forecast guidance is performed by adding the analysis for the months February, 

May, and August 2009 to the November 2008 analysis of Chapter 1. In Chapter 5, an 

annual analysis of ETSS, RTOFS, and G-NCOM is performed by comparing the 

November 2008 results of Chapter 1 to an analysis of results for November 2009. In 

Chapter 6, we present the extension of the evaluation procedures to include the additional 

statistical measures of relative RMSE, Willmott et al. (1985) relative error, and central 

frequency (Hess et al., 2003). Chapter 6 presents some conclusions drawn from the work 

as well as recommendations for routine ocean model water level evaluation. 
3

Figure 1.1. RTOFS Grid with spacing from 5 to 17 km. 

Figure 1.2. ADCIRC Western North Atlantic Grid along the East Coast with resolution 

ranging from 0.5 to 45 km. The Gulf of Mexico and Caribbean regions are not 

shown. 

GA

NJ 
4

Figure 1.3. ETSS East Coast Grid with order 10 km coastal resolution. Area coverage is 

the same for the present order 5 km coastal resolution model, which is 

analyzed in this report. 

Figure 1.4. High Resolution ~6 km ETSS East Coast Grid in the Vicinity of Chesapeake 

and Delaware Bays. The high resolution ETSS model is analyzed in this 

report. 
5

Figure 1.5. ETSS Gulf of Mexico Grid with order 10 km coastal resolution, which is 

analyzed in this report. 

Figure 1.6. G-NCOM Grid in the Mid-Atlantic Region 1/8 degree ~ 10km. Current 

speeds and directions are shown for 16 September 2009 at each grid cell to 

illustrate the grid resolution. Gulf of Mexico and other regions are not shown. 
6
7

2. AUTOMATED WATER LEVEL EVALUATION 

An automated water level evaluation procedure has been developed by streamlining the 

previous water level forecast guidance evaluation process (Richardson and Schmalz, 

2009). For each of the ocean forecast models a single analysis program was developed by 

combining the previous separate read, reformat, filter, adjust, and analysis programs. The 

plot program was modified to be run within a script environment to allow passing as an 

argument the ocean model to be considered. This further simplifies the evaluation plot 

process. Water level data acquisition was also simplified by using STEP 2 of the NOS 

skill assessment software (Zhang et al., 2006) to automatically download the water level 

data for all stations at once. Previous efforts relied on acquiring for each station the data 

from the CO-OPS website, reformatting the data to remove the HTML tags, and then 

transferring the individual station files to the Linux servers for analysis. The root mean 

square errors (RMSEs) from the automated water level evaluation are given in Table 2.1 

for each of the ocean model forecast systems and are identical to those obtained 

previously by Richardson and Schmalz (2009) verifying the new automated procedures. 

Table 2.1. ETSS, RTOFS, ADCIRC, and G-NCOM Forecast Guidance Monthly Water 

Level Comparisons to NOS Observations for November 2008 

(RMSE[meters]). Forecast hours 6-36 are used in the comparisons. ETSS 

water levels are subtidal, while the other three forecasts include the tide. 

Station ETSS RTOFS ADCIRC G-NCOM 

Eastport, ME 0.078 3.476 0.644 1.044 

Portland, ME 0.076 1.870 0.288 0.184 

Boston, MA 0.074 1.890 0.302 0.376 

Woods Hole, MA 0.064 0.320 0.183 0.118 

Sandy Hook, NJ 0.094 0.968 0.236 0.196 

Atlantic City, NJ 0.083 0.770 0.100 0.131 

Cape May, NJ 0.228 1.042 0.335 0.435 

Lewes, DE 0.072 0.865 0.109 0.276 

Duck, NC 0.056 0.622 0.111 0.220 

Wilmington, NC 0.058 1.099 0.670 0.632 

Springmaid Pier, SC 0.050 1.042 0.131 0.136 

Charleston, SC 0.053 1.191 0.239 0.237 

Fort Pulaski, GA 0.054 1.638 0.286 0.326 

Fernandina Beach, 

FL 

0.056 1.546 0.356 0.272 

Naples, FL 0.044 0.645 0.156 0.126 

St Petersburg FL 0.098 0.750 0.271 0.399 

Clearwater, FL 0.049 0.789 0.173 0.243 

Apalachicola, FL 0.101 0.210 0.129 0.166 

Panama City, FL 0.036 0.137 0.121 0.097 

Pensacola, FL 0.033 0.207 0.085 0.153 

Sabine Pass, TX 0.084 0.618 0.181 0.121 

Galveston Pleasure 

Pier, TX 

0.065 0.470 0.120 0.083 

Freeport, TX 0.060 0.217 0.125 0.099 
8

RTOFS 

The RMSE results for RTOFS from the modified plotting procedures are shown in Figure 

2.1 for both Atlantic and Gulf of Mexico stations. One notes the large RMSEs on the 

order of 80 cm for the East Coast stations due to the errors in the tidal dynamics. To 

further verify the automated results, plots are shown in Figure 2.2 of RTOFS forecast 

water levels versus observations for an Atlantic station (Sandy Hook, NJ) and a Gulf of 

Mexico station (Clearwater, FL). Both of these plots indicate an issue with the tide. 

ETSS 

The RMSE results for ETSS are shown in Figure 2.3 and indicate errors in water level 

levels on the order of 10 cm for both Atlantic and Gulf of Mexico stations. To further 

verify the automated results, plots ) are shown in Figure 2.4 of ETSS forecast subtidal 

water levels versus observations for an Atlantic station (Sandy Hook, NJ) and a Gulf of 

Mexico station (Clearwater, FL. Both of these plots indicate an excellent agreement 

between ETSS subtidal water level forecasts and observations. 

G-NCOM 

The RMSE results for G-NCOM are shown in Figure 2.5 and indicate errors in water 

levels on the order of 20 cm for the Atlantic and 10 cm for Gulf of Mexico stations. To 

further verify the automated results, plots are shown in Figure 2.6 of G-NCOM forecast 

water levels versus observations for an Atlantic station (Sandy Hook, NJ) and a Gulf of 

Mexico station (Clearwater, FL). Both of these plots indicate a reasonable agreement 

between G-NCOM total water level forecasts and observations. While forecasts are at 

three hour intervals, the tidal dynamics seem to be reasonably captured. 

ADCIRC 

The RMSE results for UNC ADCIRC are shown in Figure 2.7 and indicate errors in 

water levels on the order of 10 cm for both Atlantic and Gulf of Mexico stations. To 

further verify the automated results, plots of ADCIRC forecast water levels versus 

observations for an Atlantic station (Sandy Hook, NJ) and a Gulf of Mexico station 

(Clearwater, FL) are shown in Figure 2.8. Both of these plots indicate an excellent 

agreement between the ADCIRC total water level forecasts and observations. 
9

(a) 

(b)

Figure 2.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2008. Note 

there are no observations at Mayport, FL. 
10

(a) 

(b) 

Figure 2.2. RTOFS Total Water Level Forecast Guidance against NOS Observations for 

November 2008. Sandy Hook, NJ (a) Clearwater Beach FL (b). Note 

sequential forecasts hours 6 to 36 are illustrated at hourly intervals by a 

repeating sequence of symbols (+ , *, Δ , and ). 
11

(a) 

(b) 

Figure 2.3. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2008. Note 

there are no observations at Mayport, FL. 
12

(a) 

(b) 

Figure 2.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations for 

November 2008. Sandy Hook, NJ (a) Clearwater Beach, FL (b). Note 

sequential forecasts hours 6 to 36 are illustrated at hourly intervals by a 

repeating sequence of symbols (+ , *, Δ , and ). At this vertical scale the 

model results appear to be represented by +. 
13

(a) 

(b) 

Figure 2.5. G-NCOM Total Water Level Forecast Guidance RMSE (meters) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008.

Note there are no observations at Mayport, FL. 
14

(a) 

(b) 

Figure 2.6. G-NCOM Total Water Level Forecast Guidance against NOS Observations 

for November 2008. Sandy Hook, NJ (a) Clearwater Beach FL (b). Note 

sequential forecasts hours 6 to 36 are illustrated at hourly intervals by a 

repeating sequence of symbols (+ , *, Δ , and ).
15

(a)

(b) 

Figure 2.7. ADCIRC Total Water Level Forecast Guidance RMSE (meters) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 

Note there are no observations at Mayport, FL. 
16

(a) 

(b) 

Figure 2.8. ADCIRC Total Water Level Forecast Guidance against NOS Observations for 

November 2008. Sandy Hook, NJ (a) Clearwater Beach, FL (b). Note 

sequential forecasts hours 6 to 36 are illustrated at hourly intervals by a 

repeating sequence of symbols (+ , *, Δ , and ). 
17

3. ETSS MODEL MONTHLY ANALYSIS 

To further exercise the automated water level evaluation procedures, ETSS was evaluated 

on a monthly basis over the period December 2008 to November 2009. The major issue 

in the timeliness of the evaluation is the availability of the NOS verified hourly data. It 

usually is available for all 24 stations at the middle of the following month. Monthly 

RMSEs are given in Table 3.1 and exhibit very little change from month to month. There 

is some indication that along the northeast Atlantic coast, the errors are largest during the 

winter season. To further investigate the ETSS subtidal water level response, RMSE error 

plots for the Atlantic and Gulf of Mexico stations as well as time series plots at two 

Atlantic coast stations (Sandy Hook, NJ and Mayport, FL) and two Gulf of Mexico 

stations (Clearwater Beach, FL and Galveston Pleasure Pier, TX) are shown in Figure 3.1 

and Figure 3.2 for February 2009, in Figure 3.3 and Figure 3.4 for May 2009, and in 

Figure 3.5 and 3.6 for August 2009. Excellent agreement in the predicted subtidal water 

levels is achieved during all months. 

Table 3.1. ETSS Monthly Subtidal Water Level Forecast Guidance Comparisons to NOS 

Observations for December 2008 – March 2009 (RMSE[meters]). Blanks 

indicate months for which observations were not available, while * denotes 

problems with water level observations.

Station December 2008 January February March 

Eastport, ME 0.102 0.838* 0.082 0.055 

Portland, ME 0.095 0.084 0.075 0.047 

Boston, MA 0.105 0.085 0.079 0.051 

Woods Hole, MA 0.156 0.089 0.094 0.060 

Sandy Hook, NJ 0.099 0.108 0.095 0.066 

Atlantic City, NJ 0.102 0.103 0.060 

Cape May, NJ 0.099 0.098 0.094 0.066 

Lewes, DE 0.092 0.097 0.094 0.061 

Duck, NC 0.065 0.080 0.093 0.061 

Wilmington, NC 0.106 0.110 0.091 0.073 

Springmaid Pier, 

SC 

0.060 0.069 0.061 0.064 

Charleston, SC 0.065 0.072 0.068 0.087 

Fort Pulaski, GA 0.065 0.051 0.052 0.080 

Fernandina Beach, 

FL 

0.043 0.051 0.046 0.063 

Mayport, FL 0.052 0.045 0.046 0.048 

Naples, FL 0.059 0.053 0.042 0.056 

St Petersburg FL 0.139 0.117 0.094 0.088 

Clearwater, FL 0.091 0.064 0.052 0.063 

Apalachicola, FL 0.128 0.116 0.119 0.096 

Panama City, FL 0.067 0.063 0.046 0.049 

Pensacola, FL 0.067 0.059 0.055 0.055 

Sabine Pass, TX 0.117 0.074 0.087 0.100 

Galveston Pleasure 

Pier, TX 

0.095 0.066 0.065 0.085 
18

Table 3.1. (Cont.) ETSS Monthly Subtidal Water Level Forecast Guidance Comparisons 

to NOS Observations for April through July 2009 (RMSE[meters]).

Station April May June July 

Eastport, ME 0.065 0.043 0.029 0.049 

Portland, ME 0.052 0.041 0.030 0.035 

Boston, MA 0.053 0.043 0.028 0.039 

Woods Hole, MA 0.063 0.044 0.041 0.038 

Sandy Hook, NJ 0.061 0.042 0.042 0.047 

Atlantic City, NJ 0.066 0.044 0.047 0.058 

Cape May, NJ 0.071 0.040 0.047 0.058 

Lewes, DE 0.076 0.042 0.047 0.059 

Duck, NC 0.078 0.055 0.039 0.042 

Wilmington, NC 0.101 0.072 0.044 0.050 

Springmaid Pier, 

SC 

0.066 0.055 0.045 0.048 

Charleston, SC 0.078 0.069 0.049 0.058 

Fort Pulaski, GA 0.065 0.070 0.047 0.059 

Fernandina Beach, 

FL 

0.063 0.078 0.049 0.060 

Mayport, FL 0.053 0.056 0.039 0.049 

Naples, FL 0.044 0.042 0.030 0.033 

St Petersburg FL 0.078 0.056 0.034 0.030 

Clearwater, FL 0.060 0.067 0.027 0.032 

Apalachicola, FL 0.079 0.082 0.028 0.025 

Panama City, FL 0.043 0.047 0.026 0.029 

Pensacola, FL 0.047 0.047 0.025 0.022 

Sabine Pass, TX 0.117 0.048 0.040 0.037 

Galveston 

Pleasure Pier, TX 

0.091 0.047 0.043 0.035 
19

Table 3.1. (Cont.) ETSS Monthly Subtidal Water Level Forecast Guidance Comparisons 

to NOS Observations for August through November 2009 (RMSE[meters]). 

Station August September October November 

Eastport, ME 0.052 0.053 0.079 0.059 

Portland, ME 0.040 0.044 0.063 0.051 

Boston, MA 0.249* 0.248* 0.067 0.055 

Woods Hole, MA 0.035 0.043 0.069 0.054 

Sandy Hook, NJ 0.042 0.045 0.087 0.066 

Atlantic City, NJ 0.040 0.045 0.069 0.070 

Cape May, NJ 0.045 0.050 0.069 0.072 

Lewes, DE 0.043 0.071 0.078 0.076 

Duck, NC 0.040 0.058 0.066 0.090 

Wilmington, NC 0.036 0.068 0.033 

Springmaid Pier, 

SC 

0.044 0.055 0.060 0.071 

Charleston, SC 0.051 0.050 0.065 0.075 

Fort Pulaski, GA 0.041 0.053 0.066 0.076 

Fernandina 

Beach, FL 

0.045 0.052 0.065 0.068 

Mayport, FL 0.036 0.041 0.053 0.053 

Naples, FL 0.035 0.026 0.038 0.046 

St Petersburg FL 0.037 0.031 0.063 0.082 

Clearwater, FL 0.038 0.031 0.108 0.056 

Apalachicola, FL 0.092 0.029 0.084 0.107 

Panama City, FL 0.050 0.028 0.041 0.064 

Pensacola, FL 0.040 0.032 0.042 0.066 

Sabine Pass, TX 0.036 0.039 0.075 0.090 

Galveston 

Pleasure Pier, TX 

0.038 0.043 0.068 0.083 
20

 

(a) 

(b) 

Figure 3.1. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for February 2009. Note there 

are no observations at Atlantic City, NJ and Freeport, TX. 
21

(a) 

(b) 

Figure 3.2. ETSS Subtidal Water Level Forecast Guidance against NOS Observations for 

February 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential 

forecasts hours 6 to 36 are illustrated at hourly intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). At this vertical scale the model results 

appear to be represented by +. 
22

(c) 

(d) 

Figure 3.2. (Cont.) ETSS Subtidal Water Level Forecast Guidance against NOS 

Observations for November 2009. Clearwater Beach, FL (c) Galveston 

Pleasure Pier, TX. Note sequential forecasts hours 6 to 36 are illustrated at 

hourly intervals by a repeating sequence of symbols (+ , *, Δ , and ). At this 

vertical scale the model results appear to be represented by +. 
23

(a) 

(b) 

Figure 3.3. ETSS Subtidal Water Level Guidance RMSE (meters) for East Coast Stations 

(a) and for Gulf of Mexico Stations (b) for May 2009. Note there are no 

observations at Freeport, TX. 
24

(a) 

(b) 

Figure 3.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations for 

May 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential forecasts 

hours 6 to 36 are illustrated at hourly intervals by a repeating sequence of 

symbols (+ , *, Δ , and ). At this vertical scale the model results appear to be 

represented by +. 
25

(c) 

(d) 

Figure 3.4. (Cont.) ETSS Subtidal Water Level Forecast Guidance against NOS 

Observations for May 2009. Clearwater Beach, FL (c) Galveston Pleasure 

Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at hourly 

intervals by a repeating sequence of symbols (+ , *, Δ , and ). At this vertical 

scale the model results appear to be represented by +. 
26

(a) 

(b) 

Figure 3.5. ETSS Subtidal Water Level Guidance RMSE (meters) for East Coast Stations 

(a) and for Gulf of Mexico Stations (b) for August 2009. Note there are no 

observations at Mayport, FL and Freeport, TX. 
27

(a)

(b) 

Figure 3.6. ETSS Subtidal Water Level Forecast Guidance against NOS Observations for 

August 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential 

forecasts hours 6 to 36 are illustrated at hourly intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). At this vertical scale the model 

results appear to be represented by +. 
28

(c) 

(d) 

Figure 3.6. (Cont.) ETSS Subtidal Water Level Forecast Guidance against NOS 

Observations for August 2009. Clearwater Beach, FL (c) Galveston Pleasure 

Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at hourly 

intervals by a repeating sequence of symbols (+ , *, Δ , and ). At this vertical 

scale the model results appear to be represented by +. 
29

In Table 3.2, the minimum, maximum, and range of the monthly RMSEs over the entire 

one year period from December 2008 through November 2009 are examined. All 

maximum RMSE errors are below 15 cm except at Woods Hole, MA and the majority are 

on the order of 10 cm and generally occur during the winter. Minimum RMSEs are on the 

order of 4 cm and occur during the late spring and summer. 

Table 3.2. ETSS December 2008 – November 2009 Monthly Subtidal Water Level 

Forecast RMS Errors (cm): Minimum, Maximum, and Range. *Denotes 

problems in water level observations for some months. 

Station MaximumMonth Minimum-Month Range 

Eastport, ME 10-December 3-June 7* 

Portland, ME 10-December 3-June 7 

Boston, MA 11-December 3-June 8* 

Woods Hole, 

MA 

16-December 4-June 12 

Sandy Hook, NJ 11-Jaunuary 4-May 7 

Atlantic City, 

NJ 

10-January 4-August 6 

Cape May, NJ 10-December 4-May 6 

Lewes, DE 10-January 4-May 6 

Duck, NC 9-February 4-June 5 

Wilmington, 

NC 

11-January 4-June 7 

Springmaid 

Pier, SC 

7-November 4-August 3 

Charleston, SC 9-March 5-June 4 

Fort Pulaski, 

GA 

8-March 4-August 4 

Fernandina 

Beach, FL 

8-May 4-December 4 

Mayport, FL 6-May 4-August 2 

Naples, FL 6-December 3-September 3 

St Petersburg 

FL 

14-December 3-July 11 

Clearwater, FL 11-October 3-June 8 

Apalachicola, 

FL 

13-December 3-July 10 

Panama City, 

FL 

7-December 3-June 4 

Pensacola, FL 7-December 3-September 4 

Sabine Pass, TX 12-December 4-August 8 

Galveston 

Pleasure Pier, 

TX 

10-December 4-July 6 
30

It should be noted that by performing a yearly analysis, one learns the error behavior at 

each station. If in one month, the results are drastically different, plotting the time series 

of water levels and observations usually indicates a problem with the water level 

observations. In Table 3.1 problems were noted at Eastport, ME in January 2009 and at 

Boston, MA during August and September 2009. 

Based on these results, the behavior of the monthly RMSEs associated with the ETSS 

subtidal water level forecasts have been well established. It may not be necessary to 

continue to evaluate this ocean forecast system, unless updates are made to the system. 

To our knowledge, no updates in the near future are planned. 
31

4. RTOFS MODEL SEASONAL ANALYSIS 

To further exercise the automated water level evaluation procedures, RTOFS was 

evaluated on a seasonal basis for the months of February, May, August, and November 

2009. Monthly RMSEs are given in Table 4.1 and exhibit very little change from month 

to month. To further investigate the RTOFS total water level response, RMSE error plots 

for the Atlantic and Gulf of Mexico stations as well as time series plots at two Atlantic 

coast stations (Sandy Hook, NJ and Mayport, FL) and two Gulf of Mexico stations 

(Clearwater Beach, FL and Galveston Pleasure Pier, TX) are shown in Figure 4.1 and 

Figure 4.2 for February 2009, in Figure 4.3 and Figure 4.4 for May 2009, and in Figure 

4.5 and Figure 4.6 for August 2009. One notes in these time series plots the improvement 

in the tidal dynamics from the results for November 2008 given in Figure 2.2. 

Table 4.1. RTOFS Seasonal Water Level Forecast Guidance Comparisons to NOS 

Observations (RMSE[meters]). Blanks indicate no observations are available. 

Station February 2009 May 2009 August 2009 November 2009 

Eastport, ME 2.227 2.500 2.356 2.035 

Portland, ME 1.254 1.421 1.307 1.161 

Boston, MA 1.297 1.456 1.418 1.215 

Woods Hole, MA 0.227 0.221 0.301 0.245 

Sandy Hook, NJ 0.457 0.447 0.463 0.379 

Atlantic City, NJ 0.329 0.297 0.336 

Cape May, NJ 0.348 0.335 0.367 0.263 

Lewes, DE 0.314 0.295 0.293 0.229 

Duck, NC 0.720 0.704 0.639 0.620 

Wilmington, NC 0.683 0.660 0.591 

Springmaid Pier, 

SC 

0.511 0.501 0.492 0.469 

Charleston, SC 0.601 0.587 0.553 0.560 

Fort Pulaski, GA 0.750 0.767 0.709 0.682 

Fernandina Beach, 

FL 

0.639 0.633 0.617 0.541 

Mayport, FL 0.430 0.421 0.429 0.360 

Naples, FL 0.541 0.414 0.405 0.533 

St Petersburg FL 0.649 0..588 0.629 0.576 

Clearwater, FL 0.330 0.264 0.290 0.303 

Apalachicola, FL 0.318 0.354 0.344 0.304 

Panama City, FL 0.210 0.178 0.176 0.209 

Pensacola, FL 0.284 0.221 0.230 0.262 

Sabine Pass, TX 0.464 0.437 0.496 0.420 

Galveston 

Pleasure Pier, TX 

0.349 0.368 0.379 0.318 
32

(a) 

(b) 

Figure 4.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for February 2009. Note no 

observations were available at Atlantic City, NJ. 
33

(a) 

(b) 

Figure 4.2. RTOFS Total Water Level Forecast Guidance against NOS Observations for 

February 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential 

forecasts hours 6 to 36 are illustrated at hourly intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). 
34

(c) 

(d) 

Figure 4.2. (Cont.) RTOFS Total Water Level Forecast Guidance against NOS 

Observations for February 2009. Clearwater Beach, FL (c) Galveston 

Pleasure Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at 

hourly intervals by a repeating sequence of symbols (+ , *, Δ , and ). 
35

(a) 

(b) 

Figure 4.3. RTOFS Total Water Level Guidance RMSE (meters) for East Coast Stations 

(a) and for Gulf of Mexico Stations (b) for May 2009. No observations were 

available at Freeport, TX. 
36

(a) 

(b) 

Figure 4.4. RTOFS Total Water Level Forecast Guidance against NOS Observations for 

May 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential forecasts 

hours 6 to 36 are illustrated at hourly intervals by a repeating sequence of 

symbols (+ , *, Δ , and ). 
37

(c) 

(d) 

Figure 4.4. (Cont.) RTOFS Total Water Level Forecast Guidance against NOS 

Observations for May 2009. Clearwater Beach, FL (c) Galveston Pleasure 

Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at hourly 

intervals by a repeating sequence of symbols (+ , *, Δ , and ).
38

 

(a) 

(b) 

Figure 4.5. RTOFS Total Water Level Guidance RMSE (meters) for East Coast Stations 

(a) and for Gulf of Mexico Stations (b) for August 2009. No observations 

were available at Freeport, TX. 
39

(a) 

(b) 

Figure 4.6. RTOFS Total Water Level Forecast Guidance against NOS Observations for 

August 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential forecasts 

hours 6 to 36 are illustrated at hourly intervals by a repeating sequence of 

symbols (+ , *, Δ , and ). 
40

(c) 

(d) 

Figure 4.6. (Cont.) RTOFS Total Water Level Forecast Guidance against NOS 

Observations for August 2009. Clearwater Beach, FL (c) and at Galveston 

Pleasure Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at 

hourly intervals by a repeating sequence of symbols (+ , *, Δ , and ). 
41

5. ETSS, RTOFS, AND G-NCOM MODELS ANNUAL ANALYSIS 

Here, we compare the results of Chapter 2 for November 2008 with those obtained a year 

later for November 2009 for all of the ocean models in Table 5.1 except UNC ADCIRC, 

whose forecasts were not available during November 2009. Note the inconsistency in 

RMSE at Cape May, NJ between November 2009 and November 2008 for both ETSS 

and G-NCOM. An examination of the times series plots for November 2008 indicated an 

issue with the Cape May, NJ water level observations. 

Table 5.1 ETSS, RTOFS, and G-NCOM Forecast Guidance Monthly Water Level 

Comparisons to NOS Observations for November 2008 and 2009 

(RMSE[meters]). ETSS water levels are subtidal, while RTOFS and G-NCOM 

water levels are total. Blanks indicate no observations are available. 

Station ETSS 

 11/ 2009 11/2008 

 RTOFS 

 11/ 2009 11/2008 

 G-NCOM 

 11/ 2009 11/2008 

Eastport, ME 0.059 0.078 2.035 3.476 1.041 1.044 

Portland, ME 0.051 0.076 1.162 1.870 0.151 0.184 

Boston, MA 0.055 0.074 1.215 1.890 0.378 0.376 

Woods Hole, 

MA 

0.054 0.064 0.245 0.320 0.124 0.118 

Sandy Hook, NJ 0.066 0.094 0.379 0.968 0.199 0.196 

Atlantic City, 

NJ 

0.070 0.083 0.336 0.770 0.121 0.131 

Cape May, NJ 0.072 0.228 0.263 1.042 0.348 0.435 

Lewes, DE 0.076 0.072 0.229 0.865 0.295 0.276 

Duck, NC 0.090 0.056 0.619 0.622 0.222 0.220 

Wilmington, 

NC 

 0.058 1.099 0.632 

Springmaid 

Pier, SC 

0.071 0.050 0.469 1.042 0.115 0.136 

Charleston, SC 0.075 0.053 0.560 1.191 0.258 0.237 

Fort Pulaski, 

GA 

0.076 0.054 0.682 1.638 0.329 0.326 

Fernandina 

Beach, FL 

0.068 0.056 0.541 1.546 0.302 0.272 

Mayport, FL 0.053 0.360 0.244 

Naples, FL 0.046 0.044 0.535 0.645 0.139 0.126 

St Petersburg, 

FL 

0.082 0.098 0.576 0.750 0.409 0.399 

Clearwater, FL 0.056 0.049 0.303 0.789 0.225 0.243 

Apalachicola, 

FL 

0.107 0.101 0.304 0.210 0.164 0.166 

Panama City, 

FL 

0.064 0.036 0.209 0.137 0.099 0.097 

Pensacola, FL 0.066 0.033 0.262 0.207 0.148 0.153 

Sabine Pass, TX 0.090 0.084 0.420 0.618 0.113 0.121 
42

Galveston 

Pleasure Pier, 

TX 

0.083 0.065 0.318 0.470 0.088 0.083 

Freeport, TX 0.060 0.217 0.099 

RMSE error plots are given in Figure 5.1, Figure 5.3, and in Figure 5.5, for ETSS, 

RTOFS, and G-NCOM for November 2009 and may be compared with those in Chapter 

2 for November 2008. Time series plots for November 2009 at two Atlantic coast stations 

(Sandy Hook, NJ and Mayport, FL) and two Gulf of Mexico stations (Clearwater Beach, 

FL and Galveston Pleasure Pier, TX) are shown in Figure 5.2, Figure 5.4, and Figure 5.6 

for ETSS, RTOFS, and G-NCOM, respectively, and may be compared with those of 

Chapter 2. While one notes the improvement in the RTOFS tidal dynamics in the 

November 2009 forecasts, there is still room for improvement compared to UNC 

ADCIRC and G-NCOM. 
43

(a) 

(b) 

Figure 5.1. RTOFS Total Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2009. 
44

(a) 

(b) 

Figure 5.2. RTOFS Total Water Level Forecast Guidance against NOS Observations for 

November 2009. Sandy Hook, NJ (a) Mayport, FL (b). Note sequential 

forecasts hours 6 to 36 are illustrated at hourly intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). 
45

(c) 

(d) 

Figure 5.2. (Cont.) RTOFS Total Water Level Forecast Guidance against NOS 

Observations for November 2009. Sandy Hook, NJ (c) Clearwater Beach, FL 

(d). Note sequential forecasts hours 6 to 36 are illustrated at hourly intervals 

by a repeating sequence of symbols (+ , *, Δ , and ). 
46

(a) 

(b) 

Figure 5.3. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2009. 
47

(a) 

(b) 

Figure 5.4. ETSS Subtidal Water Level Forecast Guidance against NOS Observations for 

November 2009. Sandy Hook, NJ (a). Mayport, FL (b) Note sequential 

forecasts hours 6 to 36 are illustrated at hourly intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). At this vertical scale the model results 

appear to be represented by +. 
48

(c) 

(d) 

Figure 5.4. (Cont.) ETSS Subtidal Water Level Forecast Guidance against NOS 

Observations for November 2009. Clearwater Beach, FL (c) Galveston 

Pleasure Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at 

hourly intervals by a repeating sequence of symbols (+ , *, Δ , and ). At this 

vertical scale the model results appear to be represented by +. 
49

(a) 

(b) 

Figure 5.5. G-NCOM Total Water Level Forecast Guidance RMSE (meters) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2009. 
50

(a) 

(b) 

Figure 5.6. G-NCOM Total Water Level Forecast Guidance against NOS Observations 

for November 2009. Sandy Hook, NJ (a) Mayport FL (b). Note sequential 

forecasts hours 6 to 36 are illustrated at 3-hour intervals by a repeating 

sequence of symbols (+ , *, Δ , and ). 
51

(c) 

(d) 

Figure 5.6. (Cont.) G-NCOM Total Water Level Forecast Guidance against NOS 

Observations for November 2009. Clearwater Beach, FL (c) Galveston 

Pleasure Pier, TX (d). Note sequential forecasts hours 6 to 36 are illustrated at 

3 hour intervals by a repeating sequence of symbols (+ , *, Δ , and ). 
52
53

6. ADDITIONAL STATISTICAL MEASURES 

To further aid in the evaluation, several additional statistical measures have been 

computed. The Willmott et al. (1985) dimensionless (0-1) relative error is computed as 

follows, 2 2 (Y  X ) / (Y  X  X  X ) where Y is the prediction and X the 

observation, with zero representing perfect agreement. While this statistic can be used to 

represent the overall agreement in shape of the model prediction time series curve to the 

observation time series curve, a difference in means will also affect the value. In fact, if 

both Y and X are constant and Y ≠ X then the relative error is 1. Therefore, it is possible 

for the relative error to be large when the RMSE is small. In general, we desire a value 

less than 0.05. In addition, a relative RMSE is computed as the RMSE divided by the 

standard deviation of the observations. It is desirable for this statistic to be less than one. 

Following NOS standard skill assessment procedures (Hess et al., 2003), the central 

frequency (CF) is also computed at a reference level of 15 cm. We desire a CF value 

greater than 0.9. The results for November 2008 are presented in turn below for each 

ocean model. 

6.1. RTOFS 

The relative error is shown for RTOFS in Figure 6.1. It is above 0.5 at all the East Coast 

stations, and is below 0.5 only at Panama City, FL and Freeport, TX along the Gulf of 

Mexico coast. The relative RMSE is shown in Figure 6.2. It is above 1.0 at all the East 

Coast stations, and is below 1.0 only at Panama City, FL and Freeport, TX along the Gulf 

of Mexico coast. The central frequency at a reference level of 15 cm is shown in Figure 

6.3. It is near 0.2 for most stations along the East Coast, and between 0.2 and 0.8 

(Panama City, FL) for the Gulf of Mexico coastal stations. These statistics confirm the 

findings, that there is significant error in the RTOFS November 2008 total water level 

forecasts, primarily due to the errors in the astronomical tide. 

6.2. ETSS 

The relative error is shown for the ETSS subtidal water level forecast in Figure 6.4. It is 

below 0.1 at all the East Coast stations, and is above 0.1 only at Apalachicola, FL and St. 

Petersburg, FL along the Gulf of Mexico coast. The relative RMSE is shown in Figure 

6.5. It is below 0.8 at all the East Coast stations except Cape May, NJ (data problem), and 

is below 0.8 along the Gulf of Mexico coast. The central frequency at a reference level of 

15 cm is shown in Figure 6.6. It is above 0.9, meeting the NOS standard from Hess et al. 

(2003) for all stations along the East Coast and Gulf of Mexico. These statistics confirm 

the findings that the quality of the ETSS subtidal water level forecasts serves as a 

benchmark for other ocean models to meet. 

6.3. G-NCOM 

The relative error is shown for the G-NCOM total water level forecasts in Figure 6.7. It 

is below 0.1 at all the East Coast stations except Cape May, NJ (data problem) and at 

Wilmington, NC (station located within the Cape Fear estuary), and is above 0.2 only at 
54

St. Petersburg, FL along the Gulf of Mexico coast. The relative RMSE is shown in Figure 

6.8. It is below 0.5 at all the East Coast stations except at Cape May, NJ (data problem) 

and at Wilmington, NC (station located within the Cape Fear estuary), and is below 0.4 

along the Gulf of Mexico coast. The central frequency at a reference level of 15 cm is 

shown in Figure 6.9. It is above 0.6 for all stations along the East Coast and Gulf of 

Mexico. At Freeport, TX and Galveston Pleasure Pier, TX, it exceeds 0.9, meeting the 

NOS standard. These statistics confirm the findings, that the quality of the G-NCOM 

total water level forecasts is substantially better than RTOFS. However, water levels are 

available only at a 3-hour data interval. 

6.4. ADCIRC 

The relative error is shown for the UNC ADCIRC total water level forecast in Figure 

6.10. It is below 0.1 at all the East Coast stations except at Wilmington, NC (station 

located within the Cape Fear estuary), and is above 0.2 only at St. Petersburg, FL along 

the Gulf of Mexico coast. The relative RMSE is shown in Figure 6.11. It is below 0.8 at 

all the East Coast stations except at Wilmington, NC (station located within the Cape 

Fear estuary), and is below 0.8 along the Gulf of Mexico coast except at St. Petersburg, 

FL. The central frequency at a reference level of 15 cm is shown in Figure 6.12. It is 

above 0.8 for all stations along the mid-Atlantic coast degrading to near 0.4 at each end 

and it is above 0.8 except for the stations between the panhandle and Naples, FL in the 

Gulf of Mexico. These statistics confirm the findings that the quality of the ADCIRC 

total water level forecasts is substantially better than RTOFS and is improved relative to 

G-NCOM at only the mid-Atlantic coast stations. 
55

(a) 

(b) 

Figure 6.1. RTOFS Total Water Level Forecast Guidance Relative Error (-) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
56

(a) 

(b) 

Figure 6.2. RTOFS Total Water Level Forecast Guidance Relative RMSE (-) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
57

(a) 

(b) 

Figure 6.3. RTOFS Total Water Level Forecast Guidance CF (15 cm reference level) for 

East Coast Stations (a) and for Gulf of Mexico Stations (b) for November 

2008. 
58

(a) 

(b) 

Figure 6.4. ETSS Subtidal Water Level Forecast Guidance Relative Error (-) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
59

(a) 

(b) 

Figure 6.5. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
60

(a) 

(b) 

Figure 6.6. ETSS Subtidal Water Level Forecast Guidance RMSE (meters) for East Coast 

Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
61

(a) 

(b) 

Figure 6.7. G-NCOM Total Water Level Forecast Guidance Relative Error (-) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
62

(a) 

(b) 

Figure 6.8. G-NCOM Total Water Level Forecast Guidance Relative RMSE Error (-) for 

East Coast Stations (a) and for Gulf of Mexico Stations (b) for November 

2008. 
63

(a) 

 (b) 

Figure 6.9. G-NCOM Total Water Level Forecast Guidance Relative RMSE Error (-) for 

East Coast Stations (a) and for Gulf of Mexico Stations (b) for November 

2008. 
64

(a) 

(b) 

Figure 6.10. ADCIRC Total Water Level Forecast Guidance Relative Error (-) for East 

Coast Stations (a) and for Gulf of Mexico Stations (b) for November 2008. 
65

(a) 

(b) 

Figure 6.11. ADCIRC Total Water Level Forecast Guidance Relative RMSE Error (-) 

for East Coast Stations (a) and for Gulf of Mexico Stations (b) for 

November 2008. 
66

(a) 

(b) 

Figure 6.12. ADCIRC Total Water Level Forecast Guidance Relative RMSE Error (-) for 

East Coast Stations (a) and for Gulf of Mexico Stations (b) for November 

2008. 
67

7. CONCLUSIONS AND RECOMMENDATIONS 

The previous water level forecast guidance evaluation procedures (Richardson and 

Schmalz, 2009) were streamlined by combining into a single analysis program for each 

ocean model, the read, reformat, filter, adjust, and analysis programs. The plot program 

was modified to be run within a script to further simplify the plotting process. Water level 

data acquisition was also simplified by using STEP 2 of the NOS skill assessment 

software (Zhang et al., 2006) to automatically download the water level data for all 

stations at once. 

For November 2008, daily 00Z forecast results of both subtidal (ETSS) and total water 

level response (RTOFS, G-NCOM, and ADCIRC) along the East coast and throughout 

the Gulf of Mexico from hours 6-36 for all models were evaluated using the new 

automated procedures. Results were compared to the previous results obtained by 

Richardson and Schmalz (2009) to verify the automated water level evaluation 

procedures. 

A monthly evaluation of the ETSS water level forecast guidance over the period 

December 2008 – October 2009 was performed. Next, a seasonal analysis of the RTOFS 

forecast guidance was performed by adding the analysis for the months February, May, 

and August 2009 to the November 2008 analysis. Finally, an annual analysis of ETSS, 

RTOFS, and G-NCOM was performed by comparing the November 2008 results to an 

analysis of results for November 2009. The evaluation procedures were extended to 

include the additional statistical measures of relative RMSE, Willmott et al. (1985) 

relative error, and central frequency (Hess et al., 2003) as illustrated for November 2008. 

While the ETSS model is subtidal only, the quality of the ETSS hourly subtidal water 

level forecasts serves as a benchmark for other total water level ocean forecast models to 

meet. To date, the quality of the total water level forecast models is not as good as the 

ETSS subtidal water level forecast model. For the total water level forecast models, the 

quality of the G-NCOM forecasts is substantially better than RTOFS. However, water 

levels are available only at a 3-hour data interval. The quality of the ADCIRC total water 

level hourly forecasts is substantially better than RTOFS and is improved relative to GNCOM at only the mid-Atlantic coast stations. It is anticipated that the horizontal 

resolution in both the weather prediction models and ocean models will continue to 

increase leading to improvements in the atmospheric forcings as well as in the 

representation of the hydrodynamics; all leading to improved water level forecast 

guidance. Therefore it is necessary to evaluate the ocean models as they continue to 

evolve. 

The automated analysis procedures developed here are sufficiently general to evaluate 

both subtidal and total water level forecast models. Additional comparisons of the total 

and subtidal water level forecasts to observations should be performed on a routine 

monthly basis using this automated software package. Specifically, it is recommended 
68

that the automated water level evaluation procedures be transferred to CO-OPS and run 

on a monthly basis with results posted on the CO-OPS operational forecast system 

website. 

ACKNOWLEDGMENTS 

This work has been carried out in the Coast Survey Development Laboratory (CSDL) of 

NOS’s Office of Coast Survey supervised by Mary Erickson, Director, under the 

direction of Dr. Frank Aikman, Chief of the Marine Modeling and Analysis Programs. 

REFERENCES 

Bleck, R., G. Halliwell, A. Wallcraft, S. Carroll, K. Kelley, and K. Rushing, 2002: 

HYCOM User’s Manual, Manual Version 2.0.01. 

Blumberg, A.F. and G.L. Mellor, 1987: A Description of a Three-dimensional Ocean 

Circulation Model, in Three Dimensional Coastal Ocean Models, N.S. Heaps 

(editor), American Geophysical Union, Washington, DC. 

Blumberg, A.F. and H.J. Herring, 1987: Circulation Modeling Using Orthogonal 

Curvilinear Coordinates, in Three-Dimensional Models of Marine and Estuarine 

Dynamics (J.C.J. Nihoul and B.M. Jamart, eds), Elsevier Oceanography Series, 45, 

55-88. 

Chen, J., W.A. Shafer, and S.C. Kim, 1993: A Forecast Model for Extratropical Storm 

Surge, Advances in Hydro-Science and Engineering, (ed.) Sam S.Y. Wang, 

Volume I Part B, University of Mississippi, 1437-1444. 

Edwards, C.R. and C.A. Blain, 2002: Operational Evaluation of ADCIRC-2DDI as 

Applied to the Western North Atlantic Ocean, NRL/FR/7320—02-10,005, Stennis 

Space Center, MS. 

Egbert, G.D. and S.Y. Erofeeva, 2002: Efficient Inverse Modeling of Barotropic Ocean 

Tides, Journal of Atmospheric and Oceanic Technology, 19(2), 183-204. 

Hess, K.W., T.F. Gross, R.A. Schmalz, J.G.W. Kelley, F. Aikman, E. Wei, and M.S. 

Vincent, 2003: NOS Standards for Evaluating Operational Nowcast and Forecast 

Hydrodynamic Model Systems, NOAA Technical Report NOS CS 17, Silver Spring, 

MD. 

Kolar, R.L, W.G. Gray, J.J. Westerink, and R.A. Luettich, 1994: Shallow Water 

Modeling in Spherical Coordinates: Equation Formulation, Numerical 

Implementation, and Application, Journal of Hydraulic Research, 32, 3-24. 
69

Luettich, R.A., J.J. Westerink, and N.W. Scheffner, 1992: ADCIRC: An Advanced 

Three-dimensional Circulation Model for Shelves, Coasts, and Estuaries, Report1: 

Theory and Methodology of ADCIRC-2DDI and ADCIRC-3DL, USACE Technical 

Report DRP-92-6, Vicksburg, MS. 

Martin, P.J., 2000: Description of the Navy Coastal Ocean Model Version 1.0, NRL 

Report FR/7322-00-9962, Stennis Space Center, MS. 

Richardson, P.H. and R.A. Schmalz, 2007: Comparison of NOAA Water Level Forecast 

Guidance from the Real Time Ocean Forecast System, Extratropical Storm Surge 

System, and the National Ocean Service Gulf of Mexico System: January-April 

2006 Monthly Analysis. NOAA, National Ocean Service, Coast Survey 

Development Laboratory, CSDL Informal Technical Note No. 8, Silver Spring, MD. 

Richardson, P.H. and R.A. Schmalz, 2009: Evaluation of National Weather Service and 

Extramural Water Level Forecast Guidance from Real Time Ocean Forecast 

Systems. NOAA, National Ocean Service, Coast Survey Development Laboratory, 

CSDL Technical Memorandum 22, Silver Spring, MD. 

Willmott, C. J., S. G. Ackleson, R. E. Davis, J. J. Feddema, K. M. Klink, D.R. Legates, J. 

O’Donnell, and C. M. Rowe, 1985: Statistics for the Evaluation and Comparison of 

Models, Journal of Geophysical Research, 90, 8995-9005. 

Zhang, A., K.W. Hess, E. Wei, and E. Myers, 2006: Implementation of Model Skill 

Assessment Software for Water Level and Current in Tidal Regions, NOAA 

Technical Report NOS CS 24, Silver Spring, MD.