Forecast Verification Using Citizen-Science Rain Gauges
By: Mike Dvorak
Tags: Python, Weather Forecasts, Rain, Jupyter Notebook, California

The complete details of the analysis can be found in this Jupyter Notebook. The basic procedure was this:

Load up a bunch of GFS forecasts into a WinDB2 Postgres database, an open source geospatial database project I started to store weather and other environmental forecasts. Download all of the CoCoRaHS observations from California from February 1–8, 2019. Ignore all observations that fall outside a +/- 2 hour time window from 7 am local time. Find the GFS grid point closest to the CoCoRaHS precipitation observation. Calculate the error for all forecasts made for a particular CoCoRaHS gauge location and time. Because the GFS forecast is hourly (i.e. every hour) for 0–120 hours, 3-hourly for 120–240 hours, and 12-hourly from 240–384 hours, we use Postgresql database to do some fancy aggregation to make the math easy for us, explained below.

Elegant error calculation using SQL

Due to the varying forecast intervals talked about in #4 above, calculating which forecast corresponds to which 24-hour precipitation forecast period could be pretty challenging. But since we have everything loaded into Postgres, it was relatively simple to account for this mismatch in forecast interval. The SQL statement from the Jupyter Notebook is shown below. obs represents the observation, fcst is the forecast, and in stands for inches. prate is the GFS forecast variable for total precipitation rate.

sql = """SELECT %s AS station_id,

COUNT(*),

AVG(value)*3600*24*%s AS fcst_rain_in_per_day,

AVG(EXTRACT(epoch FROM t - init)/3600) AS fcst_age_hr_avg

FROM prate_1

WHERE geomkey=%s AND t>=%s AND t<%s

GROUP BY init

ORDER BY fcst_age_hr_avg;""" windb2.curs.execute(sql,

(obs['st_num'],

conv_prate_to_inch_per_sec.magnitude,

station_meta['closest_geomkey'],

obs_t_nominal,

obs_t_nominal + timedelta(days=1)))

If we assume that the forecast amount of rain for the 24-hour CoCoRaHS observation period is simply the average of all forecasts that fall within that period, the calculation is relatively simple. For forecasts 0–120 hours out, you’re always going to have 24-hourly forecasts per day, so the forecast is simply the average precipitation rate for each hour times 24 hours.

Admittedly, there are some small edge-case errors that are not accounted for here. For example, if your 24-hour forecast includes hours 115 thru 128, the first six forecasts will be hourly and the last eight hours will be three hourly. So technically the forecast hours should be weighted accordingly. But in the interest of simplicity, we ignore this small error in weighting and as we’ll see below, this doesn’t present a significant issue with the results.

Do forecasts improve with time?

As explained above, generating the precipitation forecast error was relatively easy using the WinDB2 schema and utilities. From the Jupyter notebook using Pandas and Matplotlib to plot histograms of the forecasts and observations (below image), we can immediately see that the GFS forecasts severely under-predict the forecast rain for the study period. Whereas the highest forecast is somewhere around 0.5 inches of rain per day (leftmost plot below), actual observations are as high 1.5 inches per day (rightmost plot below).

In my experience, the relatively low resolution GFS forecasts under-forecasting rain really isn’t surprising. Much of the heavy rain in California is generate by orographic effects, i.e. mountains pushing up moist air and generating more rain. At 0.25 degrees horizontal resolution (i.e. about 24x24 km² per model grid cell), you’re not really resolving the sharp coastal and inland mountain ranges that are rain makers.

Distribution of CoCoRaHS precipitation forecasts (left), age of forecast in hours (middle), and precipitation observations (right)for the 1st week of 2019 in California.

What is surprising however is that my hypothesis, at least in this case study, proved to be wrong. In fact, precipitation forecast error got worse with time, as can be seen by the blue line below, which represents the root mean square error between the precipitation forecast and the CoCoRaHS observation. Pay careful attention of the x-axis in the plot. The newest forecasts (i.e. the youngest forecast ages) are on the the left-hand side.

Precipitation forecast error and descriptive statistics versus the forecast age in hours (all y-axis variables are inches per hour). RMSE is root mean square errors.

So why didn’t the forecast get more accurate over time? I’m not sure at this point if I have a good answer to that question. This study did however encourage me to create an automated forecast verification website called Weather Verify. You can check the Weather Verify MVP app out here.