SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4. aggregated the 2m_temperature and total_precipitation_6hr forecasts over days 14-28,
5. regridded the aggregated forecasts to our standard  $1 \times 1^{\circ}$  grid using xarray interp_like [22], and
6. converted temperature to degrees Celsius and precipitation to millimeters.
Since GraphCast was trained on data through 2017, Table 7 summarizes the performance of GraphCast across the post-training years 2018-2020. Similarly to the deep learning models previously evaluated in Table 1, GraphCast outperforms debiased CFSv2 in terms of skill, underperforms debiased CFSv2 in terms of RMSE, and strongly underperforms the ABC ensemble models in both metrics when forecasting either temperature or precipitation in weeks 3-4.
Table 7: Average percentage skill and percentage improvement over mean debiased CFSv2 RMSE across 2018-2020 in the contiguous U.S. along with a  $95\%$  bootstrap confidence interval. The best performing model overall is shown in green.
GROUP	MODEL	RMSE % IMPROVEMENT	AVERAGE % SKILL
WEEKS 3-4
TEMPERATURE	PRECIPITATION	TEMPERATURE	PRECIPITATION
BASELINES	DEB. CFSV2	-	-	19.98±4.84	3.12±2.99
LEARNING	GRAPHCAST	-34.96±7.46	-34.26±7.57	20.17±4.70	5.55±2.81
ENSEMBLES	UNIFORM ABC	9.70±2.19	8.41±1.18	34.37±4.58	18.34±2.59
ONLINE ABC	9.37±2.07	8.37±1.25	34.67±4.71	18.07±2.76
U.S. Temperature, weeks 3-4
Figure 19 displays the spatial improvement of GraphCast over debiased CFSv2 when forecasting U.S. temperature and precipitation across 2018-2020. Unlike the ensemble ABC models, GraphCast seldom improves over the debiased CFSv2 baseline.
U.S. Precipitation, weeks 3-4
Figure 19: Percentage improvement over mean debiased CFSv2 RMSE when forecasting temperature or precipitation in the contiguous U.S. over 2018-2020. White grid points indicate negative or  $0\%$  improvement, and the mean percentage improvement is given in parentheses.
Figure 20 compares the spatial bias (the mean forecast minus the mean observation at each grid point in the contiguous U.S.) of GraphCast versus the ABC ensemble models across 2018-2020. For
precipitation, GraphCast has a substantially drier bias than the ABC models throughout most of the contiguous U.S. For temperature, GraphCast has a warmer bias in the West.
Figure 20: Bias of GraphCast vs. ABC ensembles when forecasting temperature or precipitation in the contiguous U.S. over 2018-2020.
C.7 Western U.S. Competition Results
Table 8 displays the performance improvement over the baseline debiased CFSv2 in the Western U.S., over 26 dates in 2019-2020. This region was the focus of the Subseasonal Forecast Rodeo I, and we include the performance of the three best contestants in the table. The ABC ensembles were either very competitive or better than the other contestants.
Table 8: Percentage improvement over mean debiased CFSv2 RMSE over 26 contest dates (2019-2020) in the Western U.S. The best performing models within each class of models are shown in bold, while the best performing models overall are shown in green.
Group	Model	Temp. weeks 3-4	Temp. weeks 5-6	Precip. weeks 3-4	Precip. weeks 5-6
Contest baselines	Salient	-	-	11.10	7.02
Climatology	10.22	-0.76	5.82	2.25
Contestants	\(1^{st}\) place	17.12	8.47	11.54	8.63
\(2^{nd}\) place	16.67	7.04	11.10	8.03
\(3^{rd}\) place	15.47	6.90	10.62	7.94
Learning	AutoKNN	13.09	2.90	7.50	3.05
LocalBoosting	12.85	4.09	7.25	3.71
MultiLLR	9.54	1.12	8.95	4.58
Prophet	15.68	6.86	6.88	3.40
Salient 2.0	11.15	2.91	12.65	8.56
ABC	Climatology++	15.54	6.43	8.35	4.69
CFSv2++	6.67	9.26	8.70	5.51
Persistence++	16.59	8.27	8.20	4.51
Ensembles	Uniform ABC	14.96	9.58	9.31	5.89
Uniform ABC + Learning	15.89	8.79	10.43	6.79
Online ABC	16.71	8.70	8.85	5.19
Online ABC + Learning	14.70	7.97	12.52	8.18
C.8 Salient 2.0 Dry Bias
We hypothesized that the exceptional Western U.S. contest performance of Salient 2.0 in Table 8 was due in part to the dry bias observed in Figure 18, as 2020 was an unusually dry year in the Western U.S. To explore this hypothesis, we focus on forecasting precipitation weeks 3-4 in the Western U.S. and display in Figure 21 (left) the percentage improvement of Salient 2.0 and CFSv2++ over debiased CFSv2 alongside the inverse total precipitation each year in 2011-2020. As anticipated, the steep decrease in cumulative precipitation for 2020 is accompanied by a steep increase in Salient 2.0 predictive accuracy. In addition, the rises and falls in total precipitation track the accuracy of Salient 2.0 well but appear largely unassociated with the performance curve of other accurate models like CFSv2++. The scatter plots and best-fit lines of Figure 21 (right) paint a similar picture. Salient 2.0 exhibits a distinctly negative correlation between percentage improvement and total precipitation in the Western U.S., while this relationship is absent for other accurate models like CFSv2++.
Figure 21: Temporal plot (left) and scatter plot (right) of yearly total precipitation and percentage improvement over mean debiased CFSv2 RMSE in the Western U.S. across 2011-2020.
The developers of the Rodeo I Salient model attribute this dry bias to the log-normal distribution of precipitation, which leads to more frequent anomalously dry conditions than anomalously wet conditions and in turn encourages drier model forecasts [58].
More recent versions of the Salient model mitigate this bias by training on seasonal anomalies in place of raw temperature and precipitation values [58].