SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\boldsymbol {\beta}} _ {g} \in \operatorname {a r g m i n} _ {\boldsymbol {\beta}} \sum_ {t \in \mathcal {T}} \left(y _ {t, g} - \boldsymbol {\beta} ^ {\top} \left[ 1, c _ {t, g}, y _ {t - l ^ {\star} - L - 1, g}, y _ {t - 2 l ^ {\star} - L - 1, g}, \bar {f} _ {t - l ^ {\star} - 1, g} \right]\right) ^ {2}
$$
output coefficients  $(\hat{\beta}_g)_{g = 1}^G$
Lagged temperature, 29 days
Lagged temperature, 58 days
CFSv2 ensemble
Figure 7: Spatial variation in Persistence++ learned regression weights when forecasting temperature in weeks 3-4 for the final target date, December 23, 2020.
Lagged temperature, 43 days
Lagged temperature, 86 days
Figure 8: Spatial variation in Persistence++ learned regression weights when forecasting temperature in weeks 5-6 for the final target date, December 23, 2020.
Lagged precipitation, 29 days
Lagged precipitation, 58 days
Figure 9: Spatial variation in Persistence++ learned regression weights when forecasting precipitation in weeks 3-4 for the final target date, December 23, 2020.
Lagged precipitation, 86 days
Figure 10: Spatial variation in Persistence++ learned regression weights when forecasting precipitation in weeks 5-6 for the final target date, December 23, 2020.
B.4 AutoKNN
The AutoKNN model of [23] was part of a winning solution in the Subseasonal Climate Forecast Rodeo I [46] and was shown to outperform deep fully connected neural networks [19]. AutoKNN first identifies a set of historical dates most similar to the target date and then forecasts a weighted locally linear combination of the anomalies measured on similar dates and recent dates. Our implementation matches that of [23] but adapts the model to target our primary RMSE objective by (i) using mean (negative) RMSE as the similarity measure instead of mean skill, (ii) using raw measurement vectors  $\mathbf{y}_t$  instead of anomaly vectors  $\mathbf{a}_t$ , and (iii) using equal datapoint weights in the final local linear regression.
Training The k-nearest neighbors (KNN) step of AutoKNN identifies a set of historical dates most similar to the target date and while the autoregression step forecasts a weighted locally linear combination of the anomalies measured on similar dates and recent dates. For a given target date  $t^{\star}$  and lead time  $l^{\star}$ , the AutoKNN training set is restricted to data fully observable one day prior to
the issuance date, that is, to dates  $t \leq t^{\star} - l^{\star} - L - 1$  where  $L = 14$  represents the forecast period length.
Tuning All hyperparameters were set to the default values specified in [23].
B.5 Informer
The Informer [79], a transformer-based deep learning model, was retrained every four months to predict temperature from past temperature and precipitation from past precipitation independently at each grid point.
Features For a given grid point and target date  $t^{\star}$ , the input features used to construct a forecast are the lagged target variable observations from dates  $t_{\mathrm{last}}$ ,  $t_{\mathrm{last}} - 1$ ,  $t_{\mathrm{last}} - 2$ , ...,  $t_{\mathrm{last}} - 95$  for where  $t_{\mathrm{last}} = t^{\star} - l^{\star} - L$  represents the last complete observation prior to  $t^{\star}$  and  $L = 14$  represents the forecast period length.
Training We divide the set of target dates in 2011-2020 into consecutive, non-overlapping four-month blocks and retrain the Informer model after every four-month block. For a given lead time  $l^{\star}$ , grid point, and four-month block beginning with date  $t^{\star}$ :
1. The training set is chosen to start at most 10,000 days before  $t$  (or at the beginning of the training set, whichever is later) and then ends 301 days before  $t$ .
2. The validation set is chosen to start 300 days before  $t$  and to end on the date prior to  $t^{\star} - l^{\star} - L$ .
3. We use early stopping with patience equal to three to determine when to stop the training: when we have three consecutive epochs  $e + 1, e + 2, e + 3$  with validation loss no lower than that of epoch  $e$ , we terminate training and use the model at epoch  $e$  as the final trained model.
4. We use the trained model to generate forecasts for each target date in the four-month block.
Tuning We use the default Informer architecture and hyperparameters for univariate time series forecasting: the model has 3 encoder layers, 2 decoder layers, and has an 8-headed attention with 7-dimensional keys and feed-forward layers with 1024 hidden units, and has GeLU activations [20].
B.6 LocalBoosting
In recent subseasonal experiments of [19], boosted decision tree models yielded the best performance. Our boosted decision tree model, based on CatBoost [51], uses as features the value of 10 SubseasonalClimateUSA variables in a geographic region around the target grid point. This gives the algorithm enough flexibility to adapt the weights of the features to each particular grid point while still taking into account neighboring spatial information. The geographic region is determined by a bounding box of 2 degrees in each direction, and the 10 variables are chosen for each task via their predictive power on validation years.
Training For a given target date  $t^\star$  and lead time  $l^\star$ , the LocalBoosting training set  $\mathcal{T}$  is restricted to data fully observable one day prior to the issuance date, that is, to dates  $t \leq t^\star - l^\star - L - 1$  where  $L = 14$  represents the forecast period length. LocalBoosting uses CatBoost [51] to regress, for each gridpoint and each date, the value of a set of lagged weather variables in a geographic region around the gridpoint.
Tuning There are two hyperparameters to consider: (i) which lagged weather variables to use; and (ii) the number of neighborhood cells around a gridpoint to define the geographic region. Bounding boxes of width, length of 2 or 3 cells were considered. Larger sizes were computationally infeasible. In each case, the 10 or 20 most important features in the SubseasonalClimateUSA dataset were considered. Here, features were chosen by their performance over 2001-2010 in terms of RMSE.
For each target date, LocalBoosting is run with the hyperparameter configuration that achieved the smallest mean RMSE over the preceding 3 years. See Figure 11 for a visualization of the hyperparameters automatically selected for each target date in 2011-2020.
Figure 11: LocalBoosting hyperparameters automatically selected for each target date in 2011-2020.
B.7 MultiLLR
The MultiLLR model of [23] was also part of a winning solution in the Subseasonal Climate Forecast Rodeo I [46] and has since been used to improve subseasonal precipitation prediction in China [67]. For each target date, MultiLLR uses a backward stepwise procedure to select the most predictive features and then applies local linear regression for each grid point to combine those features into a final prediction. Our implementation matches that of [23] but adapts the model to target our primary RMSE objective by using mean (negative) RMSE instead of mean skill as the feature selection criterion. In addition, we replace their MEI features with corresponding MEI.v2 features and their monthly dynamical forecast features with daily debiased CFSv2 forecasts.
Training For a given target date  $t^{\star}$  and lead time  $l^{\star}$ , the MultiLLR training set is restricted to data fully observable one day prior to the issuance date, that is, to dates  $t \leq t^{\star} - l^{\star} - L - 1$  where  $L = 14$  represents the forecast period length. The coarse-grained dynamical input feature nmme_wo_ccsm3_nasa of [23] was replaced with the daily debiased CFSv2 forecast features
- subx_cfsv2_tmp2m-14.5d_shift15 and subx_cfsv2_tmp2m-0.5d_shift15 for predicting temperature at weeks 3-4,
- subx_cfsv2_tmp2m-28.5d_shift29 and subx_cfsv2_tmp2m-0.5d_shift29 for predicting temperature at weeks 5-6,
- subx_cfsv2_precip-14.5dshift15 and subx_cfsv2_precip-0.5dshift15 for predicting precipitation at weeks 3-4, and
- subx_cfsv2_precip-28.5d_shift29 and subx_cfsv2_precip-0.5d_shift29 for predicting precipitation at weeks 5-6.
Tuning All hyperparameters were set to the default values specified in [23], save for the tolerance parameter which was set to 0.001 to accommodate the new RMSE selection criterion.
B.8 N-BEATS
N-BEATS [48], a neural network for time series data, is retrained N-BEATS every two months to predict temperature from past temperature and precipitation from past precipitation independently at each grid point.
Features For a given grid point and target date  $t^{\star}$ , the input features used to construct a forecast are the lagged target variable observations from dates  $t_{last}, t_{last} - 4, t_{last} - 8, \dots, t_{last} - 48$  for where  $t_{last} = t^{\star} - l^{\star} - L$  represents the last complete observation prior to  $t^{\star}$  and  $L = 14$  represents the forecast period length.
Training We divide the set of target dates in 2011-2020 into consecutive, non-overlapping two-month blocks and retrain the N-BEATS model after every two-month block. For a given lead time  $l^{\star}$ , grid point, and two-month block beginning with date  $t^{\star}$ :
1. We train on all dates  $t \leq t^{\star} - l^{\star} - L$ .
2. For the initial two-month block, we train for 30 epochs.
3. For subsequent two-month blocks, we initialize our model weights to the learned weights from the prior block and then fine-tune for 8 epochs.
4. We use the trained model to generate forecasts for each target date in the two-month block.
Tuning We use the default N-BEATS architecture and hyperparameters for univariate time series forecasting [48]. The N-BEATS model has two stacks, where each stack is used to understand different patterns. Each stack consists of three blocks, which themselves are each comprised of six fully connected layers with ReLU activations. We used the Adam optimizer with learning rate 0.001 and a batch size of 512.
B.9 Prophet
The Prophet model of [62] was one of the winning solutions in the Subseasonal Forecast Rodeo II [46]. Prophet is an additive regression model for time-series forecasting that predicts weekly and yearly seasonal trends on top of a piecewise linear or logistic growth curve. We trained the model to predict each grid point independently with yearly seasonality enabled (to capture predictable whether trends) and weekly seasonality disabled.
Training Prophet takes as input a sequence of (univariate) time-series values and then predicts the next  $k$  dates from those, arbitrarily far in the future.
First, we split the problem into a many univariate time-series prediction problems. When evaluating, we consider periods of 4 moths (e.g. January 2010 - April 2010), train the model on all available historical temperatures (which may depend on the lead time) at that grid point, and make predictions for that four month period. We then run this for all relevant periods of four months.
Tuning The prophet model is trained in the univariate mode with yearly seasonality on (to capture predictable weather trends), weekly seasonality off (as weekends are unlikely to be special).
B.10 Salient 2.0
We developed the Salient 2.0 model based on Salient [59], a winning solution for the Subseasonal Forecast Rodeo I [46]. Salient consists of an ensemble of feed-forward fully-connected neural networks, using historical sea surface temperature (SST) data from 1990 to 2017 and an encoding of the day of the year as features. Salient's training protocol follows a multi-task learning framework [56]. It starts by training 50 randomly generated fully connected neural networks, each of which provides a prediction for the average temperature and accumulated precipitation at 3, 4, 5, and 6 weeks ahead, at every grid cell. The forecasts are then obtained by combining the predictions for weeks 3 and 4 and for weeks 5 and 6. The final ensemble model forecasts correspond to the mean of the top 10 ensemble members with the lowest validation error.
For Salient 2.0 in this work, the input features were augmented with geopotential heights at different pressure levels (10, 100, 500 and  $850\mathrm{hPa}$ ) along with MEI and MJO indices. In addition, instead of training the ensemble on the whole of the 1990-2017 data, a sequence of models was trained using data up until each of the years in our validation period of 2010-2020. These submodels with earlier training data cut-offs were then used to generate hindcasts that informed the model's tuning decisions.
Salient 2.0 relies on two sources of sea surface temperature training data. The first data source is the weekly sea surface temperature from NOAA [54] and covers dates from 17 January 1990 to 02 February 2017. This dataset contains weekly data cenetered around Wednesdays and has a  $1^{\circ} \times 1^{\circ}$  resolution. It was re-gridded to a  $4^{\circ} \times 4^{\circ}$  using spline interpolation of order equal to 2, under the Python Scipy package [63].
For dates from February 2017 to present, a second source of sea surface temperature data from the MET Office [30] is used. This dataset has a daily temporal resolution and is averaged to obtain weekly data centered around Wednesdays. This data is initially downloaded in a  $0.25^{\circ} \times 0.25^{\circ}$  spatial resolution and is re-gridded to a  $4^{\circ} \times 4^{\circ}$ . A linear interpolation is then used to fill in any missing values (under the Python Scipy package [63]).
Training Salient 2.0 is an ensemble of 50 feed-forward fully connected neural networks. All 50 neural networks are trained on a pre-determined combination of input features, including weekly sea surface temperature, MEI, as well as the phase and amplitude features of MJO. The combinations of input features considered are:
- d2wk_cop_sst: sea surface temperature,
d2wk_cop_sst_mei: sea surface temperature and ENSO,
d2wk_cop_sst_mjo: sea surface temperature and MJO,
- d2wk_cop_sst_mei_mjo: sea surface temperature, ENSO and MJO.
In total, four ensemble models, corresponding to the four input feature combinations above, each including 50 neural networks, are trained. Each ensemble model is trained in a rolling fashion, where the start year of the training dataset is 1990 and the ensemble is trained up until each year in the range [2006, 2019], where a model ending on a year  $y$  is used to generate forecasts for target dates with year  $y + 1$ .
For each of the 50 neural networks within a given ensemble model, the input features can further be augmented using a time vector obtained by converting dates to a float representing the fraction of the year passed by that date. The addition of the time vector is decided by generating a random integer in the range  $[0,1]$ , with 1 corresponding to the addition of the time vector and 0 otherwise. Additionally, for each of the 50 neural networks, the input feature vector for an individual training example consists of a concatenation of the prior 10 weeks of data.
For each of the 50 neural networks within a given ensemble model, the output consists of a prediction for the average temperature and accumulated precipitation at 3, 4, 5 and 6 weeks ahead. The predictions for weeks 3 and 4 and the predictions for weeks 5 and 6 are combined separately, by averaging temperatures and summing precipitation. The top 10 neural networks with the lowest validation error are selected as the final ensemble members. The final predictions for each ensemble model correspond to the mean of the 10 selected ensemble members.
Tuning Each of the 50 neural networks within a given ensemble model is trained using a batch size equal to 128 and a train ratio equal to 0.89. In addition, each neural network's hyperparameters are randomly generated, with the number of epochs sampled in the range [100, 500], the number of layers in the range [3, 7], and the number of units per layer in the range [100, 600].
At test time, for each target date, Salient 2.0 is run using the ensemble model that achieved the smallest mean RMSE over the preceding 3 years. Figure 12 shows which ensemble models were used to generate predictions for which target dates.
Figure 12: Salient 2.0 hyperparameters automatically selected for each target date in 2011-2020.
B.11 Uniform Ensemble
We consider two Uniform Ensemble models, where the ensemble is produced using a set of models  $C$  as input:
1. Uniform ABC:  $C = \{\mathrm{Climatology} + + ,\mathrm{CFSv2} + + ,\mathrm{Persistence} + + \}$
2. Uniform ABC + Learning:  $C = \{ \text{Climatology}++, \text{CFSv2}++, \text{Persistence}++, \text{LocalBoosting}, \text{MultiLLR}, \text{AutoKNN}, \text{Prophet}, \text{Salient} 2.0 \}$ , the ABC models plus all learning models save the very low performing Informer and N-BEATS models.
Uniform ensemble forecasts are produced as the uniform or unweighted average of input models. Letting  $X_{t,c}$  be the forecast made by model  $c \in C$  on target date  $t$ , we produce the forecast:  $\hat{\mathbf{y}}_t = \frac{1}{|C|}\sum_{c \in C} X_{t,c}$ .
B.12 Online Ensemble
We consider two Online Ensemble models, where the ensemble is produced using a set of models  $C$  as input:
1. Online ABC:  $C = \{\text{Climatology}++, \text{CFSv2}++, \text{Persistence}++\}$
2. Online ABC + Learning:  $C = \{ \text{Climatology}++, \text{CFSv}2++, \text{Persistence}++, \text{LocalBoosting}, \text{MultiLLR}, \text{AutoKNN}, \text{Prophet}, \text{Salient}2.0 \}$ , the ABC models plus all learning models save the very low performing Informer and N-BEATS models.
To learn a time-dependent adaptive ensemble weight  $\mathbf{w}_t$ , we employ the online learning method presented in [15]. We applied the AdaHedgeD algorithm with the recommended recent_g optimism setting. The algorithm was run with a delay parameter of  $D = 2$  for the 3-4 weeks horizon tasks and  $D = 3$  for the 5-6 weeks horizon tasks. We ran the online learning algorithm over the full set of target dates  $T = 520$ , without performing the yearly resetting suggested in the original implementation. The learner optimized the RMSE loss over gridpoints in the region of interest, as described in the experimental details of [15].
Online ensemble forecasts are produced as the weighted average of input models, with weight  $\mathbf{w}_t$  determined by the online learning algorithm. Letting  $X_{t,c}$  be the forecast made by model  $c\in C$  on target date  $t$  and  $\mathbf{w}_t$  be the weights produced by AdaHedgeD, we produce the forecast:  $\hat{\mathbf{y}}_t = \sum_{c\in C}\mathbf{w}_{t,c}*X_{t,c}$ .
B.13 Debiased ECMWF
We implement the operational ECMWF bias correction protocol detailed in [69]. For each target forecast date, we debias both our ECMWF control and ensemble forecasts using the last 20 years of reforecasts with dates within  $\pm 6$  days from the target forecast date. The average of the 1 control and 10 ensemble reforecasts on the  $1.5 \times 1.5$  degree grid are used for debiasing.
C.1 Percentage Improvement over Meteorological Baselines
To highlight the improvement of individual ABC models over their traditional counterparts, Figures 13 and 14 show the percentage RMSE improvements of Climatology++, CFSv2++, and Persistence++ relative to their respective baselines Climatology, debiased CFSv2, and Persistence by season and by year.
For all four tasks, the ABC models are consistently better across seasons and years. The result is particularly striking for Persistence++ and highlights the value of integrating lagged measurements, numerical weather prediction, and climatology. Figures 13 and 14 show the per season and per year improvement of each ABC model over its corresponding baseline across the contiguous U.S. and the years 2011-2020. Note the learned ABC benchmarks yield consistent improvements in mean RMSE.
C.2 Yearly Percentage Improvement over Mean Debiased CFSv2 RMSE
Tables 3 and 4 present the yearly improvement of each model over debiased CFSv2, as measured by mean RMSE across the contiguous U.S. in the years 2011-2020.
Figure 13: Per season improvement of each ABC model over its corresponding baseline across the contiguous U.S. and the years 2011-2020. The learned ABC benchmarks yield consistent improvements in mean RMSE.
Figure 14: Per year improvement of each ABC model over its corresponding baseline across the contiguous U.S. and the years 2011-2020. The learned ABC benchmarks yield consistent improvements in mean RMSE.
C.3 Yearly Average Skill
Tables 5 and 6 present the yearly average skill of each model across the contiguous U.S. in the years 2011-2020.
Table 5: Average percentage skill when forecasting temperature in the contiguous U.S. The best performing models within each group are shown in bold, while the best performing models overall are shown in green.
C.4 Spatial Improvement over Mean Debiased CFSv2 RMSE
Figure 15 presents the spatial improvement of each model over debiased CFSv2 when predicting U.S. temperature across 2011-2020. For both the weeks 3-4 and the weeks 5-6 lead times, the ABC models uniformly improve over debiased CFSv2 and outperform both the baseline models and the state-of-the-art learning methods. The best overall performance at each lead time is obtained by the Online ABC ensemble.
Figure 15: Percentage improvement over mean debiased CFSv2 RMSE when forecasting temperature in the contiguous U.S. over 2011-2020. White grid points indicate negative or  $0\%$  improvement, and the mean percentage improvement is given in parentheses.
Figure 16 displays the spatial improvement of each model over debiased CFSv2 when forecasting U.S. precipitation across 2011-2020. For precipitation, all models exhibit larger gains over debiased CFSv2, and all models, including Climatology, achieve larger improvements in the Western U.S. than in the Eastern U.S.
Figure 16: Percentage improvement over mean debiased CFSv2 RMSE when forecasting precipitation in the contiguous U.S. over 2011-2020. White grid points indicate negative or  $0\%$  improvement, and the mean percentage improvement is given in parentheses.
C.5 Spatial Bias Maps
This section explores the spatial bias (the mean forecast minus the mean observation at each grid point in the contiguous U.S.) of each model across 2011-2020. The temperature maps of Figure 17 indicate a cold bias for most models over the southern half of the U.S. and an additional warm bias for several models in the center north. This warm bias is particularly pronounced for Salient 2.0. In precipitation maps of Figure 18, all models Salient 2.0 and AutoKNN show wet biases in the western half of the U.S. and dry biases in the eastern half. AutoKNN exhibits a dry bias extending from the Eastern U.S. to include the Northern U.S. as well, while Salient 2.0 displays a strong dry bias across the entire contiguous U.S. that is especially pronounced in the eastern half.
The Prophet model is noticeably less biased than the other evaluated models; however, this bias reduction does not immediately translate into improved performance, as Prophet is outperformed by ABC models with larger bias in all four tasks. This indicates that Prophet's reduced bias comes at a cost of unnecessarily high variance relative to the dominating ABC model forecasts.
Figure 17: Model bias when forecasting temperature in the contiguous U.S. over 2011-2020.
Figure 18: Model bias when forecasting precipitation in the contiguous U.S. over 2011-2020.
C.6 GraphCast Comparison Details
To generate a weeks 3-4 GraphCast forecast of temperature and precipitation for a given target date, we
1. downloaded the pretrained GraphCast - ERA5 1979-2017 - resolution 0.25 - pressure levels 37 - mesh 2to6 - precipitation input and output model weights from https://console.cloud.google.com/storage/browser/dm_graphcast,
2. initialized the model with the three days of ECMWF Reanalysis v5 (ERA5) data preceding the associated issuance date, as described in https://github.com/google-deepmind/graphcast,
3. repeatedly ran the model (using autoregressive rollout as in https://github.com/google-deepmind/graphcast/blob/main/graphcast_demo.ipynb) for the maximum supported number of time steps (12) and reinitialized the model with its own forecasted output until 30 days of forecasts had been produced,