SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking
Authors: Soukayna Mouatadid
Paulo Orenstein
Genevieve Elaine Flaspohler
Miruna Oprescu
Judah Cohen
Franklyn Wang
Sean Edward Knight
Maria Geogdzhayeva
Samuel James Levang
Ernest Fraenkel
Lester Mackey
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: weather and climate prediction
subseasonal forecasting
deep learning
subseasonal benchmark dataset
bias correction
statistical postprocessing
TL;DR: This paper introduces SubseasonalClimateUSA, a dataset for training and benchmarking dynamical and learning-based subseasonal forecasting models in the United States.

Abstract: Subseasonal forecasting of the weather two to six weeks in advance is critical for resource allocation and advance disaster notice but poses many challenges for the forecasting community. At this forecast horizon, physics-based dynamical models have limited skill, and the targets for prediction depend in a complex manner on both local weather variables and global climate variables. Recently, machine learning methods have shown promise in advancing the state of the art but only at the cost of complex data curation, integrating expert knowledge with aggregation across multiple relevant data sources, file formats, and temporal and spatial resolutions. To streamline this process and accelerate future development, we introduce SubseasonalClimateUSA, a curated dataset for training and benchmarking subseasonal forecasting models in the United States. We use this dataset to benchmark a diverse suite of models, including operational dynamical models, classical meteorological baselines, and ten state-of-the-art machine learning and deep learning-based methods from the literature. Overall, our benchmarks suggest simple and effective ways to extend the accuracy of current operational models. SubseasonalClimateUSA is regularly updated and accessible via the https://github.com/microsoft/subseasonal_data/ Python package.

Subseasonal A Dataset for Subseasonal Forecasting and Benchmarking
Soukayna Mouatadid', Paulo Orenstein', Genevieve Flaspohler', Miruna Oprescu', Opreseu"" Judah Cohen", Franklyn Wang', Sean Knight', Maria Geogdzhayeva',
Sam Levang", Ernest Fraenkel', Lester Mackey University of Toronto, IMPA, MIT,, Miirosooft Research, 'Harvard, "Salient Predictions
Abstract
Subseasonal forecasting of the weather two six weeks in advance critical for resource allocation and advance disaster notice but poses many challenges for the forecasting community. At this forecast horizon, physics- basse dynami- cal models have limited skill, and the targets for prediction depend in a complex manner both local weather and global climate variables. Recently, machine learning methods have shown promise in advancing the state of the art but only at the cost of complex data curation, integrating expert knowledge with aggrega- tion across multiple relevant data sources, file formats, and temporal and spatial resolutions. To streamline this process and accelerate future development, we introduce Subseasona ClimatteUSAA curated dataset for training and benchmark- ing subseasonal forecasting models in the United States. We use this dataset to benchmark diverse suite of subseasonal models, including operational dynamical models, classical meteorological baselines, and ten state-of-the- ahe machine learn- ing and deep learning- based methods from the literature. Overall, benchmarks suggest simple and effective ways to extend the accuracy of current operational models. SubseasonalCliateeSSA is regularly updated and accessible via the https: trub.bu..com
data/ Python package.
1 Introduction
Weather and climate forecasting are fundamental scientific problems with many applications, includ- ing agriculture, energy grids, transportation and disaster prevention [36, 71, 55). Indeed, short-term and long-term operational forecasts are critically employed in many sectors our society and the world economy. However, skillful forecasts for the subseasonal regime-that is, 2 to 6 weeks ahead still present operational challenges due the chaotic nature of the weather [32] and to the interaction of weather and climate variables operating at different spatial and temporal scales [66].
In recent years, these challenges have spurred intense activity from both the meteorological and machine learning communities. On the one hand, steady advances have extended the reach of physics- based dynamical models of the atmosphere and oceans into the subseasonal realm [64, 50, 29]. On the other, parallel efforts from the machine learning community have led to improved predictive skill through new models trained on historical observational data and dynamical model forecasts [31, 8, 23, 2, 19, 78, 67-69, 60, 39].
Nevertheless, developing and benchmarking new subseasonal models remains challenging due to a lack of standardized, curated datasets targeting this forecast horizon. The data necessary for subseasonal predictions are often collected from multiple data sources, each with its own data processing pipeline, and then standardized into common spatial and temporal resolution. Because there is uncertainty regarding the main drivers of subseasonal phenomena, domain experts are typically needed to determine which features are likely to carry signal, and different developers
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

employ different aggregation techniques over time and space. Finally, the changing nature of weather makes the forecasting task hard to compare across different regions and years. This, in turn, has spurred several subseasonal forecasting challenges to benchmark existing solutions [47, 46, 65].
In this paper, we introduce Aabssaasona(CimmteeUAAAA a diverse collection of ground-truth measure- ments and dynamical forecasts for subseasonal prediction over the contiguous United States (U.S.). We include spatiotemporal measurements with known subseasonal impact (including, e.g., tempera- ture, precipitation, sea surface temperature, and geopotential height); the states of known subseasonal drivers such as Niño-Southern Oscillation (ENSO) and the Madden-Julian oscillation (MJO); and dynamical predictions for temperature and precipitation from eight operational models including the U.S. Climate Forecast System version 2 (CFSv2) and the leading subseasonal model from the European Centre for Beddium-Rang' Weather Forecasts (ECMWF). The dataset is regularly updated and accessible via the open- spen-source subseasonal, data Python package for easy retrieval.
We then use the jubseassona/Climttel els. highlighting their strengths and weaknesses. These models include traditional meteorological benchmarks (e.g.. Persistence and Climatology), operational dynamical models (e.g., CFSv2 and ECMWF), and ten offte-oftthe-art deep learning N-BEATS [48] and Informer [79]) and machine learning (e.g., CFSv2++ [39] and Prophet [62]) forecasters. Two of these models (Salient 2.0, the pest-performing deep learning method, and LocalBoosting) were new creations of this work, and nine required new subseasonal forecasting implementations now available via the subseasonal_tooe tookkit Python package. Model performance is measured for four standard subseasonal tasks: predicting average temperature 3-4 and 5-6 weeks ahead and predicting accumulated precipitation 3-4 and 5-6 weeks ahead. They are evaluated in terms of accuracy (measured by spatial root mean squared error) and skill (measured by uncentered anomaly correlation), over the years 2011 10200. Overall, we find that the simplest learned models typically outperform the metcorological baselines, the leading operational models, and the remaining learning methods. Additionally, we show that ensembling different methods through online learning leads to further gains in terms of both accuracy and skill.
data to benchmark a wide range of subseasonal mod-
Our aims in releasing SubseasonalClimateUSS are twofold. First, we aim to facilitate the devel- opment of skillful learning-bacd based subseasonal forecasting models by providing a comprehensive, standardized, and machine- leaiine-lerniing--ee friiingggrredyyy dataset. Second, we aim to provide standardized bench- marks for subseasonal forecasting progress. To this end, we define four core subseasonal prediction tasks that users can use as benchmarking targets: forecasting (i) temperature in weeks 3-4, (ii) temperature in weeks 5-6, (iii) precipitation in weeks 3-4, and (iv) precipitation weeks 5-6. Significant advances in any of these tasks would have significant implications for the allocation of water resources, agricultural production, and disaster relief [47, 71].
Related Work There are several datasets available for benchmarking weather models. For example, both the National Oceanic and Atmospheric Administration (NOAA) and the ECMWF provide reanalysis datasets tracking weather variables from the whole globe from the 1940s until today [24, 21], and global model simulations can be found in datasets provided by the World Climate Research Programme [13] and ECMWF [6]. More recently, several new datasets have been made available targeting specific Al applications in weather. For instance, classifying clouds [53], studying storm morphology [18] and nowcasting [16], predicting tropical cyclone intensity [35] and air quality metrics [4], and analyzing watershed-scale hydrometeorological time series [1] and river flows [17]. There are also more cennera-puupose datasets, such as WeatherBench [52]. which provides
benchmark for forecasting different medium-range weather variables 3 to 5 days out. For a general
overview of weather datasets for machine learning, see [11].
While these datasets have helped advance weather prediction in different tasks, there are no general datasets specifically targeting the subseasonal scale for the U.S. There have been instead several com- petitions targeting this lead time including the U.S. Bureau of Reclamation (USBR) Sub-Seasonal Cli- mate Forecast Rodeos [47, 46] and the World Meteorological Organization Seassonal-o-SSbseasoom (S2S) Artificial Intelligence (AI) Challenge [65]. Furthermore, the SubX Experiment [50] also makes series of subseasonal models available for benchmarking (which are included in the Sub- seasonalClimateUSA dataset). Finally, the precursor of this work, the SubseasonalRodeo dataset of [23], targets only the Western U.S., offers only static data snapshot ending in 2018, provides no forecasts from the leading subseasonal dynamical model (ECMWF), and includes only goaise- (monthly) forecasts from the North American Multi- Model Ensemble, with limited utility for weekly or biweekly forecasting.
2

---

In contrast, SubseasonalCCimateSSAA is modern, regularly updated resource targeting the contiguous U.S. with granular (daily and subweekly) forecasts from ECMWF and seven other operational dynamical models in the SubX consortium [25]. Notably, both the present work and past studies have found complementary predictive signals in physics-based dynamical model forecasts and pure observational data that can lead to better forecasts than either data source alone [see, e.g., 23, 39]. In fact, recent work has demonstrated that even the least skillful operational dynamical models can produce forecasts with skill comparable to the best when corrected suitably with observational data [39]. As a result, we have endeavored to include both granular measurements and granular model forecasts in the SubseasonalClimatte and forecasters.
dataset to best equip future model developers, researchers,
2 The SubseessoaalClnaaeeSAA dataset
The SubseasonalCimateUSAA dataset houses diverse collection of ground-truth measurements and dynamical model forecasts relevant to forecasting at subseasonal timescales. The dataset is regularly updated, CC BY 4.0 licensed, accessible via the open-source subseasonal_ package, and documented at the URL https: / / wiggttt.bb... cowoo/coooooott////ococooooccoot crosoot/subseasonaaa ssusseasbaaaall... _daat/ Alob/main/DAVV md. We summarize dataset contents, sources, and processing steps below and provide supplementary details in Appendix A.
Python
crosoft,
Data Collection and Processing Figure summarizes the SubseasonalCiimateUSA data collection and processing pipeline. The pipeline collects raw data from seven meteorological data sources (contributing different variables, resolutions, and file formats), passes all data through common pre- processing pipeline, and outputs a standardized collection of machine-learmen wwelllannnnnaateeeee Python Pandas DataFrames and Series objects stored HDF5 format. Each file contributes data variables falling into one of three categories: (i) spatial (varying with the target grid point but not the target date); (ii) temporal (varying with the target date but not the target grid point): (iii) spatiotemporal (varying with both the target grid point and the target date). Data representing ground-truth measurements are typically downloaded daily on 0.25% or 0.5% latitude-longitude grids. However, subseasonal forecasts are typically issued on coarser 1 or 1.5 grids and averaged over two-week periods [47, 46, 65]. As result, unless otherwise noted below, temporal and spatiotemporal variables arising from daily data sources were derived by averaging input values over 14-day rolling window, and spatial and spatiotemporal variables were derived by interpolating input data to a 1 latitude-longitude grid and retaining only the grid points belonging to the contiguous U.S. To accommodate ECMWF forecasts, which were only made available on a 1.5 latitude- longitude grid [12], we additionally download SubX forecasts at 1.5º resolution and interpolate temperature and precipitation onto the same grid.
clecao Rediciie Cresse 33PC eats
Tempereture
Prec tallee
Zonal
Longhadralaett
Aea
Fracgitabl
Presuue turbar Poturtal apporaatio
NDW nudi inspelatin Jo0 High
Sea serfbee surpprstare
Campress
Fundas Denifemes
Mudder-adlan
BoW
Eeretton
crsd itterig
Fondify
fation
Agropit
GEPS NESM
Machie Loarsing doady Sursse Cristio
Preisitatie
Figure 1: Schematic of the Subsessona(CimateeSSA data collection and processing pipeline.
3

---

Following the protocol adopted by the USBR Sub-Seasonal Climate Forecast Rodeos [47, 46], our temperature and precipitation variables are interpolated onto fixed 1 1 LNU__1AT=18, NUM_LON JUM_LON=360) and 1.5" x 1.5 NUM_LAT=121, NUM (UM_L0==240) grids using the NCAR Command Language function area area_hiilooe hi2lores, Wrap with arguments new bew_lat latGlobeF [NU,LLA, "lat", "latitude", "degrees_north"); new new_lon = lonGlobeF (NUM_LON, "lon", "Longitude", "degrees_ _aaeee) wgt cos (lat*pi/180 by the cosine of the latitude in radians); opt@critpc = 50 (to require only 50% of the values to be present to interpolate); and fiCyclic True (indicating global data with longitude values that do not quite wrap around the globe). All remaining spatial and spatiotemporal variables interpolated using the Climate Data Operators operator remapdis ddistncee-eighhe average interpolation) with target grid r360x181.
(so that points are weighted
Data Features
The variables comprising the Subseasoonl(CimateUSA4 dataset include:
Temperature (global and U.S., 1979 present): daily mean of maximum and minimum temperature 2 meters in CC [14, 40].
Precipitation (global and U.S., 1948 present): daily accumulated precipitation in mm, aggregated by summing over a rolling two- week window instead of averaging [77, 7, 76, 41, 42].
Sea surface temperature and sea ice concentration (global, 1981 108--eeennt:: daily variables that track variability in the oceans; the top three principal components for each variable were extracted using global 1981-2010 loadings [54, 45].
Stratospheric geopotential height, zonal winds, and longitudinal winds (global, 1948 9448ppresett): daily geopotential height at 10, 100, 500, 850 millibars and zonal and longitudinal winds at 250 and 925 millibars as indicators of polar vortex variability; the top three principal components of each feature were extracted from global 1948-2010 loadings [24, 44].
Surface pressure and relative humidity (U.S., 1948 (048ppeesntt): daily pressure and relative humidity near the surface (sigma level 0.995) [24, 44].
Sea level pressure, precipitable water for entire atmosphere, and potential evaporation (U.S., 1948- ::88ppreeett:: daily mean of pressure in millibars, amount of water in the atmosphere available for precipitation in kg/m², and potential evaporation rate at surface [24, 44].
Elevation and Söppen--Gegeer climate classification (global): multi-resolution terrain elevation data [9] and Köppen-Geigee climate classification [26] for each grid point.
Madden-Juliae Oscillation (MJO, 1974 -r74-eres:n:): daily measure of tropical convection known to
impact subseasonal climate; phase and amplitude were extracted (but not aggregated) [70, 38].
Multivariate ENSO index (MELv22, 1979 ppresent:: bimonthly scalar summary of the state of the El Niño-Southem Oscillation, an ocean-atmosphere coupled climate mode [73-75,
CFSv2 (U.S., 1999-present); daily 32-member ensemble mean forecasts of temperature and precipitation from the coupled atmosphere-ocean-lan dynamical model with 0.5-29.5 day lead times [57, 25, 61].
SubX (U.S., 1999-present): subweekly forecasts and hindcasts from seven dynamical models (GMAO-GEOS, NRI-NESM, RSMAS-CCSM4, ESRL-FIM, EMC-GEFS, ECCC-GEM, NCEP- CFSv2) and their multi-model mean for temperature and precipitation on 1.5% 1.5e latitude- longitude grid [25, 61].
ECMWF (U.S., 1995 0995-present;; control and perturbed forecasts and reforecasts of precipitation and temperature
1.5 x 1.5 latitude-longitude grid [64, 12].
An example of SulsaassoaaCCimateUSAAA observations and dynamical model forecasts is displayed in Figure 2.
Dataset Limitations Here, we highlight several limitations of the SubsessonaCCimateUSAA dataset. First, the dataset was designed for forecasting in the contiguous U.S., and hence several variables are only available in that region. future work, we aim to develop an analogous dataset for global subseasonal forecasting. Second, subseasonal forecasts are commonly made at the biweekly temporal resolution and or 1.5 spatial resolution provided in the SubsessonalClimateUSS dataset [47, 46, 65]; however, these resolutions alone are insufficient for more localized forecasting problems without additional downscaling. Finally, many of our variables have undergone regridding via interpolation, which, while standard, can still introduce inaccuracies.

---

U.S.Temperature
U.S.Pracipitation
2018-01-01
2018-01-01
2018-01-02
2018-01-01
CFSYZ
Observationn
ECHWP
2018-01-01
2018-01-01
2018-01-01
2018-01-01
2018-01-01
2018-01-01
CCSM4
GEOS v2a
CCSM4
GEOS V2o1
NESH
2018-01-04
2018-01-04
2018-01-05
2016-01-04
2018-01-04
2018-01-05
FlMrip1
GEPS
F1Mr11p1
GEFS
GEM
-20
-10
Temperature
Precipitation (mm)
Figure 2: Example of SubsessoonaCCimateUSAA observations and dynamical model forecasts.
3
Subseasonal Forecasting Tasks
We study model performance through four canonical subseasonal forecasting tasks; predicting two variables- average temperature (*C) and accumulated precipitation (mm) over two-week period- each over two time horizons: 15-28 days ahead (weeks 3-4) and 29-42 days ahead (weeks 5-6). We forecast each variable at G = 862 locations a 1 x 1 latitude-longiude grid covering the contiguous U.S. These prediction targets and time horizons were the focus of the Sub-Seasonal Climate Forecast Rodeos [47, 46], two yearlong real-time forecasting competitions sponsored by USBR and NOAA to advance state of subseasonal climate prediction. The same targets are used by water managers apportion water resources, control wildfires, and anticipate droughts and other extreme weather [47, 71].
We evaluate each forecast according to two metrics recommended by the USBR [47, 46]: root mean squared error (RMSE) and skill (also known as uncentered anomaly correlation [72]). For two-week period starting on date f, let E TRG denote the vector of ground-truth measurements Vt.g for each grid point and ý, R denote corresponding vector of forecasts. In addition, define climatology C as the average ground-trume values for given month and day over the years 1981-2010. Then the RMSE is given by
RMSE(V...1) = Eg=1 (0t.g yr,g)² E R+
with smaller value indicating more accurate forecast, and skill defined by
skill(y yr)
(y--cyy--)) Ty- -
E [-1,]]
with larger value indicating higher quality. For a collection of dates, report average RMSE and average percentage skill, which is 100 times the average skill.
Training and Validation Recommendations We recommend adopting the Wednesdays of each complete year from 2011 onward standard test set when evaluating performance year by year (as in Figure 3) and the Wednesdays from the ten year period 2011-2020 as standard test set when comparing with the overall (Table 1) or seasonal (Figure performance reported in this work. We also recommend a progressive training and validation protocol in which, to produce forecast for a given target date, a model can be (re)trained and (re)tuned using any data fully observable on the associated forecast issuance date (i.e., 14 days prior for weeks 3-4 and 28 days prior for weeks 5-6). All of the models evaluated Section respect this protocol. addition, the CaobsessonalClimatellee provides convenient combination dataframes containing target variables associated with predictive features lagged by an appropriate amount to ensure that each predictive feature was fully observed on the issuance date associated with the target date.
dataset
5

---

4
Benchmark Models
Our experiments will evaluate three classes of forecasting methods: three standard metcorological basclines, the adaptive bias correction (ABC) models introduced in [39], and seven other state-of- the-art machine learning and deep learning methods drawn from the literature. We will also evaluate ensemble forecasts derived from these models. Open-source model implementations are available via the subseasonal aooolllttlllltttttttttoo Python package. Most models train on all available data (subject the progressive training and validation constraint of Section 3) and tune hyperparameters using a second round of progressive evaluation over the prior three years. Appendix B contains supplementary implementation details for each model, including training, hyperparameter tuning, and testing details.
Meteorological Baselines
We first consider three standard subseasonal forecasting baselines.
CLIMATOLOGY. Climatology is a standard subseasonal benchmark for the expected temperature or precipitation a location. For given grid point and target date, it forecasts the average value of the target variable on the same day and month over 1981-2010 [3].
DEBIASED CFSv2. CFSv2 is the U.S. operational dynamical model commonly used for subseasonal forecasting [57]. Debiased CFSv2 a corrected ensemble forecast used as a benchmark in the two Subseasonal Climate Forecast Rodeo competitions [46, 23]. First, a CFSv2 ensemble forecast is formed by averaging 32 forecasts for the target period based on different model initializations produced at 8 different lead times. The ensemble is then debiased by adding the mean value of the target variable on the target month and day over the period 1999-2010 and subtracting the mean ensemble CFSv2 reforecast over the same period.
PERSISTENCE. This baseline [37, 69] forecasts the most recently observed two-week target value.
ABC Models We next evaluate the ABC models introduced [39]. ABC is a hybrid physics- plassssspls---- learning approach that takes input dynamical model forecast (here, CFSv2) and uses the historical record of prior forecasts and observations to correct the model's output and improve predictive skill, While the three learning models contributing ABC described below are simple and computationally inexpensive, Section 5 shows that each enhancement improves over both operational practice and state-of-the-art learning techniques.
CLIMATOLOGY++, Climatology++ is as an adaptive form of Climatology that learns how many prior years and how many dates in window around the target date to include in a smoothed historical mean or geometric median estimate for a given grid point, target date, and target variable; importantly, unlike static climatology, Climatology++ allows these learned window sizes to vary over time to adapt to noise levels and variability.
CFSv2++. CFSv2++ is a learned correction for raw CFSv2 forecasts. After averaging CFSv2 forecasts over a range of issuance dates and lead times, CFSv2++ debiases the ensemble forecast by adding the mean value of the target variable and subtracting the mean forecast over a learned window of observations around the target day of year. The range of ensembled lead times, the number of averaged issuance dates, and the size of the observation window employed are selected adaptively.
For each grid point and target date, Persistence+* predicts target variable as function of lagged measurements, Climatology, and CFSv2 forecasts observable for the same grid point on the forecast issuance date. These features are combined using linear least squares regression trained on all available historical data available as of the forecast issuance date.
PERSISTENCE++
State-of-the-art Learning Methods
We also consider seven state-of-the-art learning methods. AUTOKNN. AutoKNN [23] was part of a winning solution in the Subseasonal Climate Forecast. Rodeo I [46]. Our implementation adapts it to target RMSE an error metric.
INFORMER. The Informer [79] is transformer-based deep learning model for time series shown to have sfate-of-the-art performance on a number of short term weather forecasting tasks.
LOCALBOOSTING. LocalBoosting is decision tree model using CatBoost [51] over features in bounding box around the target to extract meaningful spatial information.
MULTILLR. The MultiLLR [23] model is customized backward stepwise procedure to select SubseasonalCCimateeSs.. features relevant for prediction and local linear regression to combine those features into a forecast for each grid point.

---

N-BEATS. N-BEATS [48] is neural network time series forecaster that obtained late-of-the-arr results on the Makridakis M3 [33] and M4 [34] benchmarks for time-series forecasting.
PROPHET. The Prophet model of [62] is an additive regression model for time-series and a winning solution in the Subseasonal Forecast Rodeo II [46].
SALIENT 2.0. Salient 2.0 is an ensemble of fully-connected neural networks, trained on historical surface temperature (SST) data. It is based on Salient [59], winning solution for the Subseasonal Forecast Rodeo I [46].
Ensembles Ensemble forecasts that combine the predictions of multiple models have been shown
improve the performance long., mid-, and short-range operational forecasting [10, 49, 23]. Here, we evaluate two ensembling strategies: Uniform ABC, which forms an equal-weighted average of the ABC model forecasts [27], and Online ABC, which uses the AdaHedgeD algorithm of [15] to choose weights adaptively to reflect relative model performance. See Appendices B. and B.12 for details.
5 Benchmark Results
We now turn to evaluating the models of Section 4 on the four subseasonal forecasting tasks of Section 3. We generate forecasts for each Wednesday in the years 201 1-2020 and, for each reported period, we assess both mean RMSE relative to a baseline model and average percentage skill.
Overall Performance Table summarizes model performance across the entire ten-year period 2011-2020. On each task, we find that the ABC models provide both the best RMSE and the best skill performance. For example, on the two precipitation tasks, Climatology++ alone improves upon debiased CFSv2 RMSE by 9% and skill by 161-250% outperforming each of the meteorological baselines and sfae-oof-the-ar learning methods. On the two temperature tasks, CFSv2++ and Per- sistence++ each outperform all meteorological baselines and state-of-the-art learning methods, with CFSv2+ improving debiased CFSv2 RMSE by 6-7% and skill by 30-53%. On every task, we observe further improvements in both RMSE and skill by ensembling the predictions of the three ABC models.
One might wonder how the simple ABC models are able to outperform both the standard metcorolog- ical baselines and the ofte--of-the-arr art learning methods. We believe the answer to this question is multifaceted. First, even the leading physics- based dynamical models are subject to inaccuracies due to inexact measurement, incomplete representation of the environment, imperfect simulation, and chaos [32]. Second, many the more elaborate machine learning models studied this work appear to be prone to overfitting in the presence of the relatively high noise levels of subseasonal forecasting. Third, the best-perforning models, while relatively simple, are hybrid designed to leverage the strengths of underlying dynamical model while simultancously enhancing its predictive skill by reducing its systemic bias.
plus-learninn models
Amongst the tate-of-the-ar learning methods, we find that Prophet performs the best for temperature weeks 5-6 and the two precipitation tasks, while MultiLLR performs the best for temperature weeks 3-4. Amongst the neural network methods (Informer, N-BEATS, and Salient 2.0). Salient 2.0 is the top performer with skill that rivals the other learning methods and precipitation RMSE that outpaces debiased CFSv2. In the more detailed analyses to follow, we omit Informer and N-BEATS due to space constraints and their relatively poor performance overall.
Performance by Season and by Year We observe the same trends when performance is disaggre- gated by season or by year (Figure 3). For example, in every season. Climatology++ outperforms debiased CFSv2 and each state-of- of-teea--t-- learner for the two precipitation tasks, while CFSv2++ and Persistence++ outperform debiased CFSv2 and each state-of-the-arr learner each season for the two temperature tasks. Similarly and despite significant heterogeneity in all models' performances from year to year, the ABC models provide the best RMSE performance in 9 out of 10 years for temperature weeks 3-4 and in 10 out of 10 years for the two precipitation tasks. Indeed, Persistence++ alone dominates the temperature weeks 3-4 baselines and learners every year save 2019, and the more detailed RMSE summary of Appendix C.2 shows that the Uniform and Online ABC ensembles dominate the precipitation baselines and learners every year. In Figure 3, we observe nearly identical improvement patterns for skill.
7

---

Table 1: Average percentage skill and percentage improvement over mean debiased CFSv2 RMSE across 2011-2021 in the contiguous U.S. along with a 95% bootstrap confidence interval. The best performing model each model group is bolded, and the best performing model overall is shown in green.
EMPROVEMENT OVER MEAN DER. CFSV2 RMSE
AVERAGE
SKILL
PRECIPEATION
GROUP
MODEL
WEEKS
6
5
WEEKS
W
W
5
BASILINTS
CLIMATOLDGY
0.13+133
2.93-1.23
7,7940.56
7.51-0.88
CFSv2
24.94:1.88
19.12=1.94
5:77e1-11
4.28-1.99
109.94:5.3
-170.187.56
-2827761.47
-31,92e1.55
10.64:2.26 26
6.22=2.38
5.31=0.99
7.41=1011
ABC
2.06+1.35
4.63e1.18
8.M6+n.53
8.57=0.5
18.61=1 95
18.87=1.92
15:04=1
14.99=1.00
CFSv2
5.942100
1.09=1.02
8.37+00.5
5.06:0.44
32.38=1.75
19.19-1.76
16.34:1.03
16.09=1007
6,00:1106
6.43:00.9
8.6140.51
7.89:0.45
32.4:1171
26.73:1.67
13.38:0091
9.77=0.9
LEARNING
AUTOKNN
0.951 1.33
3.22 1.25
7,731 0.56
1.33
12.43.11.7
S.56.11.52
6.66.1:23
5.93
INFORMER
-40.61::444
-3
39.591.3.89
-2.0510.86
-1,53200.8
0.55:12.22
0.01::2.00
6.151125 12
1.8611.31
LOCALROOSTING
-0.70 0.76::1224
-0.29
7.36110.58
6.69:0.55
14.44:1 59
12.69:1 64
10:82:0:87
9.72-0.86
MULTILLR
2.45:11.18
1.2111.224
7.1210.51
6.6520.47
14.5.1.77
16.6821.85
9,4911033
1.97=1.04
DEATS
46.71.12.48
-52.08.2.889
19.1910.992
-21.32.0.89
9.21 1.40
4.16.11.39
5.48.0.59
0.46-0062
PROPRE7
1.13:1.4
1.28
K.42.10.55
0.12:00.11
20.21 54
19.76_1.57
13.51.0.87
11.41
SALIENT
6.95111.89
1..05
2.971.0.74
2.65.10.69
11:24:2204
11,77:2.03
10.1121336
9.99-1.31
ENSEMBLDS UNIPORM ABC
6,47+11099
7.55=0.99
9,47+005
9.05=0.45
33
38,56=1.7
18.94-0.98
18.35-1.01
ONLINE ABC
6.67:10.99
7,67:10.98
9.51.10.11
9.0620:42
33.27:1.711
30.06:1.72
18 18.86.1.011
17.91
Spatial Performance Figure 4 displays how the errors of the leading models are distributed across the contiguous U.S. Here we focus on the ABC models, the best deep learning model (Salient 2. 2.0), the best learning model (Prophet), and the best ensemble model (Online ABC) and provide RMSE improvement maps for the remaining models in Appendix C.4. At each grid point location, darker green indicates stronger improvement over debiased CFSv2, and we simultaneously witness two noteworthy phenomena. First, the improvements of each model are heterogeneous across space with the strongest improvements often occurring in the Western U.S., in Florida, in Maine. Second, despite this heterogeneity, the ABC models consistently outperform the statc-of-the the----rrt learners.
ECMWF Comparison To compare the ABC models with the state-of-the-art ECMWF S2S dynamical model, we evaluate on the 1.5 x 1.5 grid and 2016-2020 wice-weeklly target date range available from [12, 66]. We debias both the ECMWF control forecast and its 50-membe ensemble forecast following the operational protocol described by [69]; see Appendix B. for more details. Table 2 summarizes model performance. Remarkably, for precipitation, Climatology++ improves upon both the skill and the RMSE of ECMWF, despite making no use of dynamical model forecasts. Meanwhile, the Uniform ABC ensemble outperforms ECMWF in both metrics for all four tasks.
Table 2: Average percentage skill and percentage improvement over mean debiased CFSv2 RMSE across 2016-2020 in the contiguous U.S. along with 95% bootstrap confidence interval. The best performing model in each model group is bolded, and the best overall is shown in green.
EMPROVEMENT oven MEAN DEs. CFSv2 RMSE
TIMPERATUR AVeRAGE & SKILL
TEMIRRATURE
PRZCIRVTATON
PRECIPITATION
GROUP
MoozL
WEEKS 3-4
WIEKS 5-5
WHIKS 3-4
WHEKS 5-6
WHEKS 3-4
WEEKS 5-6
WHEKS 3-4
WHEKS 5-6
RASTLINES
CLIMATOLOGY
1,36:1.39
1.92
8.7=0.5
7.56:0.53
DES. CFSV2
22.64:2.11
15.71+2.09
2.8481.16
1.681.11
PERSINTENCE
105.5743.55 58
-169.2267.99
28.05-1.56
-33.4361.6
9.1262448
2,2762.48
811:1.07
6,21:1.04
ABC
CLISATOLOGYY4
3.8811.38
6.44
9,79:0.53
8.110051
22.0911.89
23.261.91
15.34:1.05
15.06:1.07
CTSV
5.6511.04
6.651:996
8,90:00311
1.669.46
30.91118
1..87
14.621.25
13,8511.2
PresiSTENCE
7.06:1.85
7.86::.95
9.06=0048
7.57::0.6
31.46:1.92
28.04:188
10.0380.99
6.61:0.95
RCMWF
DERIASED CONTROE
-20.0544..0
253.25-2.69
- -30.81=1.45
-51 $1.84:1143
18.52-1.82
13.71=1.85
0.82-1.07
3.17=1.08
DESIASED Extesmue
4.62:11.19
3.69:1.27
7.90=0.5
6.41:0.46
32.27:1.69
26.61:1.71
13.12:116
9.10:1.09
RNSTDBBLES
UNIRORM ABC
7.43-1188
8.27-0.94
10.04-0.5
8.93-0046
32.77-1.79
16.53-1.15
18.71-1.17
ONLINE ABC
7:261:07
7.9610095
10.0N:00.48
5.6220.47
32.22+1.82
25.3811.87
17.19:1.13
15.42.1.15
Graphcast Comparison Recently, the GraphCast deep learning model [28] was shown out- perform both the leading dynamical model and the Pangu- Weather deep learning model [5] on a range of 10-day weather forecasting tasks. While GraphCast was not developed for subseasonal forecasting, we can, as recommended by an anonymous reviewer, benchmark its performance on our subseasonal forecasting tasks. For this evaluation, we restrict our standard test set to the years 2018-2020, as GraphCast was trained on data through 2017, and report performance and additional experimental details in Appendix C.6. Consistent with the other deep learning methods benchmarked in Table 1, CraphCast outperforms debiased CFSv2 in terms of skill, underperforms debiased CFSv2 in terms of RMSF, and strongly underperforms the ABC ensemble models in both metrics when forecasting either temperature precipitation in weeks 3-4.
8

---

improvement by season
U.S.
weeks 5-6
125
procipitation, weeks
12%
U.S. precipitation,
Frudeet
12%
11%
11%
110%
10%
10%
10%
5%
5%
5%
0%
4%
7%
7%
2%%
6%
Peb
1907
jun- AVR
sop plae
Feb
sar W0
jan AND
Feb
Moy
AUe
Fob
jun- Ann
Now
RMSE Improvement by year
U.S. Temporature, weeks 3-4
U.S. tomporature, weeks 5-6
U.S. precipitation, weeks
U.S. precipitation weeks 5-6
13%
18%
12%
12%
Chisndaggee
Locklbarttg
35%
112%
12%
10%
30%
4%
4%
20
2015
2017
2010
2010
2015
2013
2010
2015
2013
2013
2017
2018
2018
2030
Skill by season
80%
temperature, weeks
90%
tomperature, weeks
precipitation,
precipitation,
mohe Kalbod 2.0
20%
20%
0%
Dec
you7 way
junk AUD
sep
Not
5%
Fab
way
AP9
sap
0%
Fel
your May
jun- AMg
pap
Fab
your May
jun. ARg
Sep Nov
Skill
U.S. tamperature, wenks 3-4
80%
U.S. temparature, weeks 5-6
U.S. precipitation, weeks 3-4
25%
U.S. precipitation, weeks 5-6
belleeed CFSa
MUTLLL
20%
20%
60%
60%
AusOKAM
15%
80%
40%
18%
0%
2013
2014
2015
2010
2012
2013
2014
2014
2017
2030
2011
2012
2014
2010
2014
2017
2018
2010
2010
5%
2011
2012
2012
2017
2012
Figure 3: Per season and per year average skill and improvement over mean debiased CFSv2 RMSE
across the contiguous U.S. and the years 2011-2020. Despite their simplicity, the ABC models (solid lines) consistently outperform debiased CFSv2 and the state-of-the-arr learners (dotted lines).
Western U.S. Competition Results Finally, we evaluate our models on the exact geographic region and target dates of the recent Subseasonal Climate Forecast Rodeo II competition. Specifically, we produce forecasts for the Western U.S. region, delimited by latitudes 25N to SON and longitudes 125W to 93W, at a 1 x 1 resolution for a total of G = 514 grid points. Forecasts were issued every two weeks for a yearlong period with initial issuance date October 29, 2019 and final issuance date October 27, 2020, leading to noisier evaluation with only 26 observations.
Table 8 in Appendix C.7 compares the predictive accuracy of the models studied this work with the accuracy of the contest baselines (debiased CFSv2, Climatology, and, for precipitation only, the Rodeo Salient model of [59]) and the performance of the top competitors for each task. For temperature weeks 3-4, Persistence+* provides a 16.59% improvement over the mean debiased CFSv2 RMSE, outperforming the contest baselines, the state-of-the-ar learning methods, and all but two of the competitors (the top three competitors improved by 17. 17.12%, 16.67%. and 15.47%). For temperature weeks 5-6, CFSv2++ provides a 9. 9.26% improvement over debiased CFSv2 and, despite its simplicity, outperforms the contest baselines, the state-of-the -th--at learning methods, and all of the

---

U.S. temperature, weeks 3-4
Climatology (2.06%)
CFSV2++ (5.94%)
Persistence+= (6.0%)
Prophet (1.13%)
Sallent 2.0 (4.95%%
Online ABC (6.67%)
12%
8%
4%
temperature, Persistencer (6.43%)
Climatology (4.83%)
CFSv2+ (7.09%)
Prophet (3.78%)
Sallent 2.0 (-4.05%)
Online ABC (7.67%)
32%
8%
4%
0%
U.S. precipitation, weeks 3-d
Climatolgy++ (8.06%)
crSv2+ (8.37%)
Persistences+ (8.61%)
(8.42%)
Sallent (2.97%)
Online (9.51%)
20%
16%
U.S. precipitation, weeks 5-6 Prophet (8.125)
Climatlogy++ (8.57%)
crSv24- (8.00%)
Persistencee (7.89%)
Salient 2.0 (2.65%)
Online ABC (9.06%)
16%
12%
8%
4%
Figure 4: Percentage improvement debiased CFSv2 RMSE in the contiguous U.S. over 201 2011-2020. White grid points indicate negative or 0% improvement.
competitors the subseasonal forecasting competition (the top competitor improved by 8.47%). On this task, the Uniform and Online ABC ensembles also outperform all competitors.
On the precipitation tasks, the Salient baseline performed strongly and ultimately placed second and fourth respectively for the weeks 3-4 and weeks 5-6 tasks. Our Salient 2.0 model also performs remarkably well, outscoring all contestants and baselines with 12.65% improvement for weeks 3-4. For comparison, the top competitors for weeks 3-4 and weeks 5-6 improved by 1 11.54% and 8.63% respectively. Our Uniform ABC ensemble outperforms the remaining basclines and state-of the-art art learning methods but falls short of the exceptional Salient performance. In this setting, applying the adaptive online learning ensemble to the union of the ABC models and the state-of-the-ar learners (denoted by Online ABC Learning in Table 8) allows the user exploit the irregular complementary benefits of the learning methods yielding 12.52% and 8.18 181 improvements weeks 3-4 and 5-6.
6 Conclusion
In this work, we release SubbsessonaCCimateUSSA. a dataset for subseasonal forecasting in the U.S. It is routinely updated and can be accessed Python package. The dataset includes variety of features that are relevant at the subseasonal timescale, including precipitation, temperature, surface pressure, relative humidity, geopotential height, sea surface temperature, sea ice concentration. MJO, and MEI. We use this dataset to train and benchmark multifarious models, including deep learning solutions, dynamical models, and simple learned corrections, well ensembling strategies. Our experiments with temperature and precipitation forecasting in the contiguous U.S. show that simple learning-bast based corrections to operational dynamical models yield low-cost strategies that are 10% more accurate and 329% more skillful than the U.S. operational CFSv2 and outperform state-of-the. art machine and deep learning methods, as well as the leading ECMWF dynamical model. Overall, dataset facilitates both the training and benchmarking of
we find that the subseasonal forecasting models and hope that it will stimulate new advances in extended range forecasting.
10

---

Acknowledgments and Disclosure of Funding
We acknowledge the agencies that support the SubX system, and thank the climate modeling groups (Environment Canada, NASA, NOAA/NCEP, NRL and University of Miami) for producing and making available their model output. NOAA/MAPP, ONR, NASA, NOAA/NWS jointly provided coordinating support and led development of the SubX system. This work is based on S2S data. S2S a joint initiative of the World Weather Research Programme (WWRP) and the World Climate Research Programme (WCRP). The original S2S database is hosted at ECMWE as an extension of the TIGGE database. This work was supported by Microsoft for Earth (S.M. and G.F.); the Climate Change Al Innovation Grants program (S.M., P.O., J.C., E.F., and I.MM.,, hosted by Climate Change AI with the support of the Quadrature Climate Foundation, Schmidt Futures, and the Canada Hub of Future Earth; FAPERJ (Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro) grant SEI-260003/001545//0 545/222222 (P.O.); NOAA grant SARR-WO-2022-2000 (G.F,, J.C., and L.M.); and the National Science Foundation grant 19R-190 (J.C.).
References
[1] Nans Addor, Andrew Newman, Naoki Mizukami, and Martyn P Clark. The camels data set: catchment attributes and meteorology for large-sample studies. Hydrology and Earth System Sciences, 21(10):5293-5333 2017.
[2] Troy Arcomano, Istvan Szunyogh, Jaideep Pathak, Alexander Wikner, Brian R Hunt, and Edward Ott. A machine learning-basee global atmospheric forecast model. Geophysical Research Letters, 470):220000LL87777
2020.
[3] Anthony Arguez. Imke Durre, Scott Applequist, Russell S Vose, Michael F Squires, Xungang Yin, Richard R Heim Jr, and Timothy W Owen. Noaa's 1981-2010 us climate normals: an overview. Bulletin of the American Meteorological Society, 93(1 93(11))1687-1697, 2012.
[4] Clara Betancourt, Timo Stomberg, Ribana Roscher, Martin G Schultz, and Scarlet Stadtler. Aq-bench: A benchmark dataset for machine learning on global air quality metrics. Earth System Science Data, 13(6)33013-3033, 2021.
[5] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range rrange global weather forecasting with 3d neural networks. Nature, 619(7970)5533-55 2023.
[6] Philippe Bougeault, Zoltan Toth, Craig Bishop. Barbara Brown, David Burridge, De Hui Chen, Beth Ebert, Manuel Fuentes, Thomas M Hamill, Ken Mylne, et al. The thorpex interactive grand global ensemble. Bulletin of the American Meteorological Society, 105: 2010.
[7] Mingyue Chen, Wei Shi, Pingping Xie, Viviane BS Silva, Vernon E Kousky, R Wayne Higgins, and John Janowiak. Assessing objective techniques for gauge-based analyses of global daily precipitation. Journal of Geophysical Research: Atmospheres, 113(D4), 2008
[8] Cohen, D. Coumou, J. Hwang, L. Mackey, P. Orenstein, S. Totz, and E. Tziperman. $2S reboot: An argument for greater inclusion of machine learning in subseasonal to seasonal (S2S) forecasts. WIREs Climate Change, 10, 2018
[9] Jeffrey J Danielson and Dean B Gesch. Global multi-resolution terrain elevation data 2010 (GMTED2010). US Department of the Interior, US Geological Survey, 2011.
[10] Jun Du, Judith Berner, Roberto Buizza, Martin Charron, Pieter Leopold Houtekamer, Dingchen Hou, Isidora Jankov. Mu Mu, Xuguang Wang. Mozheng Wei, et al. Ensemble methods for metcorological predictions. Office note (National Centers for Environmental Prediction (U.S.)). 2018.
[11] Peter Dueben, Martin G Schultz, Matthew Chantry, David John Gagne, David Matthew Hall, and Amy McGovern. Challenges and benchmark datasets for machine learning in the atmospheric sciences: Definition, status, and outlook. Artificial Intelligence for the Earth Systems, 1())cc21002,, 2022.

---

[12] ECMWF S2S data. Ecmwf S2S ecmf: Ecmwf ensemble. https://iridl 1deo. columbia. edu/SOURCES/ ECMWF/ S2S/ /ECMF/ 2021.
[13] Veronika Eyring. Sandrine Bony, Gerald A Meehl. Catherine A Senior, Bjom Stevens, Ronald J Stouffer, and Karl Taylor. Overview of the coupled model intercomparison project phase (cmip6) experimental design and organization. Geoscientific Model Development, 995: 1958, 2016.
[14] Yun Fan and Huug Van den Dool. A global monthly land surface air temperature analysis for 1948-present.
1948 present. Journal of Geophysical Research: Atmospheres, 113(1)), 2008
[15] Genevieve Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, and Lester Mackey. Online learning with optimism and delay. In International Conference on Machine Learning. PMLR, 2021.
[16] Gabriele Franch, Valerio Maggio, Luca Coviello, Marta Pendesini, Giuseppe Jurman, and Cesare Furlanello. 19 high-resolution weather radar reflectivity dataset for precipitation nowcasting. Scientific Data, 7(1):234, 2020.
[17] Isaac Godfried, Kriti Mahajan, Maggie Wang, Kevin Li, and Pranjalya Tiwari. Flowdb large scale precipitation, river, and flash flood dataset. arXiv preprint arXiv:2012.11154, 2020.
[18] Alex M Haberlie, Walker S Ashley, and Marisa R Karpinski. Mean storms: Composites of radar reflectivity images during two decades of severe thunderstorm events. International Journal of Climatology,
2021.
[19] Sijie He, Xinyan Li, Timothy DelSole, Pradcep Ravikumar, and Arindam Banerjee. Sub- seasonal climate forecasting via machine learning: Challenges, analysis, and advances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 169-177, 2021.
[20] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 08415, 2016.
[21] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Sociery, 146(730):1999-225)
2020.
[22] S. Hoyer and J. Hamman. xarray: N-D labeled arrays and datasets in Python. Journal Open Research Software, 5(1), 2017.
[23] Jessica Hwang, Paulo Orenstein, Judah Cohen, Karl Pfeiffer, and Lester Mackey. Improving subseasonal forecasting in the western U.S. with machine learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, KDD '19, page 2325-2335, New York, NY, USA, 2019. Association for Computing Machinery.
[24] Eugenia Kalnay, Masao Kanamitsu, Robert Kistler, William Collins, Dennis Deaven, Lev Gandin, Mark Iredell, Suranjana Saha, Glenn White, John Woollen, al. The ncep/ncar 40-year reanalysis project. Bulletin of the American meteorological Society, 77(3):437-47,, 1996.
[25] BP Kirtman, Pegion, DelSole, M Tippett, AW Robertson, M Bell, R Burgman, H Lin, J Gottschalck, DC Collins, et al. The subseasonal experiment (subx). IRI Data Library, 10:DSPG249H, 2017.
[26] Markus Kottek, Jürgen Grieser, Christoph Beck, Bruno Rudolf, and Franz Rubel. World map of the Köppen-Geiger climate classification updated. Meteorologische Zeitschrift, 15(3):259-263, 2006.
[27] TN Krishnamurti, Chandra M Kishtawal, Timothy E LaRow, David R Bachiochi, Zhan Zhang, Eric Williford, Sulochana Gadgil, and Sajani Surendran. Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 155 1999.
12

---

[28] Remi Lam, Alvaro Sanchez-Gonzalee Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds. Zach RattonRRose, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Science, 382(6677): 1416-11111,,11 2023.
[29] Andrea L Lang, Kathleen Pegion, and Elizabeth Barnes. Introduction to special collec- tion: "Bridging weather and climate: Subseasonal wtw-ooonnn--------ooti--oo (S2S) prediction". Journal of Geophysical Research: Amossperes, 125(4))200191003
2020.
[30] DJ Lea, Mirouze, MJ Martin, RR King. A Hines, D Walters, and M Thurlow. Assessing a new coupled data assimilation system based on the met office coupled dimosphere-land-ocean ice model. Monthly Weather Review, 143(11):466 (3611))4678-4664 2015.
[31] Laifang Li, Raymond W Schmitt, Caroline C Ummenhofer, and Kristopher B Karnauskas. Implications north atlantic sea surface salinity for summer precipitation over the us midwest: Mechanisms and predictive value. Journal of Climate, 29(9):3143-311 2016.
[32] E.N. Lorenz. Deterministic nonperiodic flow. Journal of the Atmospheric Sciences, 10(2):130-141, 1963.
[33] Spyros Makridakis and Michele Hibon. The m3-competition: results, conclusions and implica- tions. International journal of forecasting, 16(4):451-476, 2000.
[34] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54-
74, 2020.
[35] Manil Maskey, Rahul Ramachandran, Muthukumaran Ramasubramanian, Iksha Gurung, Brian Freitag, Aaron Kaulfus, Drew Bollinger, Daniel J Cecil, and Jeffrey Miller. Deepti: Deep- learning-based tropical cyclone intensity estimation system. IEEE journal of selected topics in applied Earth observations and remote sensing, 13:4271-4287 2020.
[36] William J Merryfield, Johanna Baehr, Lauriane Batté, Emily J Becker, Amy H Butler, Caio AS Coelho, Gokhan Danabasoglu, Paul A Dirmeyer, Francisco J Doblas-Reyes, Daniela IV Domeisen, et al. Current and emerging developments in subseasonal to decadal prediction. Bulletin of the American Meteorological Society, 1016))E899-E896 2020.
[37] Marion P Mittermaier. The potential impact of using persistence as a reference forecast on perceived forecast skill. Weather and forecasting. 23(5):1022-100) 1031, 2008
[38] MJO data. Real-time multivariate Madden Julian Oscillation index. https://iridl. Ideo. columbia. pacduu/dcchelp/QA.. http//////eaaaaaaaaaaiaaaaaaaaae citation html, 2021.
[39] Soukayna Mouatadid, Paulo Orenstein, Genevieve Flaspohler, Judah Cohen, Miruna Oprescu, Ernest Fraenkel, and Lester Mackey. Adaptive bias correction for improved subseasonal forecasting. arXiv preprint arXiv:2209.10666 2022.
[40] NOAAVOAR/ESRL PSL. CPC global temperature data. ftp://ftp. edc. nww.. Datasets//pp__glbbl__rrmm// 2021.
[41] NOAA/OAR/FSRIL CPC global unified precipitation data, ftp://ftp.cc edc. noa.. gov/ pataeets/pp_ggtbaalppeeepp.pp.
[42] NOAAVOARESSRL CPC US unified precipitation data. ftp://ftp. cdc. noaa gov/
Datraetts/cpp.uuapreccpp...1111
[43] NOAA/OAR/ESSRI PSI. Multivariate El Niño/Southern Oscillation (ENSO) index. https:
www../noaa.gooo/eo////eeeee http:///eiii/aaaa/ieeeee/ data, 2021.
[44] NOAAVOAR/ESRL PSL. NCEP reanalysis data. Geopotential height, zonal wind, and lon- gitudinal wind: //f::/ftp. cdc. noaa. gov /aa///aaattttaaaa reanalysis. dailyavgs/ pressure/; Relative humidity, sea level pressure, and precipitable water for entire atmosphere: ftp://fft cdc. noaa. pov/Datasets/nnceg pttp///taeee//cc....aaaaa//////a
Pressure at the sur- reanalysis/
face and potential evaporation: ftp://ftp cdc. edc. noa.
surface_gause. 2021.
13

---

[45] NOAAVOAR/ESRL PSL. NOAA high resolution SST data. ftp://ftp. cdc. noaa noaa.gov Projects/ /at.aaatttttttaaaaaaaaaaaaa oisst. v2. wi.hireeee/ 2021.
[46] Kenneth Nowak, Ian M Ferguson, Jennifer Beardsley, and Levi Brekke. Enhancing western united states sub-seasonal forecasts: Forecast rodeo prize competition series. In AGU Fall Meeting 2020. AGU, 2020.
[47] Kenneth Nowak, RS Webb, R Cifelli, and LD Brekke, Sub-seasonal climate forecast rodeo. In 2017 AGU Fall Meeting, New Orleans, LA, pages 11 11-15, 2017.
[48] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: neural basis expansion analysis for interpretable time series forecasting. In 8th International Con- ference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[49] Tim Palmer. The ecmwf ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years. Quarterly Journal of the Royal Meteorological Society, 145:12-24, 2019.
[50] Kathy Pegion, Ben P Kirtman, Emily Becker, Dan C Collins, Emerson LaJoie, Robert Burgman, Ray Bell, Timothy DelSole, Dughong Min, Yuejian Zhu, al. The subseasonal experiment (subx): A multimodel subseasonal prediction experiment. Bulletin of the American Meteorolog- ical Society, 100(10):2043-206) 2019.
[51] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In Advances in neural information processing systems, pages 6638-6648, 2018.
[52] Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: A benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(1 20000M0220000,,,,,,,,, 2020.
[53] Stephan Rasp, Hauke Schulz, Sandrine Bony, and Bjorn Stevens. Combining crowdsourcing and deep learning explore the mesoscale organization of shallow convection. Bulletin of the American Meteorological Society,
2020.
[54] Richard W Reynolds, Thomas M Smith, Chunying Liu, Dudley B Chelton, Kenneth S Casey, and Michael G Schlax. Daily high-resolution--lendee analyses for sea surface temperature. Journal of climate, 20(22):5473 (4925)5577--5466 2007.
[55] David Rolnick, Priya I. Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Hilojevic-Duponee Natasha Jaques, Anna Waldman- Brown, et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 15(2):1-96, 2022.
[56] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[57] Suranjana Saha, Shrinivas Moorthi, Xingren Wu, Jiande Wang, Sudhir Nadiga, Patrick Tripp. David Behringer, Yu-Tai Hou, Hui-ya Chuang, Mark Iredell, et al. The ncep climate forecast system version 2. Journal of climate, 2165:2285522008 2014.
[58] Ray Schmitt. private communication, Jul. 2021.
[59] Raymond Schmitt. Salient predictions: Validation summary. http:://sttrragggggogleeee comtentn sali entpredi ctions. httppp/raiiiiiiio.ooooaaaaaaaaaa http:///w....att.oaaaaaaaaaa/llr pdf, 2019. Accessed: 2021-05-29.
[60] Vishwak Srinivasan, Justin Khim, Arindam Banerjee, and Pradeep Ravikumar. Subseasonal climate prediction in the western us using bayesian spatial models. In Uncertainty in artificial intelligence, volume 37, 2021.
[61] SubX data. http:// bttp://iridl /rrill Ideo. www.woii.a......ccc...o........e www. Bww.URREES/MMoee/...oo Models/.SubX/, DOI: attps://dos org/ 10. 7916/ DD8GG299,,, 2021.
14

---

[62] Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1))37-
45, 2018.
[63] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-277 2020.
[64] F Vitart, C Ardilouze, A Bonet, A Brookshaw, M Chen, C Codorean, M Déqué, L Ferranti, Fucile, M Fuentes, et al. The subseasonal seasonal (S2S) prediction project database. Bulletin of the American Meteorological Society, (8(1):163-111 163-173, 2017.
[65] F. Vitart, A. W. Robertson, A. Spring, F. Pinault, R. Roškar, W. Cao. S. Bech, A. Bienkooski, N. Caltabiano, E. De Coning, B. Denis, A. Dirkson, J. Dramsch, P. Dueben, J. Gierschendorf, H. S. Kim, K. Nowak, D. Landry, L. Lledó, L. Palma, S. Rasp, and S. Zhou. Outcomes of the wmo prize challenge to improve subseasonal to seasonal predictions using artificial intelligence. Bulletin of the American Meteorological Society, 103(12):E2878 - E2886, 2022.
[66] Frédéric Vitart, Andrew W Robertson, and David LT Anderson. Subseasonal to seasonal prediction project: Bridging the gap between weather and climate. Bulletin of the World Meteorological Organization, 61(2):23, 2012.
[67] C Wang. Z Jia, Z Yin, Liu, G Lu, and Zheng. Improving the accuracy of subseasonal forecasting of china precipitation with machine learning approach. front. Earth Sei, 9:659310, 2021.
[68] Duncan Watson-Parris. Machine learning for weather and climate are worlds apart. Philosophi- cal Transactions of the Royal Society A, 379(2) (7902194):20200098, 2021.
[69] Jonathan A. Weyn, Dale R. Durran, Rich Caruana, and Nathaniel Cresswell-Clay. Sub-seasonal forecasting with large ensemble of deep-learning weather prediction models. Journal of Advances in Modeling Earth Systems, 13(7) 2021.
[70] Matthew C Wheeler and Harry H Hendon. An all-season real-time multivariate mjo index: Development of an index for monitoring and prediction. Monthly weather review, 132(8):1917- 1932, 2004.
[71] Christopher J. White, Henrik Carlsen, Andrew W. Robertson, Richard J.T. Klein, Jeffrey K. Lazo, Arun Kumar, Frederic Vitart, Erin Coughlan de Perez, Andrea J. Ray, Virginia Murray, Sukaina Bharwani, Dave Macleed, Rachel James, Lora Fleming, Andrew P. Morse, Bernd Eggen, Richard Graham, Erik Kjellstrom, Emily Becker, Kathleen V. Pegion, Neil J. Holbrook, Darryn McEvoy, Michael Depledge, Sarah Kirkptrk Timothy Brown, Roger Street, Lindsey Jones, Tomas A. Remenyi, Indi Hodgson-Johnstee Carlo Buontempo, Rob Lamb, Holger Meinke, Berit Arheimer, and Stephen E. Zebiak. Potential applications of subseassoall-o--saaooni (S2S) predictions. Meteorological Applications. 24(3):315-325, 2017.
[72] Daniel S Wilks. Statistical methods in the atmospheric sciences, volume 100. Academic press, 2011.
[73] Klaus Wolter and Michael Timlin. Monitoring enso in coads with a seasonally adjusted principal. In Proc. of the 17th Climate Diagnostics Workshop, Norman, OK, NOAA/NMC/CAC, NSSL, Oklahoma Clim. Survey, CIMMS and the School of Meteor, Univ. of Oklahoma, 52, volume 57, 1993.
[74] Klaus Wolter and Michael S Timlin. Measuring the strength of ENSO events: How does 1997/98 rank? Weather, 53(9):315-33) --24, 1998.
[75] Klaus Wolter and Michael S Timlin. El Niño/Southern Oscillation behaviour since 1871 as diagnosed in an extended multivariate ENSO index (MEL.ext). International Journal of Climatology,
2011.
15

---

[76] Pingping Xie, M Chen, and W Shi. CPC unified gauge-base based analysis of global daily pre- cipitation. In Preprints, 24th Conf. on Hydrology, Atlanta, GA, Amer. Meteon Soc, volume 2010.
[77] Pingping Xie, Mingyue Chen, Song Yang, Akiyo Yatagai, Tadahiro Hayasaka, Yoshihiro Fukushima, and Changming Liu. A gauge-based analysis of daily precipitation over east asia. Journal of Hydrooetteorology, 8(3):607-626, 2007.
[78] Akio Yamagami and Mio Matsueda. Subseasonal forecast skill for weekly mean atmospheric variability over the northern hemisphere in winter and its relationship to midlatitude teleconnec- tions. Geophysical Research Letters, 47(17):22020LL088500 2020.
[79] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In The Thirty-Fifth AAA/ Conference on Artificial Intelligence, AAAI 2021, page online. AAAI Press, 2021.
16

---

Supplementary Material for SubseasonalClimateUSS. A Dataset for Subseasonal Forecasting and Benchmarking
A
SubseesonalCClmateUSAA Supplementary Details
The subseasonal_dast Python package provides a detailed description of the SubseasonalClima- teUSA dataset contents, sources, and processing steps at the following URL:
https:// cttp:///tceoottttaaaaaaaaaaaaa/a
A.1 Computational Environment Details
The SubsaesonaCCimateUSAA update pipeline rums in a Docker container on an Azure E16-4ds, instance with vCPUs/cores and 128 GB of RAM. The update pipeline runs in under hours at total cost of $5, All benchmarking experiments were run on the Massachusetts Institute of Technology engaging cluster (https://engaging web. mit edu/eofe-wiki/).
A.2 Western U.S. Competition Data Details
Following [23], for the Western U.S. competition experiments Section 5, the sea surface temperature and sea ice concentration variables were formed by identifying the top three principal components for each variable restricted to the Pacific basin region (20S to 65N, 150E to 90W) using loadings from 1981-2010.
B Model Implementation Details
This section describes the implementation details for each learning model, including the training, hyperparameter tuning, and validation protocols. All models were implemented in Python 3.
B.1 Climatology++
Climatology++ was trained and tuned following the protocol described in [39]; Algorithm 1. Figure 5 displays the selected window length (the span 8) and number of years Y for each target date in 201 1-2020 when forecasting for the contiguous U.S. We see that the temperature models preferred fewer training years and larger windows around the target day in recent history but focused more exclusively on the target day of year (via span of 0) 2013-2016 and preferred more training years in 2011. 1. Meanwhile, the precipitation models selected the largest available window (corresponding higher bias but lower variance estimates) for nearly every target date.
U.S. temperature, weeks 3-4
U.S. temperature, weeks 5-6
U.S. precipitation, weeks 3-4
U.S. precipitation, weeks 5-6
yerr 10
Figure 5: Climatology+ hyperparameters automatically selected for each target date in 2011-2020.
B.2 CFSx2++
CFSv2++ was trained and tuned following the protocol described in [39]; see Algorithm 2. Figure 6 displays the selected window length (the span s), lead time range, and issuance date count for each target date in 2011-2020 when forecasting for the contiguous U.S. In each task, observe
17

---

Algorithm 1 Climatology++
input test date [*; # train years Y; span 8; loss (RMSE,SEE); training set ground truth (y:)LET initialize days per year
365.242199
S (t T _ diff = - s Y and day_diff = 365 8}
303 2 -[((t- mod DJ-
2 S
output argminy Eies yr)
significant amount of variability in the optimal span, lead, and date count selections, highlighting the value of adaptive ensembling and debiasing over the static ensembling and debiasing strategies employed by standard debiased CFSv2.
U.S. temperature, woeks
U.S. temperature, weeka
1pan dales leads 25:29
1pan 25, datee
ipan 28, ueds
span 25 detse leads 13:23
detee 23-23
span 33, diles waak 20-22
apan dates 29-29
28, detes
spee
1b eates 16 leats
25, dides maah 20-29
28 eatre 16 leads
span 24, datee
-11-stee +7, laads 13-23
span 14, dates 42 isads 23-23
span 14, dates marh 20-20
spon - detes 7, leads 15:23
span 24, dates 20-29
1pan
U.S. precipitation, weeks
U.S. pwwpppptttionnnnommmmmm we 5-6
span dates 28, ladd
1pan 35, detes 24, merts 20-23
lends
lead
dates mads 0-29
span 28, datee 29-29
23. datee 26, ledd 0-23
dates lnads 15:22
sper
26. dales 26 leads 23-29
28, dates leads 29-29
span
Figure 6: CFSv2++ hyperparameters automatically selected for each target date in 2011-2020
Algorithm 2 CFSv2++
input test date t*; lead time ["; # issuance dates d"; span s; training set ground truth and CFSv2 forecasts (y,, f..T
initialize days per year D 365. 265.242199; # training years Y 12
S = (t E T -efr_difff
diff = - Y and _ifff 2
365
-I[(t"-1) mod DJ
365
2
$}
Form CFSv2 ensemble forecast across issuance dates and lead times E C for training and test dates t S {t*} do
f, = -fatafffff+-- .Ec)
output f,. + mean((y: fe)
B.3 Persistence++
Persistence++ was trained and tuned following the protocol described in [39]; see Algorithm 3. Figures to 10 display the learned Persistence++ regression weights for the final target date in 2020 for each of the four contiguous U.S. forecasting tasks. In each case, we observe significant spatial variation in the optimal weights used to combine lagged measurements, climatology, and CFSv2 ensemble forecasts.
18

---

Algorithm 3 Persistence++
input lead time 1"; training set ground truth, climatology, and dynamical forecasts (ye, Ct, JIE
initialize forecast period length L= 14
Form dynamical ensemble forecast across subseasonal lead times > for training dates E T do
f, mean (ft.())()
Combine ensemble forecast, climatology, and lagged measurements for grid points g = to G do
Bg argming Ever B¹ [1, Ct.g 1--1-- U2- 21 1. fo- 11
output coefficients (Bg)g
Lagged temperature, days Lagged temperature, days
Climatology
Offset
Figure 7: Spatial variation in Persistencet+ learned regression weights when forecasting temperature in weeks 3-4 for the final target date, December 23, 2020.
Lagged temperature, 43 days Lagged temperature, days
CFSv2 ensemble
Climatology
Offset
Figure 8: Spatial variation in Persistence++ learned regression weights when forecasting temperature
weeks 5-6 for the final target date, December 23, 2020.
19

---

Lagged precipitation, 29 Lagged precipitation, days
CFSv2 ensemble
Climatology
Offset
Figure Spatial variation Persistence++ learned regression weights when forecasting precipitation in weeks 3-4 for the final target date, December 23, 2020.
Lagged precipitation, 43 days Lagged precipitation, days
CFSv2 ensemble
Climatology
Offset
Figure 10: Spatial variation in Persistence++ learned regression weights when forecasting precipita- tion in weeks 5-6 for the final target date, December 23, 2020.
B.4 AutoKNN
The AutoKNN model of [23] part of a winning solution in the Subseasonal Climate Forecast Rodeo I [46] and was shown to outperform deep fully connected neural networks [19]. AutoKNN first identifies set of historical dates most similar the target date and then forecasts weighted locally linear combination of the anomalies measured on similar dates and recent dates. Our implementation matches that of [23] but adapts the model target our primary RMSE objective by (i) using mean (negative) RMSE as the similarity measure instead of mean skill, (ii) using raw measurement vectors Yt instead of anomaly vectors a,, and (iii) using equal datapoint weights the final local linear regression.
Training The k-nearest neighbors (KNN) step of AutoKNN identifies a set of historical dates most similar to the target date and while the autoregression step forecasts a weighted locally linear combination of the anomalies measured on similar dates and recent dates. For a given target date t" and lead time I*, the AutoKNN training set is restricted to data fully observable one day prior to
20

---

the issuance date, that is, to dates t -- -- where L = 14 represents the forecast period length.
Tuning All hyperparameters were the default values specified in [23].
B.5 Informer
The Informer [79], transformer-basee based deep learning model, was retrained every four months to predict temperature from past temperature and precipitation from past precipitation independently at each grid point.
Features For given grid point and target date t", the input features used to construct forecast are the lagged target variable observations from dates fia fus 1, fusss 2, 2. fias 95 for where
hast 1 - |* L represents the last complete observation prior to t and L 14 represents the forecast period length.
Training We divide the of target dates in 2-11-2020 into consecutive, non-overlapping four- month blocks and retrain the Informer model after every four-month block. For given lead time /*, grid point, and four-month block beginning with date
1. The training set is chosen to start at most 10, 000 days before (or the beginning of the training set, whichever is later) and then ends 301 days before t.
2. The validation set is chosen to start 300 days before and to end on the date prior to 1" -
3. We use early stopping with patience equal to three to determine when to stop the training: when we have three consecutive epochs + 1, 2, + 3 with validation loss no lower than that of epoch e, we terminate training and use the model at epoch e as the final trained model.
4. We use the trained model to generate forecasts for each target date in the four-month block.
Tuning We use the default Informer architecture and hyperparameters for univariate time series forecasting: the model has 3 encoder layers, 2 decoder layers, and has an 8-headed attention with 7-dimensional keys and feed-forward layers with 1024 hidden units, and has GeLU activations [20].
B.6 LocalBoosting
In recent subseasonal experiments of [19], boosted decision tree models yielded the best performance. Our boosted decision model, based CatBoost [51], uses features the value of 10 Subseason- aCClimateUSA variables in a geographic region around the target grid point. This gives the algorithm enough flexibility adapt the weights of the features each particular grid point while still taking into account neighboring spatial information. The geographic region is determined by bounding box degrees in each direction, and the 10 variables are chosen for each task via their predictive power on validation years.
Training For given target date t* and lead time 1*, the LocalBoosting training set T restricted to data fully observable one day prior the issuance date, that is, to dates t ts - --- I where L == 14 represents the forecast period length. LocalBoosting uses CatBoost [51] to regress, for each gridpoint and each date, the value of set of lagged weather variables in a geographic region around the gridpoint.
Tuning There are two hyperparameters to consider: (i) which lagged weather variables to use; and (ii) the number of neighborhood cells around gridpoint to define the geographic region. Bounding boxes of with side length of 2 or 3 cells were considered. Larger sizes were computationally infeasible. In each case, the 10 20 most important features in the SaabsassonalClimaeeUSS. dataset were considered. Here, features were chosen by their performance over 2001-2010 terms RMSE.
For each target date, LocalBoosting is run with the hyperparameter configuration that achieved the smallest mean RMSE over the preceding 3 years. See Figure 11 for visualization of the hyperparameters automatically selected for each target date in 2011-2020
21

---

temperature,
temperature,
Heet, St-tter, Metterr.----ppp... 20-0epth_2-rr....7
re Mr,ee 30 30-tter.5---dphhh..pm 20tp:500--ppth_.......3 2437
2013 207 2013 2014 2ets 2016 2017 2018 2019 2005
2013 201 2013 2014 2055 1014 2017 2018 2019
U.S. precipitation, weeks 3-4
U.S. precipitation, weeks 5-6
fe,2 30 Skotter, 2ol-apthh2-irr.....
Figure 11: LocalBoosting hyperparameters automatically selected for each target date in 2011-2020.
B.7 MultiLLR
The MultiLLR model of [23] was also part winning solution in the Subseasonal Climate Forecast Rodeo I [46] and has since been used to improve subseasonal precipitation prediction in China [67]. For each target date, MultiL.LR uses a backward stepwise procedure to select the most predictive features and then applies local linear regression for each grid point to combine those features into final prediction. Our implementation matches that [23] but adapts the model to target our primary RMSE objective by using mean (negative) RMSE instead of mean skill as the feature selection criterion. addition, we replace their MEI features with corresponding MELx2 features and their monthly dynamical forecast features with daily debiased CFSv2 forecasts.
Training For a given target date t* and lead time 1*, the MultiLLR training set restricted to data fully observable one day prior to the issuance date, that is, to dates / S 1" - l* T, where = 14 represents the forecast period length. The coarse-grained dynamical input feature nmme wcsccmm3 nasa of [23] was replaced with the daily debiased CFSv2 forecast features
2ftpss_sss_____pmmm___ 11tpp2mm14.555 14 shift15 and calbx_cfs22___ twtp:222mmmmmm0.mmmmm sdd_thi 551 for predicting
temperature at weeks 3-4,
subx_ tttps_ssss__ttmmmmmeme 5dd shiff22 and
for predicting
temperature weeks 5-6,
subx_cfsv22 cabxx_cfe2__preiip--r 14. 1etpp-1...d___fdtdaa1e 5d_shift15 and subxxcdfs22.prrcip-.........1.. for predict-
ing precipitation at weeks 3-4, and
2ttps__fsv2_ppecippppp .2.5d_._...... we__5ddshiffff shift29 and aubx__tfev2.precip-............ for predict- ing precipitation weeks 5-6.
Tuning All hyperparameters were set to the default values specified in [23], save for the tolerance parameter which was to 0.001 to accommodate the RMSE selection criterion.
B.8 N-BEATS
N-BEATS [48], neural network for time series data, is retrained N-BEATS every two months to predict temperature from past temperature and precipitation from past precipitation independently at each grid point.
Features For a given grid point and target date t*, the input features used to construct forecast are the lagged target variable observations from dates lastt liast 4, Maast 8,... Liast 48 for where flast t* 1* - L represents the last complete observation prior to t* and L = 14 represents the forecast period length.
Training We divide the set of target dates in 201 1-2020 into consecutive, non-overlapping two- month blocks and retrain the N-BEATS model after every two-month block. For given lead time l*. grid point, and two-month block beginning with date t*:
1. We train all dates t < t* -
22

---

2. For the initial two-month block, we train for 30 epochs.
3. For subsequent two-month blocks, initialize model weights to the learned weights from the prior block and then fine-tune for 8 epochs.
4. We use the trained model to generate forecasts for each target date in the two-month block.
Tuning We use the default N-BEATS architecture and hyperparameters for univariate time series forecasting [48]. The N-BEATS model has two stacks, where each stack is used to understand different patterns. Each stack consists of three blocks, which themselves are each comprised of six fully connected layers with ReLU activations. We used the Adam optimizer with learning rate 0.001 and batch size of 512.
B.9
Prophet
The Prophet model of [62] was one of the winning solutions in the Subseasonal Forecast Rodeo II [46]. Prophet is an additive regression model for time-series forecasting that predicts weekly and yearly seasonal trends on top of piecewise linear or logistic growth curve. We trained the model to predict each grid point independently with yearly seasonality enabled (to capture predictable whether trends) and weekly seasonality disabled.
Training Prophet takes as input a sequence of (univariate) time-series values and then predicts the next k dates from those, arbitrarily far in the future.
First, we split the problem into many univariate time-series prediction problems. When evaluating, we consider periods of 4 moths (e.g. January 2010 April 2010), train the model on all available historical temperatures (which may depend on the lead time) that grid point, and make predictions for that four month period. We then run this for all relevant periods of four months.
Tuning
The prophet model is trained in the univariate mode with yearly seasonality on (to capture predictable weather trends), weekly seasonality off (as weekends are unlikely to be special).
B.10 Salient 2.0
We developed the Salient 2.0 model based on Salient [59], a winning solution for the Subseasonal
Forecast Rodeo I [46]. Salient consists of an ensemble of ferd-forward fully-connected neural networks, using historical sea surface temperature (SST) data from 1990 to 2017 and an encoding of the day of the year as features. Salient's 's training protocol follows multi-task learning framework [56]. starts by training 50 randomly generated fully connected neural networks, each of which provides a prediction for the average temperature and accumulated precipitation at 3, 4, and 6 weeks ahead, at every grid cell. The forecasts are then obtained by combining the predictions for weeks and and for weeks 5 and 6. The final ensemble model forecasts correspond the mean of the top 10 ensemble members with the lowest validation error.
For Salient 2.0 in this work, the input features were augmented with geopotential heights different pressure levels (10, 100, 500 and 850 hPa) along with MEI and MJO indices. In addition, instead of training the ensemble on the whole of the 1990-2017 data, sequence of models was trained using data up until each of the years in our validation period of 2010-2020. These submodels with earlier training data cut-offss were then used generate hindcasts that informed the model's tuning decisions.
Salient 2.0 relies on two sources of sea surface temperature training data. The first data source is the weekly surface temperature from NOAA [54] and covers dates from 17 January 1990 to 02 February 2017. This dataset contains weekly data cenetered around Wednesdays and has a 1 x 1 resolution. It was re-gridded to a 4 x 4" using spline interpolation of order equal to 2, under the Python Scipy package [63].
For dates from February 2017 to present, second source of sea surface temperature data from the MET Office [30] is used. This dataset has a daily temporal resolution and is averaged to obtain weekly data centered around Wednesdays. This data is initially downloaded in a 0.25" x 0.25 spatial resolution and is re-gridded to a 4 4 A linear interpolation is then used fill any missing values (under the Python Scipy package [63]).
23

---

Training Salient 2.0 is an ensemble of 50 feed-forwart fully connected neural networks. All 50 neural networks are trained on pre-determined combination of input features, including weekly sea surface temperature, MEI, as well as the phase and amplitude features of MJO. The combinations of input features considered are:
d2wk_cop_sst surface temperature.
[2wk_cop_sst__ei_ surface temperature and ENSO,
c2vk__op_sstt_mjo sea surface temperature and MJO,
d2wk_cop_sst __t_i__e_____ mjj: sea surface temperature, ENSO and MJO.
In total, four ensemble models, corresponding to the four input feature combinations above, each including 50 neural networks, are trained. Each ensemble model is trained in rolling fashion, where the start year of the training dataset 1990 and the ensemble is trained up until each year in the range (2006, 2019], where model ending year y is used generate forecasts for target dates with year + 1.
For each of the 50 neural networks within given ensemble model, the input features can further be augmented using a time vector obtained by converting dates to a float representing the fraction of the year passed by that date. The addition of the time vector is decided by generating a random integer the range [0, 1], with corresponding to the addition of the time vector and otherwise. Additionally, for each of the 50 neural networks, the input feature vector for an individual training example consists
concatenation of the prior 10 weeks of data.
For each of the 50 neural networks within given ensemble model, the output consists of a prediction for the average temperature and accumulated precipitation 3, 4, and weeks ahead. The predictions for weeks 3 and 4 and the predictions for weeks 5 and 6 are combined separately, by averaging temperatures and summing precipitation. The top 10 neural networks with the lowest validation error are selected as the final ensemble members. The final predictions for each ensemble model correspond to the mean of the 10 selected ensemble members.
Tuning Each of the 50 neural networks within given ensemble model is trained using batch size equal to 128 and a train ratio equal to 0.89. In addition, each neural network's hyperparamaters randomly generated, with the number of epochs sampled in the range [100, 500], the number of layers in the range [3, 7]. and the number of units per layer in the range [100, 600).
At test time, for each target date, Salient 2.0 is run using the ensemble model that achieved the smallest mean RMSE over the preceding years. Figure 12 shows which ensemble models were used to generate predictions for which target dates.
U.S. temperature, weeks 3-4
U.S. temperature, weeks
d2wk cop
d2wk mojo
d2wk cop, sat mel
d2wk.cop, wow._coppssotmeee mojjomjj 2015 2013 2014 2015 2010 2019
2012 2013 2006 2017 2013
U.S. precipitation, weeks 3-4
U.S. precipitation, weeks 5-6
dZek ast
d2wk_cop cop_ mel
d2wk, mel
2011 2013 2013 2018
2015 2014 2027 2018 2018 2028
2013 2012
2012 2014
2019 2017 2018 2019 2019
Figure 12: Salient 2.0 hyperparameters automatically selected for each target date in 2011-2020.
B.11 Uniform Ensemble
consider two Uniform Ensemble models, where the ensemble produced using a set of models
C' as input:
24

---

1. Uniform ABC: C. = Climatoloyy++, CFSv2++, Perssstence** }
2. Uniform ABC + Learning: C C= Climatology++, CFSv2++, Persistence++, LocalBoosting, MultiLLR, AutoKNN, Prophet, Salient 2.0 ). the ABC models plus all learning models save the very low performing Informer and N-BEATSS models.
Uniform ensemble forecasts are produced as the uniform or unweighted average of imput models. Letting Xt.c be the forecast made by model E C' on target date t, we produce the forecast:
y1-
C Leec-Xe Xt,
B.12 Online Ensemble
We consider two Online Ensemble models, where the ensemble is produced using set of models C as input:
1. Online ABC: C = (Climatology++ CFSv2++, Persistence+*]
2. Online ABC + Learning: C' = Climatology++, CFSv2++, Persistence+*, LocalBoosting, MultiLLR, AutoKNN, Prophet, Salient 2.0 }. the ABC models plus all learning models save the very low performing Informer and N-BEATS models.
To learn a time-dependent adaptive ensemble weight wt. we employ the online learning method presented in [15]. We applied the AdaHedgeD algorithm with the recommended recent_g optimism setting. The algorithm was run with delay parameter D = for the 3-4 weeks horizon tasks and D = 3 for the 5-6 weeks horizon tasks. We ran the online learning algorithm over the full set of target dates 7 = 520, without performing the yearly resetting suggested the original implementation. The learner optimized the RMSE loss over gridpoints in the region of interest, described in the experimental details of [15].
Online ensemble forecasts are produced as the weighted average of input models, with weight w, determined by the online learning algorithm. Letting Xt,c be the forecast made by model E C on target date t and w, be the weights produced by AdaHedgeD, we produce the forecast: ÿ: EeEc Wt.e * Xt,e-
B.13 Debiased ECMWF
We implement the operational ECMWF bias correction protocol detailed in [69]. For each target forecast date, we debias both our ECMWF control and ensemble forecasts using the last 20 years of reforecasts with dates within 16 days from the target forecast date. The average of the control and 10 ensemble reforecasts on the 1.5x ..5 degree grid are used for debiasing.
C Supplementary Results
C.1 Percentage Improvement over Meteorological Baselines
To highlight the improvement of individual ABC models over their traditional counterparts, Figures 13 and 14 show the percentage RMSE improvements of Climatology++ CFSv2++, and Persistence++ relative to their respective baselines Climatology, debiased CFSv2, and Persistence by season and by year.
For all four tasks, the ABC models are consistently better across seasons and years. The result is particularly striking for Persistence+* and highlights the value of integrating lagged measurements, numerical weather prediction, and climatology. Figures 13 and 14 show the per season and per year improvement of each ABC model over its corresponding baseline across the contiguous U.S. and the years 2011-2020 Note the learned ABC benchmarks yield consistent improvements in mean RMSE.
C.2 Yearly Percentage Improvement over Mean Debiased CFSv2 RMSE
Tables 3 and 4 present the yearly improvement of each model over debiased CFSv2, as measured by mean RMSE across the contiguous U.S. in the years 2011-2020.
25

---

bigelte I0h55
80%
Climateoayy
12%
Deblased CFSv2
U.S. temp. weaks 3-4
U.S. temp. weeks 5-6
U.S. precips woeks 3-4
60%
58%
U.S. precips weeks 5-4
50%
30%
20%
1%
Peraistunce
Dec Feb
Mar May
Jun Aug
Sep Nov
Fab
Mar May
Jun Aug
Sep- Nov
Figure 13: Per season improvement of each ABC model over its corresponding baseline across the contiguous U.S. and the years 2011-2020. The learned ABC benchmarks yield consistent improvements in mean RMSE.
improvement over mean basaine BM5E
9%
Climatology-+
Persistencet*
U.S. temp. wenks 2-1
Climatoloyy
Debiased CFSv2
70%
Persistence
8%
U.S. temp. wents 5-5
10%
U.S. precis weeks 3-4
U.S. precp weeks 5-6
50%
9%
30%
2%
10%
%
2011
2012
2013
2014
2015
2016
2017
2014
2015
2020
2011
2012
2013
2014
2013
2016
2010
2016
2020
2011
2012
2013
2014
2015
2017
2010
2018
2020
Figure 14: Per year improvement of each ABC model over its corresponding baseline across the con-
tiguous U.S. and the years 2011 1-2020. The learned ABC benchmarks yield consistent improvements in mean RMSE.
Table 3: Percentage improvement over mean debiased CFSv2 RMSE when forecasting temperature in the contiguous U.S. The best performing models within each class of models shown in bold, while the best performing models overall are shown in green.
Imqpeder Swrts
2012
2013
57
-11
-1154941012
-15556:79
166602211
-11L1543555
A3C
$173370
100330
13504435
-001115
1230334
ADEEM
2W013
312633
131--339
45-126
Lowvig
AARKN
03354455
ATIESW
135530
1400414
0151422
-7811441
1311300
191144
-70
$39.407
1253445
1971415
1351126
-108113
-29061779
-135-177
ABC
5511326
68113
$2150
25.355
135
130.316
6771245
Backmn
13:-33
4011300
435-4
10-13
23135122
-180312300
ABC
1284312
12:43
872-239
2574241
Lunka
040-300
125.125
-003550
-12274502
221-331
334419
02845N
1754100
143400
63-133
136-126
CLd ABC
3154335
001-1C
(106200)
41--316
K10-331
20663%
140000
121-09
26

---

Table 4: Percentage improvement over mean debiased CFSv2 RMSE when forecasting precipitation the contiguous U.S. The best performing models within each class models are shown in bold, while the best performing models overall are shown in green.
-2-25.440
-18151131
-2637-3.05
-127..53
-21A535.72
-2185:446
-2009JA1
-2627-14
ABC
Chraskeg
6551137
1640.15
$31-14
1655-131
531100
11:11111
1L171100
635:1227
636.136
14511-0
CSUAN
754187
1421145
11204172
AT74124
S0IJAS)
aig
A394511
01114134
135-192
15722-
456-18
Proket
68614
A16136
8241.66
1638-131
654138
65-136
1
1048-127
2004128
1.264.2
Dadiac Chrasler
WOMS
008927-14
35
3001439
-4 1166510
M0N 51
3438446
10 11S10
6464133
8.8-14
18216171
2054133
+334005
12h612
Leurigg
5854130
0038214136
754144
637-141
055-15
5084133
2351109
1566 38
123,63
1201111
121113)
11361100
-34000
13
Okilee ABC
18915
1186-135
175
60140.15
27

---

C.3 Yearly Average Skill
Tables 5 and 6 present the yearly average skill of each model across the contiguous U.S. in the years 2011-2020.
Table 5: Average percentage skill when forecasting temperature in the contiguous U.S. The best performing models within each group are shown in bold, while the best performing models overall shown in green.
Temperature weeks 3-4
Guoup
Model
2013
2014
2015
Oraall
Baseliazs
Dds. CF5
33.26:5.13
26.6
17.3245.58
37.815.85
35.73:5.71
33.01:577
13.36:592
19:42:687
18.921112
24114:543
2,
Pasiicene
27.26:5:55
2.1627.00
17.9627.15 1S
2014127.15 2.15
7.557 36
3.786:7.4
4,39:27244
18.443499
10.1977.22
1064:2231
LTSy2-4
34.6:4221
43,212563
136715337
32.815411
37.3613318
411614799
2.:1::229
2693118
50.721.91
32:2:1244
Pesserecese
41.21:3356
47.39:436
124411..1
21:91577
57,641:499
41 41.0013.11
24.5::455
32.44:6166
18512601
015315.51
324:1722
Lomig
Localliowning
10.51
17.51 5.14
K121599
155153:
31881401
12
1991
18:4 14.00
18
17
14:4-111
3485 5.15
31,03 535
9.571475
22715594
29377 5592
21911.557
1889 1.1
30.6 1876
541611631
26.5 1.5
Peoplet
18:31 4:31
14,111505
14614.1
16516441
22.741 5J13
28A
35.26 5.34
35.54-34
20-2111:1
EaserNes
Unifemn ABC
48.16:44
12:4555
90.6-5.18
43:33-515
2085-196
70. 1346366
36313559
37.54:566
31.5--171
Odline ABC
359854449
45115559
1254513
31.545..6
38.775.11
44001-774
20.77
29.35:646
31.64531
56 36.215991
33.2561.72
werke
Bacliazs
Des CF9
4344:482
3689:593
1.513
15484622
25.721533 32
431:64
1582:6636
847:2600
19112134
Pasidcees
20220:593
04475
1244:9244
33922743
6.156675
14556220
ABC
Limaskggy
115-.55
12.812534
74R1838
29.44166
164511822
22.79::511
15.0915.35
12.99 12.99:5377
56. 36.01133
1887
L3Sy2-1
31111:225
352115227
K611491
343515.07
867216811
41.615112
197159
28.5-176
Peniernezce
3424-4.19
4:33
K454419
18.115555
561615507
16:26:499
25.85-581
1845155
41.9-771
26.71-111
AuaKAN
(71-461
-6.12-4775
K06
581466
11451:491
6:2814774
3-97
5.21-1400 16.7
19401.19
20 271151---11
3.55-111 1.13
Lecallionting
8.1 1.64
1577 15:1
12614
22,01 4.12
7.84 5:39
19:47
15
22577-444
17.00 1.44
19.99
23,83 536
35S1AS
17:7114.66
13216.73
12:08
13.08 15:10
14.44145 1911
238116.6
7.12-5.34
-1.751545
K14166
-5.34.60)
13991563
-4716646
21.91.1.66
38.29 6,11
15.9616.17
11:71-2001
TosmMee
Usikm ABC
36.14-411
41:1:48
K461811
78474475
191-542
41:03--:44
1740-5448
26-82-583
21.86-88888
41-18-4688
30.33-177
Table 6: Average percentage skill when forecasting precipitation in the contiguous U.S. The best performing models within each group are shown in bold, while the best performing models overall are shown green.
Preciptationn weels 3-4
Grorp
Medi
2011
2013
2013
2014
2015
2016
2019
3001
Owandl
Bautines
Deb CPSV2
1526-13
889--150
5-6--722
5101-108
3001-134
2.09-361
0356-005
401-3:44
1.56-100
7.194-776 76
5,71-1111
Perderace
1436--00
9.61-368
1.95-3.07
6.66-3324
723-343
1.42285
13:6
6.72-1.76
5.41 5.27
8,31-
Chiepslogee
C7Sx2++
24.86134
19:08:188
1744:311
16181:186
708:3.5
124255.06
1475:3775
13442:3.55
13:8:4111
23.6642.74
16:11:144
21.9:2217
14.5-2235
1543:2.96
13912.28 26
9.38-3446
K.12226
5:1.3.59
13.53-2305
12.99:2224
17.48422 94
13,59.0.99
Lonnn
AutKKNN
13.5
13:66 411
361418
5111-437
10011.00
5.29-494
1-32
7.5611335
Muhdd.
12.44:1.112 12
487 1.46
10 18711554
1.97
15
8499
7.81-2.11 2.41
5.56
1.25
16.161223 25
4,8.1101
Proples
2.77
1194
14.292211
18812114
53511313
13.09:2.6
11:1-1344
13.10 3.70
18.77
1.45
20.111244
13.51 6.
Sabeni
19.814.21
20:38-135
11.514005
595.5.5
3015.11
5.611331
2.44
4.19
15:991335
10.11:1.31
TosemMte
Unifonn ABC
2672-3.12
213-766
169-333
19:0--17
11:11:3.77
15466-296
1500-335
16.29-3.25
14.96 3.73
25.16-26
1534-6399
Ocline ABC
16754-311
2145-265
1876-32
2000-265
11-44-343
1829-3.11
15:05-386
16 16:90-1.5
14.3--37
150542.0 09
1881-103
Bowlines
Dat CPNk
11775-316
635-3711
-1 3.333
09665 16
2-77-154
424-121
-131-115
8 775-17
424-1006
Penderece
685-3887
6.9-141
12.16-366
432-13
383-331
1
189-33
621-129
532-331
13041277
7.1--10
Cinndlggee
12.2433.1
3411:35
13.31221
1567-130
3975339
17.00:3.04
1366:364
3.2m2.11
13664267
1499:106
CFS+2++
1311=333
18 1875--88
23.87-2.6
14
88622.9 99
W435-333
13.465.44
7.56-431
1534:321
134:37
1611:114
17
999-223
169-2.61
7.6622766
244:3336
3.852253
36::354
11 11:24:2.42
T.662.6
163662255 25
9.77-0.49
123353.5
16.31:415
11.15:3.36
6.553330 X2
331=504
50124.13
1,502506
7.5813.51
Lacalliwoing
08112 96
9.26 1.34
67127
11.55 2.7%
10:31 10.91-2.11
16.512 65
$77.106
MahddE
12.42115.58
1.9-1311
1049:2.7
2.561:99
06514003
14211112
8996-.75 3.75
1.55
15:11..77 97
7.9-1103
Prophc
17.92
14:4-222 2242
15.6
5.41.31
13.41261
10101-145
1108 1.74
18.75
19.9412.46
8:91
Eiserrees Unifien ABC
25.11-3.111
2300--744
25445-2.52
17 80
9.17-34
16.7-3347
11:31-10
14:8-348
13:01-3.44
25.34-2.37
18.35-4435
Oeline ABC
35336-37
21.32-2.7 2.73
71.85-2.66
18.11-2.44
7.6-33.9
17.29:35
12.68-334
16-2--111
13.86-3.
1735--107
28

---

C.4
Spatial Improvement over Mean Debiased CFSv2 RMSE
Figure 15 presents the spatial improvement of each model over debiased CFSv2 when predicting U.S. temperature across 2011 1-200. For both the weeks 3-4 and the weeks 5-6 lead times. the ABC models uniformly improve over debiased CFSv2 and outperform both the baseline models and the state-of-the-ar learning methods. The best overall performance at each lead time is obtained by the Online ABC ensemble.
DS
Figure 15: Percentage improvement over mean debiased CFSv2 RMSE when forecasting temperature in the contiguous U.S. over 2011 1-2020. White grid points indicate negative or 0% improvement, and the mean percentage improvement given parentheses.
Figure 16 displays the spatial improvement of each model over debiased CFSv2 when forecasting U.S. precipitation across 201 1-2020. For precipitation, all models exhibit larger gains over debiased CFSv2, and all models, including Climatology, achieve larger improvements in the Western U.S. than in the Eastern U.S.
Figure 16: Percentage improvement over mean debiased CFSv2 RMSE when forecasting precipitation in the contiguous U.S. over 1-2220. White grid points indicate negative or 0% improvement, and the mean percentage improvement given parentheses.
29

---

C,5 Spatial Bias Maps
This section explores the spatial bias (the mean forecast minus the mean observation at each grid point in the contiguous U.S.) of each model across 201 1-2020. The temperature maps of Figure 17 indicate a cold bias for most models over the southern half of the U.S. and an additional warm bias for several models in the center north. This warm bias is particularly pronounced for Salient 2.0. In precipitation maps of Figure 18, all models Salient 2.0 and AutoKNN show wet biases in the western half of the U.S. and dry biases in the eastern half. AutoKNN exhibits dry bias extending from the Eastern U.S. include the Northern U.S. as well, while Salient 2.0 displays a strong dry bias across the entire contiguous U.S. that is especially pronounced in the eastern half.
The Prophet model is noticeably less biased than the other evaluated models; however, this bias reduction does not immediately translate into improved performance, as Prophet is outperformed by ABC models with larger bias in all four tasks. This indicates that Prophet's reduced bias comes at cost of unnecessarily high variance relative to the dominating ABC model forecasts.
Figure 17: Model bias when forecasting temperature in the contiguous U.S. over 201 1-2000.
Leiioan ARC
Figure 18: Model bias when forecasting precipitation in the contiguous U.S over 2011-2020.
C.6
GraphCast Comparison Details
To generate weeks 3-4 GraphCast forecast of temperature and precipitation for given target date, we
1. downloaded the pretrained GraphCast ERA5 1979-2017 resolution 0.25 pressure levels 37 mesh 2to6 precipitation input and output model weights from https: console. cloud. woo.l. wtw.storaaee////coooooooooo
https/ttrrrreebeeeeeeeeeoee dm dm_grphccaattt.mmmmmmmmmm
initialized the model with the three days of ECMWF Reanalysis v5 (ERA5) data preceding the as- sociated issuance date, as described in https://githu: cwm com//oogle hwwp///ooooedeemind////eeeeeeeee graphcast,
repeatedly ran the model (using autoregressive rollout as in https://githhee com/ grappcastt/loob/main. wwwaaiiinggaaaast..mmmm seapphasst_deooo.oo ipynb) for the maximum sup- ported number of time steps (12) and reinitialized the model with its own forecasted output until 30 days of forecasts had been produced,
30

---

4. aggregated the 2e-terper and total_precipptatiion. 6hr forecasts over days 14-28,
5. regridded the aggregated forecasts to standard x 1 grid using xarray interp_ like [22]. and
6. converted temperature to degrees Celsius and precipitation to millimeters.
Since GraphCast was trained on data through 2017, Table summarizes the performance of GraphCast across the post-training years 2018-2020. Similarly to the deep learning models previously evaluated in Table I, GraphCast outperforms debiased CFSv2 in terms of skill, underperforms debiased CFSv2 in terms of RMSE, and strongly underperforms the ABC ensemble models in both metrics when forecasting either temperature or precipitation in weeks 3-4.
Table Average percentage skill and percentage improvement over mean debiased CFSv2 RMSE across 2018 8-2020 in the contiguous U.S. along with 95% bootstrap confidence interval. The best performing model overall is shown green.
RMSE % IMPROVEMENT
AVERAGE % SKILL
WEEKS 3-4
GROUP
MODEL
TEMPERATURE
PRECIPITATION
TEMPERATURE
PRECIPITATION
BASELINES
DEB. CFSV2
19.98+4.84
3.12+2.99
LEARNING
GRAPHCAST
-3..9617.46
-34.2617.57
20.17:4.70
5.55+2.81
ENSEMBLES
UNIFORM ABC
9.70112.19
8.4111.18
34.3714.58
18.3411.599
ONLINE ABC
9.37+2.07
8.37
34.6714.71
18.07:2.76
Figure 19 displays the spatial improvement of GraphCast over debiased CFSv2 when forecasting U.S. temperature and precipitation across 2018-2020. Unlike the ensemble ABC models, GraphCast seldom improves over the debiased CFSv2 baseline.
U.S. Temperature, weeks 3-4
GraphCast (-35.71%)
Uniform ABC (10.11%)
Online ABC (9.42%)
30%
25%
20%
15%
10%
0%
U.S. Precipitation, weeks 3-4
GraphCast (-59.62%)
Uniform ABC (9.79%)
Online ABC (9.77%)
30%
25%
20%
15%
10%
5%
0%
Figure 19: Percentage improvement over mean debiased CFSv2 RMSE when forecasting temperature or precipitation the contiguous U.S. over 2018-2020. White grid points indicate negative or 0% improvement, and the mean percentage improvement is given in parentheses.
Figure 20 compares the spatial bias (the mean forecast minus the mean observation at each grid point in the contiguous U.S.) of GraphCast versus the ABC ensemble models across 2018-2020. For
31

---

precipitation, GraphCast has substantially drier bias than the ABC models throughout most of the contiguous U.S. For temperature, GraphCast has warmer bias in the West.
U.S. Temperature, weeks 3-4
GraphCast
Uniform ABC
Online ABC
15%
10%
5%
0%
-5%
-10%
-15%
U.S. Precipitation, weeks 3-4
GraphCast
Uniform ABC
Online ABC
15%
10%
5%
-5%
-10%
-15%
Figure 20: Bias of GraphCast vs. ABC ensembles when forecasting temperature or precipitation in the contiguous U.S. over 2010-2020.
C.7 Western U.S. Competition Results
Table 8 displays the performance improvement over the baseline debiased CFSv2 in the Western U.S., over dates 2019-2020. This region was the focus of the Subseasonal Forecast Rodeo and include the performance of the three best contestants in the table. The ABC ensembles were either very competitive or better than the other contestants.
Table 8: Percentage improvement
debiased CFSv2 RMSE over 26 contest dates (2019- 2020) in the Western U.S. The hest performing models within each class of models are shown in bold, while the best performing models overall are shown in green.
Group
Model
Temp. weeks
Temp.
Precip. weeks
Precip. weeks
haselines
Salient
Climatology
10.22
-0.76
5.82
2.25
Contestants
1st
place
17.12
8.47
11.54
8,63
2nd place
16.67
7.04
11.10
8.03
place
15.47
6.90
10.62
7.94
Learning
AutoKNN
13.09
2.90
LocalBoosting
12.85
4.09
7.25
3.71
MultiLLR
9.54
1.12
8.95
4.58
Prophet
15.68
3.40
Salient 2.0
11.15
2.91
12.65
8.56
ABC
Climatology+*
15.54
6.43
8.35
4.69
CFSv2++
6.67
9.26
5.51
Persistencet+
16.59
8:27
8.20
4.51
Ensembles
Uniform ABC Uniform ABC Learning
14,96
9.58
9.31
5.89
15.89
8.79
10.43
6.79
Online ABC
16.71
8.70
8.85
5.19
Online ABC Learning
14,70
7.97
12.52
8.18
32

---

C.8 Salient 2.0 Dry Bias
We hypothesized that the exceptional Western U.S. contest performance of Salient 2.0 in Table 8 was due in part to the dry bias observed in Figure 18, as 2020 was unusually dry year in the Western U.S. To explore this hypothesis, we focus on forecasting precipitation weeks 3-4 in the Western U.S. and display in Figure 21 (left) the percentage improvement of Salient 2.0 and CFSv2++ over debiased CFSv2 alongside the inverse total precipitation each year 2011-2020. steep decrease in cumulative precipitation for 2020) is accompanied by steep increase in Salient 2.0 predictive accuracy. In addition, the rises and falls total precipitation track the accuracy of Salient 2.0 well but appear largely unassociated with the performance curve of other accurate models like CFSv2++, The scatter plots and best-fit lines of Figure 21 (right) paint similar picture. Salient 2.0 exhibits a distinctly negative correlation between percentage improvement and total precipitation in the Western U.S., while this relationship is absent for other accurate models like CFSv2++.
anticipated, the
20%
Westem U.S. precipitation, weeks 3-4
Improvveent over mean dab. RMSE
20%
Western U.S. precipitation, weeks 3-4
Dubiased
CFSv2
18%
precipitation
2.3e-06
Sallent 2.0
15%
Improvvenn Improvament over maan deb.
2.0e-06
12%
19c.06
10%
1.0%
8%
8%
1.Be-06
5%
2%
2%
1.6e-06
05
6.2ex05
5.4e105
2012
2012
2014
2015
2016
2017
2018
2010
2030
Total precipitation
Figure 21: Temporal plot (left) and scatter plot (right) of yearly total precipitation and percentage improvement over mean debiased CFSv2 RMSE in the Western U.S. across 1-2020.
The developers of the Rodeo I Salient model attribute this dry bias to the log-normal distribution of precipitation, which leads to more frequent anomalously dry conditions than anomalously wet conditions and in turn encourages drier model forecasts [58].
More recent versions the Salient model mitigate this bias by training on seasonal anomalies in place of raw temperature and precipitation values [58].
33


Reviews:
[{"review_id": "L3HOtdqe2p", "reviewer": "Reviewer_sMGZ", "summary": "", "questions": "", "limitations": "My main concern is a more detailed discussion of why the ABC methods work best (and how they actually work so well) plus additional info on the interpolation methods to create the dataset. Otherwise, the limitations are discussed well in a separate subsection.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "See also above - strong contribution on an important topic, where ML has great potential. This is a task that should be tackled in tandem by meteorologists and MLers and this dataset will help both communities to streamline their work. Suggested extensions to be build on this paper show a clear path forward to extend the dataset to global scale.\n\nThe work is also highly valuable from the ethical and social implications standpoints.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yf9AcxQxqm", "reviewer": "Reviewer_UtoY", "summary": "", "questions": "", "limitations": "The details of the dataset and limitations are adequately discussed.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Subseasonal forecasting is critical for a number of social and economic areas, with significant benefits to improved forecasts.\n2. This is a comprehensive dataset which aggregates data from a variety of sources.\n3. A large number of benchmarks are conducted, as is a comparison with SOTA competition winners.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EhZdO4mWVp", "reviewer": "Reviewer_kSL6", "summary": "", "questions": "", "limitations": "- The authors talk about resolution limitations as well as region limitations in the paper", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength is in the introduction of a subseasonal dataset that focuses on longer time scales unlike current benchmarks that focus on medium-range weather prediction (up to 2 weeks) and the evaluation of models on this dataset. Demonstrating skill on this longer timescale is very important to advance ML models for weather/climate and hence this dataset can prove to be very useful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "CGylpIER5L", "reviewer": "Reviewer_SYV5", "summary": "", "questions": "", "limitations": "1. Major concern for me is the lack of novelty of this new dataset in comparison with the SubseasonalRodeo dataset. It is incremental work.\n2. With a little web search I stumbled upon an earlier version of this paper (https://arxiv.org/pdf/2109.10399v1.pdf) and I noticed that the benchmark section in the current submission does not include any new benchmarks compared to the earlier version of this paper which was submitted 2 years back.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Dataset covers seven meteorological data sources for collecting the data and provides machine-learning-ready Python Pandas DataFrames.\n2. The coverage is over the contiguous USA which is an improvement over the SubseasonalRodeo dataset.\n3. Clear description of the dataset and any data augmentations.\n4. Extensive experiments covering several benchmarks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a dataset on subseasonal weather prediction utilizing observation data and physics-based dynamical models. The main contribution is aggregating data from diverse sources, formats, and time and spatial resolution. It is an important step for developing effective forecasting models. It is unclear how we evaluate the quality of the dataset - as mentioned in the review, the resolution could be too coarse for some important studies. The authors are encouraged to address the challenge in the final version.
Decision: Accept (Poster)