SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Soukayna Mouatadid<sup>1</sup>, Paulo Orenstein<sup>2</sup>, Genevieve Flaspohler<sup>3</sup>, Miruna Oprescu<sup>4</sup>, Judah Cohen<sup>3</sup>, Franklyn Wang<sup>5</sup>, Sean Knight<sup>3</sup>, Maria Geogdzhayeva<sup>3</sup>, Sam Levang<sup>6</sup>, Ernest Fraenkel<sup>3</sup>, Lester Mackey<sup>4</sup>
<sup>1</sup>University of Toronto, <sup>2</sup>IMPA, <sup>3</sup>MIT, <sup>4</sup>Microsoft Research, <sup>5</sup>Harvard, <sup>6</sup>Salient Predictions
Subseasonal forecasting of the weather two to six weeks in advance is critical for resource allocation and advance disaster notice but poses many challenges for the forecasting community. At this forecast horizon, physics-based dynamical models have limited skill, and the targets for prediction depend in a complex manner on both local weather and global climate variables. Recently, machine learning methods have shown promise in advancing the state of the art but only at the cost of complex data curation, integrating expert knowledge with aggregation across multiple relevant data sources, file formats, and temporal and spatial resolutions. To streamline this process and accelerate future development, we introduce SubseasonalClimateUSA, a curated dataset for training and benchmarking subseasonal forecasting models in the United States. We use this dataset to benchmark a diverse suite of subseasonal models, including operational dynamical models, classical meteorological baselines, and ten state-of-the-art machine learning and deep learning-based methods from the literature. Overall, our benchmarks suggest simple and effective ways to extend the accuracy of current operational models. SubseasonalClimateUSA is regularly updated and accessible via the https://github.com/microsoft/subseasonal_data/ Python package.
Weather and climate forecasting are fundamental scientific problems with many applications, including agriculture, energy grids, transportation and disaster prevention [36, 71, 55]. Indeed, short-term and long-term operational forecasts are critically employed in many sectors of our society and the world economy. However, skillful forecasts for the subseasonal regime—that is, 2 to 6 weeks ahead—still present operational challenges due to the chaotic nature of the weather [32] and to the interaction of weather and climate variables operating at different spatial and temporal scales [66].
In recent years, these challenges have spurred intense activity from both the meteorological and machine learning communities. On the one hand, steady advances have extended the reach of physics-based dynamical models of the atmosphere and oceans into the subseasonal realm [64, 50, 29]. On the other, parallel efforts from the machine learning community have led to improved predictive skill through new models trained on historical observational data and dynamical model forecasts [31, 8, 23, 2, 19, 78, 67-69, 60, 39].
Nevertheless, developing and benchmarking new subseasonal models remains challenging due to a lack of standardized, curated datasets targeting this forecast horizon. The data necessary for subseasonal predictions are often collected from multiple data sources, each with its own data processing pipeline, and then standardized into a common spatial and temporal resolution. Because there is uncertainty regarding the main drivers of subseasonal phenomena, domain experts are typically needed to determine which features are likely to carry signal, and different developers
employ different aggregation techniques over time and space. Finally, the changing nature of weather makes the forecasting task hard to compare across different regions and years. This, in turn, has spurred several subseasonal forecasting challenges to benchmark existing solutions [47, 46, 65].
In this paper, we introduce SubseasonalClimateUSA, a diverse collection of ground-truth measurements and dynamical forecasts for subseasonal prediction over the contiguous United States (U.S.). We include spatiotemporal measurements with known subseasonal impact (including, e.g., temperature, precipitation, sea surface temperature, and geopotential height); the states of known subseasonal drivers such as El Niño-Southern Oscillation (ENSO) and the Madden-Julian oscillation (MJO); and dynamical predictions for temperature and precipitation from eight operational models including the U.S. Climate Forecast System version 2 (CFSv2) and the leading subseasonal model from the European Centre for Medium-Range Weather Forecasts (ECMWF). The dataset is regularly updated and accessible via the open-source subseasonal_data Python package for easy retrieval.
We then use the SubseasonalClimateUSA data to benchmark a wide range of subseasonal models, highlighting their strengths and weaknesses. These models include traditional meteorological benchmarks (e.g., Persistence and Climatology), operational dynamical models (e.g., CFSv2 and ECMWF), and ten state-of-the-art deep learning (e.g., N-BEATS [48] and Informer [79]) and machine learning (e.g., CFSv2++ [39] and Prophet [62]) forecasters. Two of these models (Salient 2.0, the best-performing deep learning method, and LocalBoosting) were new creations of this work, and nine required new subseasonal forecasting implementations now available via the subseasonal_toolkit Python package. Model performance is measured for four standard subseasonal tasks: predicting average temperature 3-4 and 5-6 weeks ahead and predicting accumulated precipitation 3-4 and 5-6 weeks ahead. They are evaluated in terms of accuracy (measured by spatial root mean squared error) and skill (measured by uncentered anomaly correlation), over the years 2011-2020. Overall, we find that the simplest learned models typically outperform the meteorological baselines, the leading operational models, and the remaining learning methods. Additionally, we show that ensembling different methods through online learning leads to further gains in terms of both accuracy and skill.
Our aims in releasing SubseasonalClimateUSA are twofold. First, we aim to facilitate the development of skillful learning-based subseasonal forecasting models by providing a comprehensive, standardized, and machine-learning-friendly dataset. Second, we aim to provide standardized benchmarks for subseasonal forecasting progress. To this end, we define four core subseasonal prediction tasks that users can use as benchmarking targets: forecasting (i) temperature in weeks 3-4, (ii) temperature in weeks 5-6, (iii) precipitation in weeks 3-4, and (iv) precipitation in weeks 5-6. Significant advances in any of these tasks would have significant implications for the allocation of water resources, agricultural production, and disaster relief [47, 71].
Related Work There are several datasets available for benchmarking weather models. For example, both the National Oceanic and Atmospheric Administration (NOAA) and the ECMWF provide reanalysis datasets tracking weather variables from the whole globe from the 1940s until today [24, 21], and global model simulations can be found in datasets provided by the World Climate Research Programme [13] and ECMWF [6]. More recently, several new datasets have been made available targeting specific AI applications in weather. For instance, classifying clouds [53], studying storm morphology [18] and nowcasting [16], predicting tropical cyclone intensity [35] and air quality metrics [4], and analyzing watershed-scale hydrometeorological time series [1] and river flows [17]. There are also more general-purpose datasets, such as WeatherBench [52], which provides a benchmark for forecasting different medium-range weather variables 3 to 5 days out. For a general overview of weather datasets for machine learning, see [11].
While these datasets have helped advance weather prediction in different tasks, there are no general datasets specifically targeting the subseasonal scale for the U.S. There have been instead several competitions targeting this lead time including the U.S. Bureau of Reclamation (USBR) Sub-Seasonal Climate Forecast Rodeos [47, 46] and the World Meteorological Organization Seasonal-to-Subseasonal (S2S) Artificial Intelligence (AI) Challenge [65]. Furthermore, the SubX Experiment [50] also makes a series of subseasonal models available for benchmarking (which are included in the SubseasonalClimateUSA dataset). Finally, the precursor of this work, the SubseasonalRodeo dataset of [23], targets only the Western U.S., offers only a static data snapshot ending in 2018, provides no forecasts from the leading subseasonal dynamical model (ECMWF), and includes only coarse-grained (monthly) forecasts from the North American Multi-Model Ensemble, with limited utility for weekly or biweekly forecasting.
In contrast, SubseasonalClimateUSA is a modern, regularly updated resource targeting the contiguous U.S. with granular (daily and subweekly) forecasts from ECMWF and seven other operational dynamical models in the SubX consortium [25]. Notably, both the present work and past studies have found complementary predictive signals in physics-based dynamical model forecasts and pure observational data that can lead to better forecasts than either data source alone [see, e.g., 23, 39]. In fact, recent work has demonstrated that even the least skillful operational dynamical models can produce forecasts with skill comparable to the best when corrected suitably with observational data [39]. As a result, we have endeavored to include both granular measurements and granular model forecasts in the SubseasonalClimateUSA dataset to best equip future model developers, researchers, and forecasters.
2 The SubseasonalClimateUSA dataset
The SubseasonalClimateUSA dataset houses a diverse collection of ground-truth measurements and dynamical model forecasts relevant to forecasting at subseasonal timescales. The dataset is regularly updated, CC BY 4.0 licensed, accessible via the open-source subseasonal_data Python package, and documented at the URL https://github.com/microsoft/subseasonal_data/blob/main/DATA.md. We summarize dataset contents, sources, and processing steps below and provide supplementary details in Appendix A.
Data Collection and Processing Figure 1 summarizes the Subseasonal ClimateUSA data collection and processing pipeline. The pipeline collects raw data from seven meteorological data sources (contributing different variables, resolutions, and file formats), passes all data through a common preprocessing pipeline, and outputs a standardized collection of machine-learning-ready Python Pandas DataFrames and Series objects stored in HDF5 format. Each file contributes data variables falling into one of three categories: (i) spatial (varying with the target grid point but not the target date); (ii) temporal (varying with the target date but not the target grid point); (iii) spatiotemporal (varying with both the target grid point and the target date). Data representing ground-truth measurements are typically downloaded daily on  $0.25^{\circ}$  or  $0.5^{\circ}$  latitude-longitude grids. However, subseasonal forecasts are typically issued on coarser  $1^{\circ}$  or  $1.5^{\circ}$  grids and averaged over two-week periods [47, 46, 65]. As a result, unless otherwise noted below, temporal and spatiotemporal variables arising from daily data sources were derived by averaging input values over a 14-day rolling window, and spatial and spatiotemporal variables were derived by interpolating input data to a  $1^{\circ}$  latitude-longitude grid and retaining only the grid points belonging to the contiguous U.S. To accommodate ECMWF forecasts, which were only made available on a  $1.5^{\circ}$  latitude-longitude grid [12], we additionally download SubX forecasts at  $1.5^{\circ}$  resolution and interpolate temperature and precipitation onto the same grid.
Figure 1: Schematic of the SubseasonalClimateUSA data collection and processing pipeline.
Following the protocol adopted by the USBR Sub-Seasonal Climate Forecast Rodeos [47, 46], our temperature and precipitation variables are interpolated onto fixed  $1^{\circ} \times 1^{\circ}$  (NUM_LAT=181, NUM_LON=360) and  $1.5^{\circ} \times 1.5^{\circ}$  (NUM_LAT=121, NUM_LON=240) grids using the NCAR Command Language function area_hi2lores_Wrap with arguments newLat = latGlobeF(NUM_LAT, "lat", "latitude", "degrees_north"); newlon = lonGlobeF(NUM_LON, "lon", "longitude", "degrees_east"); wgt = cos(lat*pi/180.0) (so that points are weighted by the cosine of the latitude in radians); opt@critpc = 50 (to require only  $50\%$  of the values to be present to interpolate); and fiCyclic = True (indicating global data with longitude values that do not quite wrap around the globe). All remaining spatial and spatiotemporal variables are interpolated using the Climate Data Operators operator remapdis (distance-weighted average interpolation) with target grid r360x181.
Data Features The variables comprising the SubseasonalClimateUSA dataset include:
- Temperature (global and U.S., 1979-present): daily mean of maximum and minimum temperature at 2 meters in  $^\circ \mathrm{C}$  [14, 40].
- Precipitation (global and U.S., 1948-present): daily accumulated precipitation in mm, aggregated by summing over a rolling two-week window instead of averaging [77, 7, 76, 41, 42].
- Sea surface temperature and sea ice concentration (global, 1981-present): daily variables that track variability in the oceans; the top three principal components for each variable were extracted using global 1981-2010 loadings [54, 45].
- Stratospheric geopotential height, zonal winds, and longitudinal winds (global, 1948-present): daily geopotential height at 10, 100, 500, 850 millibars and zonal and longitudinal winds at 250 and 925 millibars as indicators of polar vortex variability; the top three principal components of each feature were extracted from global 1948-2010 loadings [24, 44].
- Surface pressure and relative humidity (U.S., 1948-present): daily pressure and relative humidity near the surface (sigma level 0.995) [24, 44].
- Sea level pressure, precipitable water for entire atmosphere, and potential evaporation (U.S., 1948–present): daily mean of pressure in millibars, amount of water in the atmosphere available for precipitation in  $\mathrm{kg} / \mathrm{m}^2$ , and potential evaporation rate at surface [24, 44].
- Elevation and Köppen-Geiger climate classification (global): multi-resolution terrain elevation data [9] and Köppen-Geiger climate classification [26] for each grid point.
- Madden-Julian Oscillation (MJO, 1974-present): daily measure of tropical convection known to impact subseasonal climate; phase and amplitude were extracted (but not aggregated) [70, 38].
- Multivariate ENSO index (MEI.v2, 1979-present): bimonthly scalar summary of the state of the El Niño-Southern Oscillation, an ocean-atmosphere coupled climate mode [73-75, 43].
- CFSv2 (U.S., 1999-present): daily 32-member ensemble mean forecasts of temperature and precipitation from the coupled atmosphere-ocean-land dynamical model with 0.5-29.5 day lead times [57, 25, 61].
- SubX (U.S., 1999-present): subweekly forecasts and hindcasts from seven dynamical models (GMAO-GEOS, NRL-NESM, RSMAS-CCSM4, ESRL-FIM, EMC-GEFS, ECCC-GEM, NCEP-CFSv2) and their multi-model mean for temperature and precipitation on a  $1.5^{\circ} \times 1.5^{\circ}$  latitude-longitude grid [25, 61].
- ECMWF (U.S., 1995-present): control and perturbed forecasts and reforecasts of precipitation and temperature on a  $1.5^{\circ} \times 1.5^{\circ}$  latitude-longitude grid [64, 12].
An example of Subseasonal Climate USA observations and dynamical model forecasts is displayed in Figure 2.
Dataset Limitations Here, we highlight several limitations of the SubseasonalClimateUSA dataset. First, the dataset was designed for forecasting in the contiguous U.S., and hence several variables are only available in that region. In future work, we aim to develop an analogous dataset for global subseasonal forecasting. Second, subseasonal forecasts are commonly made at the biweekly temporal resolution and  $1^{\circ}$  or  $1.5^{\circ}$  spatial resolution provided in the SubseasonalClimateUSA dataset [47, 46, 65]; however, these resolutions alone are insufficient for more localized forecasting problems without additional downscaling. Finally, many of our variables have undergone regridding via interpolation, which, while standard, can still introduce inaccuracies.
Figure 2: Example of Subseasonal Climate USA observations and dynamical model forecasts.
3 Subseasonal Forecasting Tasks
We study model performance through four canonical subseasonal forecasting tasks: predicting two variables—average temperature  $(^{\circ}\mathrm{C})$  and accumulated precipitation (mm) over a two-week period—each over two time horizons: 15–28 days ahead (weeks 3–4) and 29–42 days ahead (weeks 5–6). We forecast each variable at  $G = 862$  locations on a  $1^{\circ} \times 1^{\circ}$  latitude-longitude grid covering the contiguous U.S. These prediction targets and time horizons were the focus of the Sub-Seasonal Climate Forecast Rodeos [47, 46], two yearlong real-time forecasting competitions sponsored by USBR and NOAA to advance the state of subseasonal climate prediction. The same targets are used by water managers to apportion water resources, control wildfires, and anticipate droughts and other extreme weather [47, 71].
We evaluate each forecast according to two metrics recommended by the USBR [47, 46]: root mean squared error (RMSE) and skill (also known as uncentered anomaly correlation [72]). For a two-week period starting on date  $t$ , let  $\mathbf{y}_t \in \mathbb{R}^G$  denote the vector of ground-truth measurements  $y_{t,g}$  for each grid point  $g$  and  $\hat{\mathbf{y}}_t \in \mathbb{R}^G$  denote a corresponding vector of forecasts. In addition, define climatology  $c_t$  as the average ground-truth values for a given month and day over the years 1981-2010. Then the RMSE is given by
$$
\mathrm {R M S E} (\hat {\mathbf {y}} _ {t}, \mathbf {y} _ {t}) = \sqrt {\frac {1}{G} \sum_ {g = 1} ^ {G} (\hat {y} _ {t , g} - y _ {t , g}) ^ {2}} \in \mathbb {R} _ {+}
$$
with a smaller value indicating a more accurate forecast, and skill is defined by
$$
\operatorname {s k i l l} \left(\hat {\mathbf {y}} _ {t}, \mathbf {y} _ {t}\right) = \frac {\left\langle \hat {\mathbf {y}} _ {t} - \mathbf {c} _ {t} , \mathbf {y} _ {t} - \mathbf {c} _ {t} \right\rangle}{\| \hat {\mathbf {y}} _ {t} - \mathbf {c} _ {t} \| _ {2} \cdot \| \mathbf {y} _ {t} - \mathbf {c} _ {T} \| _ {2}} \in [ - 1, 1 ]
$$
with a larger value indicating higher quality. For a collection of dates, we report average RMSE and average percentage skill, which is 100 times the average skill.
Training and Validation Recommendations We recommend adopting the Wednesdays of each complete year from 2011 onward as a standard test set when evaluating performance year by year (as in Figure 3) and the Wednesdays from the ten year period 2011-2020 as a standard test set when comparing with the overall (Table 1) or seasonal (Figure 3) performance reported in this work. We also recommend a progressive training and validation protocol in which, to produce a forecast for a given target date, a model can be (re)trained and (re)tuned using any data fully observable on the associated forecast issuance date (i.e., 14 days prior for weeks 3-4 and 28 days prior for weeks 5-6). All of the models evaluated in Section 5 respect this protocol. In addition, the SubseasonalClimateUSA dataset provides convenient combination dataframes containing target variables associated with predictive features lagged by an appropriate amount to ensure that each predictive feature was fully observed on the issuance date associated with the target date.
4 Benchmark Models
Our experiments will evaluate three classes of forecasting methods: three standard meteorological baselines, the adaptive bias correction (ABC) models introduced in [39], and seven other state-of-the-art machine learning and deep learning methods drawn from the literature. We will also evaluate ensemble forecasts derived from these models. Open-source model implementations are available via the subseasonal_toolkit Python package. Most models train on all available data (subject to the progressive training and validation constraint of Section 3) and tune hyperparameters using a second round of progressive evaluation over the prior three years. Appendix B contains supplementary implementation details for each model, including training, hyperparameter tuning, and testing details.
Meteorological Baselines We first consider three standard subseasonal forecasting baselines.
CLIMATOLOGY. Climatology is a standard subseasonal benchmark for the expected temperature or precipitation at a location. For a given grid point and target date, it forecasts the average value of the target variable on the same day and month over 1981-2010 [3].
DEBIASED CFSv2. CFSv2 is the U.S. operational dynamical model commonly used for subseasonal forecasting [57]. Debiased CFSv2 is a corrected ensemble forecast used as a benchmark in the two Subseasonal Climate Forecast Rodeo competitions [46, 23]. First, a CFSv2 ensemble forecast is formed by averaging 32 forecasts for the target period based on 4 different model initializations produced at 8 different lead times. The ensemble is then debiased by adding the mean value of the target variable on the target month and day over the period 1999-2010 and subtracting the mean ensemble CFSv2 reforecast over the same period.
PERSISTENCE. This baseline [37, 69] forecasts the most recently observed two-week target value.
ABC Models We next evaluate the ABC models introduced in [39]. ABC is a hybrid physics-plus-learning approach that takes as input a dynamical model forecast (here, CFSv2) and uses the historical record of prior forecasts and observations to correct the model's output and improve predictive skill. While the three learning models contributing to ABC described below are simple and computationally inexpensive, Section 5 shows that each enhancement improves over both operational practice and state-of-the-art learning techniques.
CLIMATOLOGY++. Climatology++ is as an adaptive form of Climatology that learns how many prior years and how many dates in a window around the target date to include in a smoothed historical mean or geometric median estimate for a given grid point, target date, and target variable; importantly, unlike a static climatology, Climatology++ allows these learned window sizes to vary over time to adapt to noise levels and variability.
$\mathrm{CFSv2 + + }$ .  $\mathrm{CFSv2 + + }$  is a learned correction for raw CFSv2 forecasts. After averaging CFSv2 forecasts over a range of issuance dates and lead times,  $\mathrm{CFSv2 + + }$  debiases the ensemble forecast by adding the mean value of the target variable and subtracting the mean forecast over a learned window of observations around the target day of year. The range of ensembled lead times, the number of averaged issuance dates, and the size of the observation window employed are selected adaptively.
PERSISTENCE++. For each grid point and target date, Persistence++ predicts a target variable as a function of lagged measurements, Climatology, and CFSv2 forecasts observable for the same grid point on the forecast issuance date. These features are combined using a linear least squares regression trained on all available historical data available as of the forecast issuance date.
State-of-the-art Learning Methods We also consider seven state-of-the-art learning methods.
AUTOKNN. AutoKNN [23] was part of a winning solution in the Subseasonal Climate Forecast Rodeo I [46]. Our implementation adapts it to target RMSE as an error metric.
INFORMER. The Informer [79] is a transformer-based deep learning model for time series shown to have state-of-the-art performance on a number of short term weather forecasting tasks.
LOCALBOOSTING. LocalBoosting is a decision tree model using CatBoost [51] over features in a bounding box around the target to extract meaningful spatial information.
MULTILLR. The MultiLLR [23] model is a customized backward stepwise procedure to select SubseasonalClimateUSA features relevant for prediction and local linear regression to combine those features into a forecast for each grid point.