SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In direct forecasting, we train a separate model for each lead time. This can be computationally expensive as the training cost scales linearly with the number of lead times. In this section, we consider two alternative approaches, namely, continuous forecasting and iterative forecasting, and investigate the trade-off between computation and performance. In continuous forecasting, a model conditions on lead time information to make corresponding predictions, which allows the same trained model to make forecasts at any lead times. We refer to Appendix B.3.1 for details on how to do this. In iterative forecasting, we train the model to forecast at a short lead time, i.e., 6 hours, and roll out the predictions during evaluation to make forecasts at longer horizons. We note that in order to roll out more than one step, the model must predict all variables in the input. This provides the benefit of training a single model that can predict at any lead time that is a multiplication of 6.
We compare the performance of direct, continuous, and iterative forecasting using the same ResNet architecture with training and evaluation settings identical to Section 4.1. Figure 3 shows that ResNet-cont slightly underperforms the direct model at 6-hour and 1-day lead times, but performs similarly or even better at 3-day and 5-day forecasting. We hypothesize that for difficult tasks, training with randomized lead times enlarges the training data and thus improves the generalization of the model. A similar result was observed by Rasp et al. [63]. However, the continuous model does not generalize well to unseen lead times, which explains the poor performance when evaluated at 10-day forecasting. ResNet-iter performs the worst in the three approaches, which achieves a reasonable performance at 6-hour lead time, but the prediction error accumulates exponentially at longer horizons. This was also observed in previous works [52, 63]. We believe this issue can be mitigated by multi-step training [54], which we leave as future work.
4.1.2 Extreme weather prediction
Despite the surface-level temperature being an outlier, Table 1 shows that deep learning and persistence perform better on Extreme-ERA5 than ERA5 for all different lead times. Climatology performs worse on Extreme-ERA5, which is expected since predicting the mean of the target distribution is not a good strategy for outlier data. The persistence performance indicates that the variation between the input and output values is comparatively less for extreme weather
Table 1: Latitude-weighted RMSE on the normal and extreme test splits of ERA5.
T2M	6 Hours	1 Day	3 Days
Climatology	5.87 / 6.51	5.87 / 6.53	5.87 / 6.58
Persistence	2.76 / 2.99	2.13 / 1.78	2.99 / 2.42
ResNet	0.72 / 0.72	0.94 / 0.91	1.50 / 1.33
U-Net	0.76 / 0.77	1.04 / 0.99	1.65 / 1.43
ViT	0.78 / 0.80	1.09 / 1.05	1.71 / 1.55
conditions. We hypothesize that, although the marginal distribution  $p(y)$  of the subset data is extreme, the conditional distribution  $p(y|x)$  which we are trying to model is not extreme. Thus, we are not
Table 3: Downscaling experiments on ERA5  $(5.6^{\circ})$  to ERA5  $(2.8^{\circ})$  and ERA5  $(2.8^{\circ})$  to PRISM  $(0.75^{\circ})$ . For ERA5  $(5.6^{\circ})$  to ERA5  $(2.8^{\circ})$ , Pearson's correlation coefficient was 1.0 for all models.
ERA5 to ERA5	ERA5 to PRISM
Z500 (m2 s-2)	T850 (K)	T2m (K)	Daily Max T2m (K)
RMSE	Mean bias	RMSE	Mean bias	RMSE	Mean bias	RMSE	Mean bias	Pearson
Nearest	269.67	0.04	1.99	0.00	3.11	0.00	2.91	-0.05	0.89
Bilinear	134.07	0.04	1.50	0.00	2.46	0.00	2.64	0.12	0.91
ResNet	54.20	-6.41	0.39	-0.05	1.10	-0.22	1.86	-0.11	0.95
Unet	43.84	-6.55	0.94	-0.06	1.10	-0.12	1.57	-0.14	0.97
ViT	85.32	-35.98	1.03	-0.01	1.25	-0.20	2.18	-0.26	0.94
experiencing any drop in performance on such a subset. While from a ML standpoint, it might seem necessary to evaluate the models for cases where the conditional distribution is extreme for any input variable, such a dataset might not qualify under the well-known categories of extreme events. Future studies for constructing extreme datasets could try targeting extreme events such as floods, which are usually caused by high amounts of precipitation under a short period of time [76].
4.1.3 Data robustness of deep learning models
We study the impact of data distribution shifts on forecasting performance. We consider CMIP6 and ERA5 as two different data sources. The input variables are similar to the standard setting, except that we remove relative humidity, 10-meter zonal and meridional wind, incoming solar radiation, and the 3 constant fields due to their unavailability in CMIP6. To account for differences in the temporal resolution and data coverage, we set the temporal resolution to 6 hours and set 1979-2010, 2011-12, and 2013-14 as training, validation, and testing years respectively.
Table 2: Performance of ResNet trained on one dataset (columns) and evaluated on another (rows).
ERA5	CMIP6
ACC	RMSE	ACC	RMSE
ERA5	Z500	0.95	322.86	0.93	345.00
T850	0.93	1.90	0.90	2.21
T2m	0.95	1.62	0.93	1.94
CMIP6	Z500	0.95	357.66	0.96	306.86
T850	0.91	2.11	0.94	1.70
T2m	0.93	1.91	0.96	1.53
Table 2 shows that all methods achieve better evaluation
scores if the training and testing splits come from the same dataset, but cross-dataset performance is not far behind, highlighting the robustness of the models across distributional shifts. We see a similar trend for different models across different lead times, which we refer to Appendix C.3 for more details. We also conducted an experiment where the years 1850-1978 are included in training for CMIP6. The results show that for all models across almost all lead times, training on CMIP6 leads to even better performance on ERA5 than training on ERA5. For exact numbers and setup refer to Appendix C.3.
4.2 Downscaling
Task and data We consider two settings for downscaling. In the first setting, we downscale  $5.625^{\circ}$  ERA5 data to  $2.8125^{\circ}$  ERA5, both at a global scale and hourly intervals. The input and target variables are the same as used in Section 4.1. In the second setting, we consider downscaling  $2.8125^{\circ}$  ERA5 data over the conterminous United States to  $0.75^{\circ}$  PRISM data over the same region at daily intervals. This is equivalent to downscaling a reanalysis/simulated dataset to an observational dataset, similar to previous papers [25, 68, 78]. The cropped ERA5 data has shape  $9 \times 21$  while the regridded PRISM data is padded with zeros to the shape  $32 \times 64$ . The only input and output variable is daily max T2m, which is normalized to have 0 mean and 1 standard deviation. The training period is from 1981 to 2015, validation is in 2016, and the testing period is from 2017 to 2018.
Baselines We compare ResNet, U-Net, and ViT with two baselines: nearest and bilinear interpolation.
Training and evaluation We use MSE as the loss function with the same optimizer and learning rate scheduler as in Section 4.1, with an initial learning rate of  $1 \times 10^{-5}$ . A separate model is trained for each output variable, and all models post-process the results of bilinear interpolation. We use RMSE, Pearson's correlation coefficient, and mean bias as the test metrics. All metrics are masked properly since PRISM does not have data over the oceans. See Appendix B.4 for further details.
Figure 4: Forecasting performance of recent SoTA deep learning models.
Table 4: Performance of the two baselines in probabilistic weather forecasting.
RMSE of ensemble mean (5/10 days)	ACC of ensemble mean (5/10 days)	CRPS (5/10 days)	Spread-Skill Ratio (5/10 days)
Z500	T850	T2m	Z500	T850	T2m	Z500	T850	T2m	Z500	T850	T2m
ResNet-Parametric	610/856	2.83/3.61	2.21/2.74	0.80/0.52	0.82/0.67	0.91/0.85	344.4/488.3	1.77/2.30	1.24/1.54	0.27/0.35	0.32/0.39	0.36/0.39
ResNet-Ensemble	572/807	2.67/3.47	2.09/2.74	0.82/0.57	0.84/0.69	0.92/0.85	323.9/474.8	1.68/2.30	1.19/1.61	0.38/0.29	0.34/0.26	0.34/0.37
Benchmark results Table 3 shows the performance of different baselines in both settings. As expected for the first setting, all methods achieve relatively low errors. The deep learning models outperformed both interpolation methods significantly on RMSE, but tend to overestimate the target variables, leading to negative mean bias. In the second setting—where the input and output come from two different datasets—the performance of all baselines drops. Nonetheless, the deep learning models again outperform the baseline methods on RMSE and exhibit negative mean bias, but also achieve higher Pearson's correlation coefficients.
5 Additional experiments
5.1 Benchmarking recent SoTA methods
We develop ClimateLearn as a long-term sustainable project that allows easy extension to new models, tasks, and datasets by us and others (via open-source). To demonstrate this, we added ClimaX [52] and FourCastNet [54] as two state-of-the-art deep learning models for weather forecasting to ClimateLearn. Figure 4 benchmarks these two methods against ResNet and IFS. While ClimaX follows IFS closely and even surpasses this strong baseline at the 10-day lead time, FourCastNet performs poorly on low-resolution data.
5.2 Probabilistic weather forecasting
To further showcase the flexibility and extensibility of ClimateLearn, we added probabilistic weather forecasting to the suit of tasks supported by ClimateLearn. We implemented two variants of the ResNet model for this task: ResNet-parametric which directly outputs the mean and standard deviation of a Gaussian distribution as the prediction of the future, and ResNet-ensemble where we train 10 different instances of the same ResNet architecture using different seeds and averaging the predictions of these instances to get the final prediction. We evaluate these two baselines using RMSE, ACC, and the probabilistic metrics CRPS and Spread-skill ratio introduced in [21]. Table 4 compares the two baselines in this task.
We presented ClimateLearn, a user-friendly and open-source PyTorch library for data-driven weather and climate modeling. Given the pressing nature of climate change, we believe our contribution is timely and of potential use to both the ML and climate science communities. Our objective is to provide a standardized benchmarking platform for evaluating ML innovations in climate science, which currently suffer from challenges in standardization, accessibility, and reproducibility. ClimateLearn provides users access to all essential components of end-to-end ML pipeline, including data pre-processing utilities, ML model implementations, and rigorous evaluations via metrics and visualizations. We use the flexible and modular design of ClimateLearn to design and perform diverse experiments comparing deep learning methods with relevant baselines on our supported tasks.
Limitations and Future Work In this work, we highlighted key features of the ClimateLearn library, encompassing datasets, tasks, models, and evaluations. However, we acknowledge that there are numerous avenues to enhance the comprehensiveness of our library in each of these dimensions. One such avenue involves integrating regional datasets and expanding the catalog of available data sources. On the modeling side, we plan to develop efficient implementations for training ensembles in service of critical uncertainty quantification efforts. In future iterations of our library, we will also integrate a hub of large-scale pretrained neural networks specifically designed for weather and climate applications [4, 38, 52, 54]. Once integrated, these pretrained models will be further customizable through fine-tuning, enabling straightforward adaptation to downstream tasks. Furthermore, we plan to incorporate support for physics-informed neural networks and other hybrid baselines that amalgamate physical models with machine learning methods, which will allow users to leverage the strengths of both paradigms. Ultimately, our overarching objective is to establish ClimateLearn as a trustworthy AI development tool for weather and climate applications.
We thank World Climate Research Programme (WCRP) for the CMIP6 data collection, and European Center for Medium-Range Weather Forecasting (ECMWF) for the ERA5 dataset. We thank Shashank Goel, Jingchen Tang, Seongbin Park, Siddharth Nandy, and Sri Keerthi Bolli for their contributions to ClimateLearn. Aditya Grover was supported in part by a research gift from Google. Hritik Bansal was supported in part by AFOSR MURI grant FA9550-22-1-0380.
[81] Thomas Vandal, Evan Kodra, Sangram Ganguly, Andrew Michaelis, Ramakrishna Nemani, and Auroop R Ganguly. Deepsd: Generating high resolution climate change projections through single image super-resolution. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pages 1663–1672, 2017.
[82] Thomas Vandal, Evan Kodra, and Auroop R Ganguly. Intercomparison of machine learning methods for statistical downscaling: the case of daily and extreme precipitation. Theoretical and Applied Climatology, 137:557-570, 2019.
[83] Robert T Watson and Daniel Lee Albritton. Climate change 2001: Synthesis report: Third assessment report of the Intergovernmental Panel on Climate Change. Cambridge University Press, 2001.
[84] D. Watson-Parris, Y. Rao, D. Olivie, Ø. Seland, P. Nowack, G. Camps-Valls, P. Stier, S. Bouabid, M. Dewey, E. Fons, J. Gonzalez, P. Harder, K. Jeggle, J. Lenhardt, P. Manshausen, M. Novitasari, L. Ricard, and C. Roesch. ClimateBench v1.0: A Benchmark for Data-Driven Climate Projections. Journal of Advances in Modeling Earth Systems, 14(10):e2021MS002954, 2022. doi: https://doi.org/10.1029/2021MS002954. URL https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2021MS002954. e2021MS002954 2021MS002954.
[85] NP Wedi, P Bauer, W Denoninck, M Diamantakis, M Hamrud, C Kuhnlein, S Malardel, K Mogensen, G Mozdzynski, and PK Smolarkiewicz. The modelling infrastructure of the Integrated Forecasting System: Recent advances and future challenges. European Centre for Medium-Range Weather Forecasts, 2015.
[86] Jonathan A. Weyn, Dale R. Durran, and Rich Caruana. Can Machines Learn to Predict Weather? Using Deep Learning to Predict Gridded 500-hPa Geopotential Height From Historical Weather Data. Journal of Advances in Modeling Earth Systems, 11(8):2680-2693, 2019. doi: https://doi.org/10.1029/2019MS001705. URL https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001705.
[87] Jonathan A. Weyn, Dale R. Durran, and Rich Caruana. Improving Data-Driven Global Weather Prediction Using Deep Convolutional Neural Networks on a Cubed Sphere. Journal of Advances in Modeling Earth Systems, 12(9):e2020MS002109, 2020. doi: https://doi.org/10.1029/2020MS002109. URL https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002109. e2020MS002109 10.1029/2020MS002109.
[88] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[89] Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar. Integrating physics-based modeling with machine learning: A survey. arXiv preprint arXiv:2003.04919, 1 (1):1-34, 2020.
[90] Rong Zhang, Zhao-Yue Chen, Li-Jun Xu, and Chun-Quan Ou. Meteorological drought forecasting based on a statistical model with machine learning techniques in Shaanxi province, China. Science of The Total Environment, 665:338-346, 2019. ISSN 0048-9697. doi: https://doi.org/10.1016/j.scitotenv.2019.01.431. URL https://www.sciencedirect.com/science/article/pii/S0048969719302281.