SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Accelerate ClimateSet. If run on a machine with 1 CPU core, 16GB memory, with single-threading, the complete preprocessing of the 36 climate models takes  $\sim 160$  hours. The preprocessing can be accelerated in the following ways: (1) Using the multi-thread function of the CDO-implemented processors (resolution & raw); (2) waiving the checker and/or the structure processor; (3) supplying the resolution processor with an example resolution map. This is further explained in Appendix J.
2.4 Limitations
Data Retrieval. When extending the dataset, users may run into issues with the data retrieval from ESGF's server nodes. Server nodes can be down from time to time, i.e. users need to wait until those nodes are back online, which can take up to weeks (node status: https://esgf-node.l1nl.gov/status/). Usually, only a subset of the data, e.g. one specific climate model, is affected by this.
Computational Resources. Depending on the size of the desired dataset, extending ClimateSet might only be feasible with access to a high-performance compute (HPC) cluster. The core dataset of ClimateSet is already relatively large with 2.0 TB of preprocessed data. Furthermore, for a set of multiple climate models, the preprocessing takes too long to be carried on a local machine that only supports single-threading for CDO (see "Accelerate ClimateSet" in Section 2.3). However, for a smaller set of climate models, storing and preprocessing the dataset on a local machine works well.
Weighting of Climate Models. ClimateSet currently unites 36 climate models without considering the similarity between some of them. Explanations of within- and across- climate model similarities are given in Appendix F. When training a large model on ClimateSet it would be beneficial to weight the climate models to prevent over- and under-representation of some climate models or sub-models. However, such a weighting does only exist for CMIP5 so far (Massoud et al., 2020; Wootten et al., 2020), and requires in-depth domain knowledge and is thus beyond the work presented here.
Evaluation. ClimateSet is limited by its current deterministic setting. Climate models are deterministic, however, uncertainties among and across them could be captured in future work. This requires weighting them as discussed above. Moreover, the evaluation metrics should be extended and adapted for different climatic variables and task settings; metrics will be updated continuously on GitHub.
Extension beyond ScenarioMIP. Users might want to retrieve and process data beyond ScenarioMIP. The downloader and processor pipeline of ClimateSet can be re-used to that end, however, we did not test those cases. We encourage pull and feature requests for other CMIP6-endorsed datasets.
3 Benchmarking Setup
Task. In climate emulation the objective is to simulate the output of a climate model as closely as possible. The emulator receives the same input as the climate model, but can produce climate projections for new input data a lot faster than climate models during inference time (see Fig. 5). Here, the goal is to predict a time-series of climate variables (e.g. temperature and precipitation for 2015-2100) from a given parallel time-series of climate forcing emission maps from 2015-2100. We treat the task as a diagnostic-type prediction, however, it can also be treated as an autoregressive task. Refer to Watson-Parris et al. (2022) for further explanations about emulation. We use two different versions of climate emulators: (1) Single-Emulators, and (2) Super-Emulators. By "Single-Emulator" we refer to an emulator that is trained on a single climate model. By "Super-Emulator" we refer to an emulator that is trained on a set of climate models, and is able to project the climate responses of all the participating climate models. The super-emulator is explained in more detail in Appendix L.2.
ML Models. We trained most types of models that have been used for climate emulation on ClimateBench (Watson-Parris et al., 2022) to date: Convolutional long short-term memory (ConvLSTM) (Hochreiter and Schmidhuber, 1997; LeCun et al., 1989; Watson-Parris et al., 2022), Gaussian Process regression (GP) (Williams and Rasmussen, 2006; Hensman et al., 2015), and ClimaX (Nguyen et al., 2023). ClimaX is the current state-of-the-art ML model on ClimateBench. We omitted the Random Forest (RF) (Breiman, 2001) since we could not fully reproduce ClimateBench's experiments with our training configuration of predicting two variables concurrently. We added a U-Net (Ronneberger et al., 2015) as a simple baseline. Where necessary, we adapted ClimateBench's implementations. The implementation details and differences to the original models are described in Appendix K.
Data. Each emulator receives as input the climate forcing emission fields  $(\mathrm{CO}_2, \mathrm{CH}_4, \mathrm{SO}_2, \mathrm{BC})$ , and as target the output variables of climate models (temperature, precipitation). All data was processed to have a spatial resolution of approximately  $250\mathrm{km}$  (144 x 96 longitude-latitude cells) and a temporal resolution of monthly data. For both the input and target data, there are 86-year time-series available for the 4 SSP scenarios (2015 - 2100), and 165 years for the historical scenario (1850 - 2014). Those time-series are divided into 1-year chunks. The resulting data has a shape of (scenarios * years * months, variables, longitude, latitude). Assuming we choose 86 years and four climate forcers, we would map from input shape  $(5*86*12, 4, 144, 96)$  to output  $(5*86*12, 2, 144, 96)$ .
Figure 5: Climate emulation. A) A climate model receives the future trajectories of climate forcers as input, and uses this input to simulate the future climate with the help of its different components (atmosphere, ocean, ice, land, and fire models). It outputs how the climate (e.g. temperature) would respond to a given emission scenario. B) A climate model emulator receives the same inputs as a climate model and learns to emulate its outputs. After training on several scenarios and ensemble members, it can predict new scenarios much faster than a traditional climate model.
Train-Test-Split. For training and validation, the historical scenario, SSP1-2.6, SSP3-7.0, and SSP5-8.5 are used. A random  $10\%$  split of the data is hold out for validation, and SSP2.45 for testing.
Experiments. We run experiments on (A) single-emulation, (B) super-emulation, and (C) generalization capabilities of the different ML models. For single-emulation, each ML model is trained on each of the 15 climate models separately (i.e., we result with 15 independent ML models). Our models are trained on our internal cluster, using a single Nvidia-RTX8000 with 32GB of RAM. For the super-emulation task, a single ML model is trained on 6 climate models together to demonstrate how super-emulation works (Appendix L.2). Future work can extend the super-emulator to run on all 36 climate models. During inference, the super-emulator can predict novel scenarios for each climate model that participated in training.
Further Notes. To test the generalization capabilities of the single-emulator, we use its weights, train on a climate model, finetune on NorESM2-LM and compared the results with a single-emulator trained only on NorESM2-LM. All experiments included only one ensemble member; future experiments could evaluate the influence of intra-model variability on ML model performance further. For all experiments we used the latitude-longitude weighted root mean squared error (RMSE) as implemented in (Nguyen et al., 2023) as main evaluation metric. Refer to Appendix L for additional details.
4 Benchmarking Results
Here, we present a subset of our climate emulation results to investigate the differences between using a dataset that contains a single climate model, and one that contains multiple climate models. Fig. 6 shows the RMSE of temperature projections for the test scenario (SSP2-4.5) among the neural-network based models (ClimaX, ClimaX Frozen, ConvLSTM, and U-Net) on a subset of six climate models (NorESM2-LM, NorESM2-MM, MPI-ESM1-2-HR, GFDL-ESM4, TaiESM1, and EC-Earth3). With the exception of the U-Net, the named ML models had been considered in (Nguyen et al., 2023) on NorESM2-LM data retrieved from ClimateBench. We find the performance of all ML models on NorESM2-LM to be in a similar range as reported in (Nguyen et al., 2023). ClimaX, ClimateBench, and ClimateSet observed all different performance values for ConvLSTM on NorESM2-LM due to differences in the implementations, e.g. ClimateSet's ConvLSTM outperformed ClimaX's ConvLSTM version (0.3 vs. 0.4). In general, the RMSE values are in line with previous
Figure 6: Emulation benchmark. RMSE for the temperature projection of SSP2-4.5 (2015 - 2100) for the ML models ClimaX Frozen, ClimaX, ConvLSTM, and U-Net. Each ML model (color coded) was trained separately on the climate models NorESM2-LM, NorESM2-MM, MPI-ESM1-2-HR, GFDL-ESM4, TaiESM1, and EC-Earth3. The RMSE values (lower is better) are latitude-longitude weighted and averaged over three seeds for each ML model.
work where comparable. Notably, simple models such as the U-Net and ConvLSTM (we applied no significant tuning or adoptions to those models) can keep up with CimaX. U-Net even outperformed CimaX consistently when dropping CimaX's warm-up period. Refer to Appendix M.1 for more single-emulator results. This shows that for climate emulation tasks significant performance gains can still be made with relatively simple baselines that had not yet been investigated for this task.
While the different ML models showed consistent performance across different climate models, there were exceptions. For example, when looking only at the NorESM2 models, it seems that ClimateSet's ConvLSTM significantly outperforms both ClimaX and ClimaX Frozen (Fig. 6). When looking across several climate models, however, it becomes quickly apparent that both ClimaX and ClimaX Frozen outperform the ConvLSTM across multiple climate models. The consistency of the performance might depend on (A) whether an ML model overfits one climate model (i.e. the hyperparameters match this model in particular), and (B) whether an ML model is particularly good at generalizing across different climate models. Refer to Appendix M.2 and Table 5 to see the ML models performance across multiple climate models. Figure 6 shows also the climate model on which the best performance could be achieved (TaiESM1) and the worst (GFDL-ESM4). Those climate models were the best/worst across all ML models, indicating that some climate models are easier to emulate than others. In summary, the results show that testing and comparing ML models on just one climate model is not sufficient for finding the "best" ML emulator; instead, ML models should be evaluated on a set of climate models - posing different levels of difficulty - to draw such conclusions.
When training ML-models on super-emulation, on six climate models, the leader-board of the ML-models inverts (Appendix M.2): The simplest model (ConvLSTM) yields the best, and the most complex model (ClimaX) the worst performance. The experiments to test generalizability (Appendix M.3) could explain this: The simpler models are better at generalizing across several climate models, while the more complex models fail to generalize as well. Future work is needed to investigate if larger ML models simply need a longer training period to learn a full, generalizable, representation of climate models. The results show that ClimateSet can be used to identify ML models generalizing across different climate models. Thus, ClimateSet can be the basis for developing new climate emulators that can emulate climate models across the CMIP6 archive instead of only a single one.
With ClimateSet, we respond to the widely expressed need for a large-scale, consistent climate model dataset for machine learning. Among other tasks, ClimateSet can be used to reveal new insights for ML climate emulation. We found that ClimaX is the best single-emulator, while ConvLSTM generalizes better and is the best super-emulator across multiple climate models. We found the overall ranking of ML models varied across different climate model datasets, showing the need for ClimateSet as a unified benchmark. ClimateSet also provides a pipeline to retrieve and preprocess further climate model data consistently. We envision this will be particularly useful for researchers who need large training datasets, e.g. for climate foundation models. We hope ClimateSet enables the ML-community to address a much wider range of climate-related tasks. Tackling those tasks on the scale of CMIP6 will help the ML-community to contribute meaningfully to climate policy making.
We acknowledge the WCRP, which, through its Working Group on Coupled Modeling, coordinated and promoted CMIP6. We thank the climate modeling groups for producing and making available their model output, the Earth System Grid Federation (ESGF) for archiving the data and providing access and the multiple funding agencies that support CMIP6 and ESGF. We would like to thank Sonia Yousfi for her helpful suggestions, Trevor Greco for editing guidance, and Sébastien Lachapelle and Julien Boussard for insightful discussions. Many thanks go to Björn Lütjens for being ClimateSet's beta tester and helping us to improve it with his valuable and deliberate feedback. Special thanks go to Boris Sakschewski for lending us his wonderful 3d graphic work on ESMs, and Steven J. Smith for his supportive explanations on the Input4MIPs data.
This project was supported by the Intel-Mila partnership program and Canada CIFAR AI Chairs program. J.R. has received funding from the European Research Council (ERC) Starting Grant CausalEarth under the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 948112). P.N. was supported by UK Natural Environment Research Council (grant no. NE/V012045/1).
Arias, P., N. Bellouin, E. Coppola, R. Jones, G. Krinner, J. Marotzke, V. Naik, M. Palmer, G.-K. Plattner, J. Rogelj, M. Rojas, J. Sillmann, T. Storelvmo, P. Thorne, B. Trewin, K. Achuta Rao, B. Adhikary, R. Allan, K. Armour, G. Bala, R. Barimalala, S. Berger, J. Canadell, C. Cassou, A. Cherchi, W. Collins, W. Collins, S. Connors, S. Corti, F. Cruz, F. Dentener, C. Dereczynski, A. Di Luca, A. Diongue Niang, F. Doblas-Reyes, A. Dosio, H. Douville, F. Engelbrecht, V. Eyring, E. Fischer, P. Forster, B. Fox-Kemper, J. Fusestvedt, J. Fyfe, N. Gillett, L. Goldfarb, I. Gorodetskaya, J. Gutierrez, R. Hamdi, E. Hawkins, H. Hewitt, P. Hope, A. Islam, C. Jones, D. Kaufman, R. Kopp, Y. Kosaka, J. Kossin, S. Krakovska, J.-Y. Lee, J. Li, T. Mauritsen, T. Maycock, M. Meinshausen, S. Min, P. Monteiro, T. Ngo-Duc, F. Otto, I. Pinto, A. Pirani, K. Raghavan, R. Ranasinghe, A. Ruane, L. Ruiz, J.-B. Salleee, B. Samset, S. Sathyendranath, S. Seneviratne, A. Sorensson, S. Szopa, I. Takayabu, A.-M. Tréguier, B. van den Hurk, R. Vautard, K. von Schuckmann, S. Zaehle, X. Zhang, and K. Zickfeld (2021). Technical summary. In V. Masson-Delmotte, P. Zhai, A. Pirani, S. Connors, C. Pean, S. Berger, N. Caud, Y. Chen, L. Goldfarb, M. Gomis, M. Huang, K. Leitzell, E. Lonnow, J. Matthews, T. Maycock, T. Waterfield, O. Yelekci, R. Yu, and B. Zhou (Eds.), Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change, pp. 33-144. Cambridge University Press.
Balaji, V., E. Maisonnave, N. Zadeh, B. N. Lawrence, J. Biercamp, U. Fladrich, G. Aloisio, R. Benson, A. Caubel, J. Durachta, M.-A. Foujols, G. Lister, S. Mocavero, S. Underwood, and G. Wright (2017). Cpmip: measurements of real computational performance of earth system models in cmip6. Geoscientific Model Development 10(1), 19-34.
Balaji, V., K. E. Taylor, M. Juckes, B. N. Lawrence, P. J. Durack, M. Lautenschlager, C. Blanton, L. Cinquini, S. Denvil, M. Elkington, et al. (2018). Requirements for a global data infrastructure in support of cmip6. Geoscientific Model Development 11(9), 3659-3680.
Beusch, L., L. Gudmundsson, and S. I. Seneviratne (2020). Emulating earth system model temperatures with MESMER: from global mean temperature trajectories to grid-point-level realizations on land. Earth System Dynamics 11(1), 139-159.
Bi, D., M. Dix, S. Marsland, S. O'farrell, A. Sullivan, R. Bodman, R. Law, I. Harman, J. Srbinovsky, H. A. Rashid, et al. (2020). Configuration and spin-up of access-cm2, the new generation australian community climate and earth system simulator coupled model. Journal of Southern Hemisphere Earth Systems Science 70(1), 225-251.
Bi, K., L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian (2022). Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556.
Bonev, B., T. Kurth, C. Hundt, J. Pathak, M. Baust, K. Kashinath, and A. Anandkumar (2023). Spherical fourier neural operators: Learning stable dynamics on the sphere. arXiv preprint arXiv:2306.03838.
Watson-Parris, D. (2021). Machine learning for weather and climate are worlds apart. Philosophical Transactions of the Royal Society A 379(2194), 20200098.
Watson-Parris, D., Y. Rao, D. Olivie, Ø. Seland, P. Nowack, G. Camps-Valls, P. Stier, S. Bouabid, M. Dewey, E. Fons, et al. (2022). Climatebench v1. 0: A benchmark for data-driven climate projections. Journal of Advances in Modeling Earth Systems 14(10), e2021MS002954.
Weyn, J. A., D. R. Durran, R. Caruana, and N. Cresswell-Clay (2021). Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models. Journal of Advances in Modeling Earth Systems 13(7), e2021MS002502.
Williams, C. K. and C. E. Rasmussen (2006). Gaussian processes for machine learning, Volume 2. MIT press Cambridge, MA.
Wootten, A. M., E. C. Massoud, A. Sengupta, D. E. Waliser, and H. Lee (2020). The effect of statistical downscaling on the weighting of multi-model ensembles of precipitation. Climate 8(12).
Wu, T., R. Yu, Y. Lu, W. Jie, Y. Fang, J. Zhang, L. Zhang, X. Xin, L. Li, Z. Wang, Y. Liu, F. Zhang, F. Wu, M. Chu, J. Li, W. Li, Y. Zhang, X. Shi, W. Zhou, J. Yao, X. Liu, H. Zhao, J. Yan, M. Wei, W. Xue, A. Huang, Y. Zhang, Y. Zhang, Q. Shu, and A. Hu (2021). Bcc-csm2-hr: a high-resolution version of the beijing climate center climate system model. Geoscientific Model Development 14(5), 2977-3006.
Yu, S., W. M. Hannah, L. Peng, M. A. Bhouri, R. Gupta, J. Lin, B. Lütjens, J. C. Will, T. Beucler, B. E. Harrop, et al. (2023). Climsim: An open large-scale dataset for training high-resolution physics emulators in hybrid multi-scale climate simulators. arXiv preprint arXiv:2306.08754.
Yu, Y., J. Mao, S. D. Wullschleger, A. Chen, X. Shi, Y. Wang, F. M. Hoffman, Y. Zhang, and E. Pierce (2022). Machine learning-based observation-constrained projections reveal elevated global socioeconomic risks from wildfire. Nature communications 13(1), 1250.
Yukimoto, S., H. Kawai, T. Koshiro, N. Oshima, K. Yoshida, S. Urakawa, H. Tsujino, M. Deushi, T. Tanaka, M. Hosaka, S. Yabu, H. Yoshimura, E. Shindo, R. Mizuta, A. Obata, Y. Adachi, and M. Ishii (2019). The meteorological research institute earth system model version 2.0, mri-esm2.0: Description and basic evaluation of the physical component. Journal of the Meteorological Society of Japan. Ser. II 97(5), 931-965.
Yukimoto, S., H. Kawai, T. Koshiro, N. Oshima, K. Yoshida, S. Urakawa, H. Tsujino, M. Deushi, T. Tanaka, M. Hosaka, S. YABU, H. Yoshimura, E. Shindo, R. Mizuta, A. Obata, Y. Adachi, and M. Ishii (2019, 06). The meteorological research institute earth system model version 2.0, mriesm2.0: Description and basic evaluation of the physical component. Journal of the Meteorological Society of Japan 97.
Zhou, G., Y. Zhang, J. Jiang, H. Zhang, B. Wu, H. Cao, T. Wang, H. Hao, J. Zhu, L. Yuan, and M. Zhang (2020). Earth system model: Cas-ism. Frontiers of Data and Computing 2(1), 38.
Ziehn, T., M. A. Chamberlain, R. M. Law, A. Lenton, R. W. Bodman, M. Dix, L. Stevens, Y.-P. Wang, and J. Srbinovsky (2020). The australian earth system model: Access-ism1. 5. Journal of Southern Hemisphere Earth Systems Science 70(1), 193-214.
A Data and Code Availability
Access to code and data is provided on https://climateset.github.io.
B Overview of Related ML-Datasets
Figure 7: ML datasets and models used in the weather and climate domain. Only models and datasets are included that operate on a global scale and use either re-analysis or climate model data. The dots represent ML-models, the squares datasets. The area covered by the datasets shows to which spatial and temporal scales it extends. The models and datasets are colored darker when several climate models (ESMs) are used. Some ML-models were run for several tasks on different temporal and spatial scales; their tasks are represented by different points that are connected with dotted lines. The figure illustrates that related ML work has mostly focused on the weather scale and not the centurial climate scale. It also shows that high number of climate models has been under-utilized in the ML community so far. The figure was inspired by Mikkavilli et al. (2023).
Datasets:
ClimSim (Yu et al., 2023)
- RainBench (de Witt et al., 2021)
- WeatherBench (Rasp et al., 2020) and WeatherBench2.0 (Rasp et al., 2023)
ClimateLearn (Nguyen et al., 2023)
ClimateBench (Watson-Parris et al., 2022)
Models:
PanguWeather (Bi et al., 2022)
AtmoRep (Lessig et al., 2023)
- ClimaX (Nguyen et al., 2023)
FourCastNet (Pathak et al., 2022)
- deCNN (Nielsen et al., 2022)
GraphCast (Lam et al., 2022)
FengWu (Chen et al., 2023)
- FuXi (Chen et al., 2023)
DLWP (Weyn et al., 2021)
SNFO (Bonev et al., 2023)
- UNet++ (Unal et al., 2023)
C CO2 Data
The greenhouse gas CO2 requires special handling within this dataset and within climate emulation in particular. In Fig. C.1, an example of a CO2 emission map is shown.
One reason for the special handling is the accumulative nature of CO2 (Canadell et al., 2007), i.e. it does not have a half-life as other greenhouse gases or aerosols do. While the carbon cycle does have natural sinks and sources, the anthropogenic CO2 emissions are still considered accumulative, since the CO2 does get added to the overall carbon-cycle instead of being broken down as in the case of other GHG gases. Consequently, we oversimplify and lose important information if we only use short time-chunks of CO2 emissions. The following two approaches could be used to address this problem: 1) Using the full time-series of CO2 emissions, i.e. mapping 85 years of GHG emissions to 85 years of climate response. 2) Using the cumulative CO2 emissions. Presently, our dataset does not provide the cumulative CO2 emissions for the user to select between options (1) and (2), however, we might add the cumulative data in the future.
The other reason why CO2 requires special handling is that the "fire datasets" (historical biomassburning and future openburning) Van Marle et al. (2017); Feng et al. (2020) do not contain CO2. The historic biomassburning data simply does not have this information to date. The future openburning dataset might not have this data because it partially builds upon the historic biomassburning dataset Van Marle et al. (2017). This leads to the file-structure of our dataset containing "all-fires", "anthro-fires", "no-fires" for all greenhouse gases except CO2. CO2 has only one type of file ("sum"), summing up the anthropogenic emissions and the aircraft emissions. This is true for both the historical and the SSP data. We will update our dataset in case future publications targeting Input4MIPs provide openburning and biomassburning data.