SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Julia Kaltenborn*  
Mila Quebec AI Institute & McGill University
Charlotte Emilie Elektra Lange  
Mila Quebec AI Institute & University of Osnabrück
Venkatesh Ramesh  
Mila Quebec AI Institute & University of Montreal
Philippe Brouillard  
Mila Quebec AI Institute & University of Montreal
Yaniv Gurwicz  
Intel Labs
Chandni Nagda  
University of Illinois at Urbana-Champaign
Jakob Runge  
German Aerospace Center & Technische Universität Berlin
Peer Nowack Karlsruhe Institute of Technology
David Rolnick  
Mila Quebec AI Institute & McGill University
Climate models have been key for assessing the impact of climate change and simulating future climate scenarios. The machine learning (ML) community has taken an increased interest in supporting climate scientists' efforts on various tasks such as climate model emulation, downscaling, and prediction tasks. Many of those tasks have been addressed on datasets created with single climate models. However, both the climate science and ML communities have suggested that to address those tasks at scale, we need large, consistent, and ML-ready climate model datasets. Here, we introduce ClimateSet, a dataset containing the inputs and outputs of 36 climate models from the Input4MIPs and CMIP6 archives. In addition, we provide a modular dataset pipeline for retrieving and preprocessing additional climate models and scenarios. We showcase the potential of our dataset by using it as a benchmark for ML-based climate model emulation. We gain new insights about the performance and generalization capabilities of the different ML models by analyzing their performance across different climate models. Furthermore, the dataset can be used to train an ML emulator on several climate models instead of just one. Such a "super-emulator" can quickly project new climate change scenarios, complementing existing scenarios already provided to policymakers. We believe ClimateSet will create the basis needed for the ML community to tackle climate-related tasks at scale.
Climate change poses a significant and increasing threat to humans and the environment. Understanding and projecting future climate scenarios is essential to mitigating and adapting to climate change. Those future climate scenarios - the "Shared Socioeconomic Pathways" (SSP) are determined by climate forcer emissions and depend on socioeconomic decisions made by humanity. Here, the term "climate forcers" refers to greenhouse gases (GHG), aerosols, and aerosol precursors, among others.
\*Address correspondence to julia.kaltenborn[at]mail.mcgill.ca For readers who are new to climate modeling terminology, we recommend the glossary by IPCC (2022), also available on https://www.ipcc.ch/sr15/chapter/glossary/.
To navigate citizens' future in a changing climate, policymakers rely heavily on future simulations of our climate, summarized in reports for the Intergovernmental Panel on Climate Change (IPCC), e.g. Arias et al. (2021)). Those simulations are traditionally created by climate models, which are collected in the Coupled Model Intercomparison Project (CMIP; recently in Phase 6). Climate models are based on physical parameters, equations, and coupling mechanisms that describe the climate and Earth system; these models simulate the climate under different forcing scenarios, e.g. varying greenhouse gas emissions. However, even on top high-performance computing (HPC) clusters, typical climate model simulations take months to run (Balaji et al., 2017).
The machine learning (ML) community has taken increased interest in supporting the climate science community in their efforts to scale and accelerate climate-related modeling tasks. These tasks include climate emulation/projection, downscaling, and general prediction tasks. Climate-related tasks also pose interesting ML challenges due to the high dimensionality of the data, relatively low sample size, and inherent distribution shifts within the data. When approaching those climate-related tasks, ML models have typically leveraged one climate model (Watson-Parris et al., 2022; Cachay et al., 2021; Mansfield et al., 2020; Krasnopolsky et al., 2013; Castruccio et al., 2014; Holden and Edwards, 2010; Beusch et al., 2020), and only rarely several climate models (Nguyen et al., 2023; Yu et al., 2022). This runs counter to the standard practice in ML of leveraging massive datasets. This discrepancy may be due to the difficulty in retrieving, preprocessing, and handling climate data correctly without significant domain knowledge. Indeed the ML community has experienced difficulties in retrieving data of several climate models Nguyen et al. (2023) and making them consistent Busecke and Abernathey (2020). Those data challenges might limit the community's ability to contribute to climate-related modeling tasks.
While the need for a consistent, easy-to-retrieve, and large climate model dataset to train ML models is currently unmet, it has been addressed in parts. For example, these desired datasets can be found for weather (WeatherBench by Rasp et al. (2020)) and satellite data (EarthNet2021 by Requena-Mesa et al. (2021)) rather than climate data. The access to large-scale weather data enabled the development of large ML weather forecasting models (Lam et al., 2022; Bi et al., 2022; Pathak et al., 2022; Gao et al., 2022). Additionally, ClimaX (Nguyen et al., 2023) – the first climate and weather-related large-scale model – relies primarily on weather data. However, we cannot solely use weather data capturing “the past”, to address climate change related questions concerning “the future”. Extrapolation into such a future is difficult for ML models, especially under strong distribution shifts in space and time. Thus, large-scale and consistent ML datasets are needed not only for weather, but also for climate. Efforts have been made to provide such data: xmip (Busecke and Abernathey, 2020) provides the tools to create more consistent climate model data, however, it does not address all inconsistencies and e.g. cannot align the different temporal and spatial resolutions among climate data. ClimateBench (Watson-Parris et al., 2022) provides a consistent, ML-ready dataset for climate emulation. A drawback to this dataset is that it provides only one climate model. Therefore, it does not capture the multi-model uncertainty that is essential for informing policy making, and is limited in the amount of training data it can provide to ML tasks. Refer to Appendix B for a more comprehensive overview of related ML-datasets. The need expressed by both the ML and climate science communities (Dueben et al., 2022; Runge et al., 2019; Mansfield et al., 2020; Watson-Parris, 2021; Chantry et al., 2021) for a consistent, large, and ML-ready dataset has not yet been addressed jointly for climate data.
Here, we introduce ClimateSet - a consistent, multi-climate-model dataset. We showcase the value of the dataset for the task of climate emulation; however, the dataset can also be used for a wide variety of other tasks. Our main contributions are:
- We introduce the ClimateSet data pipeline, which can be used to retrieve and preprocess climate model data from CMIP6 (climate model outputs) and Input4MIPs (climate model inputs) for climate-related ML tasks.
- We use this pipeline to build a core ClimateSet dataset with outputs of 36 climate models and inputs for the emission fields of 4 different Shared Socioeconomic Pathway (SSP) scenarios and historical data.
- We use ClimateSet to compare state-of-the-art ML methods across different climate models on a climate model emulation task. We emulate temperature and precipitation responses to climate forcers, obtaining results that are both qualitatively different and more reliable than was possible in previous work.
Figure 1: ClimateSet. A) ClimateSet builds on the Input4MIPs and CMIP6 datasets made available through multiple climate modeling teams on the Earth System Grid Federation (ESGF) servers. B) ClimateSet consists of a preprocessed, ML-ready core dataset that includes inputs and outputs for 5 scenarios, 4 climate forcing agents, 2 climatic variables (temperature and precipitation), for a set of 36 climate models. It is currently made publicly available through the Digital Research Alliance of Canada. C) The core dataset can be extended to include different variables, height levels, ensemble members, scenarios and any other information made available from climate models on the CMIP6 server of ESGF. The downloader and preprocessing pipelines are available on our GitHub repository. D) Potential use cases for ClimateSet range from climate projection, climate data downscaling, extreme weather prediction in different warming scenarios, to large ML climate models. E) The main tools provided through ClimateSet are the downloader and the preprocessor to make the climate model data consistent with each other. For further information visit https://climateset.github.io. (The 3D Earth System Model visualization was created by Boris Sakschewski, used with permission).
2 ClimateSet
The core dataset of ClimateSet consists of 36 climate models and their corresponding greenhouse gases, aerosols and aerosol precursor emission inputs for five different scenarios. The core dataset can be extended with the pipeline we provide. For an overview of ClimateSet refer to Fig. 1. The dataset and the pipeline are both publicly available on https://climateset.github.io. ClimateSet serves two main purposes: (1) Providing the amount of training data needed for large-scale ML models; and (2) capturing the projection uncertainty across climate models that is key for climate policy making. Both purposes can only be fulfilled by a dataset containing several climate models. The following describes the core dataset, how the data was collected, its usage and limitations.
2.1 Datasets
2.1.1 CMIP6
CMIP6. The backbone of the ClimateSet data pipeline is the Coupled Model Intercomparison Project Phase 6 (CMIP6), an archive uniting climate model outputs from numerous sources (Eyring et al., 2016). CMIP6 is used to inform the IPCC Assessment Reports and represents the largest available archive of comparable climate datasets (Petrie et al., 2021; Balaji et al., 2018) with 3.7 million datasets and an expected total size of  $20 - 80\mathrm{PB}$ . The core-data of ClimateSet are specifically climate model outputs of ScenarioMIP (O'Neill et al., 2016). ScenarioMIP contains projections of future
Figure 2: Forcing agent trajectories. Greenhouse gas  $\mathrm{(CO_2}$ $\mathrm{CH}_4$  ), aerosol (BC), and aerosol precursor  $(\mathrm{SO}_2)$  emission trajectories from 2015 - 2100 for different Shared Socio-economic pathways (SSPs).
climate change scenarios from 58 climate models<sup>3</sup>. Each of those climate models provides a physical simulation of how climate changes as a result of a forcing trajectory (SSP scenario) in the decades to come. The climate model receives future GHG and aerosol emission fields as input (see Section 2.1.2), and simulates outputs of climate variables such as temperature, precipitation, wind velocity, and so on. Climate models have projection uncertainties, illustrated also in Fig. 4 which shows the different temperature projections of climate models across different SSP scenarios, while Fig. 3 shows an example of different climate projections for the year 2100. Projection uncertainties arise both from (A) different climate model formulations, and (B) climate model initializations. (A) means that different climate models represent climate processes differently, leading to "inter-model variability" in the outputs. (B) means that one climate model can be initialized differently (an initialization setting is called an "ensemble member"), leading to "intra-model variability". To capture these projection uncertainties, the IPCC and policymakers rely on projections of a set of climate models and ensemble members. Similarly, we curated a dataset that contains the output of multiple climate models and ensemble members to reflect this projection uncertainty.
Specifications. For our core-dataset, we selected 36 climate models from ScenarioMIP that are summarized in Table 1. Of the 58 climate models available in ScenarioMIP we chose only those ones that had (1) monthly frequency, (2) at least a spatial resolution of  $250~km$ , (3) the scenarios SSP1-2.6, SSP2-4.5, SSP3-7.0, and SSP5-8.5 available, resulting in a total of 36 climate models. A list of relevant features is also provided in Table 2. Other features, such as spatial resolution, grids, calendar, and units are synchronized during preprocessing (see Section 2.2). This selection ensures that ClimateSet provides the main scenarios and is spatially and temporally high enough resolved.
Extensions. The dataset can be extended to more climate models, ensemble members, variables, height levels, spatial, and temporal resolution, as long as the requested data is available on the Earth System Grid Federation (ESGF) server  $^{4}$ .
2.1.2 Input4MIPs
Input4MIPs. The Input Datasets for Model Intercomparison Projects (Input4MIPs) $^5$  collect the future emission trajectories of climate forcing agents that are used as input for climate models (Durack et al., 2017). Fig. C.1 in the Appendix shows an example of a GHG emission map. Similarly as climate models do, ClimateSet uses such maps as input data for its climate emulation task. We selected specifically Input4MIPs as it has been endorsed by CMIP6, i.e. it is compatible with ClimateSet's CMIP6 data, and is considered as the best climate model input data available (Durack et al., 2017). The different climate forcing trajectories included in Input4MIPs are based on different SSP scenarios, ranging from "taking the green road" (SSP1), "a rocky road" (SSP3), to "taking the highway" (SSP5). The digits after the "SSPX" term indicate the amount of radiative forcing in  $W / m^2$  expected in 2100. Of the different datasets available in Input4MIPs, ClimateSet uses (1) a forcing dataset including CO $_2$ , CH $_4$ , SO $_2$ , and Black Carbon (BC), by Feng et al. (2020), and (2) the historic open biomass burning emissions dataset by Van Marle et al. (2017).
Specifications. For our core-dataset, we selected four main SSP scenarios (SSP1-2.6, SSP2-4.5, SSP3-7.0, SSP5-8.5), the historical scenario, four climate forcers  $\mathrm{CO}_{2}$ ,  $\mathrm{CH}_4$ ,  $\mathrm{SO}_2$ , and BC. The future trajectory scenarios of those four climate forcers are represented in Fig. 2. Of the 9 scenarios
3Link to current status of ScenarioMIP data
4https://esgf-node.llnl.gov/search/cmip6/
5 Link to current status of Input4MIPs data
Climate Model	Nominal resolution	Ensemble members	Input4MIPs Files
Name	Publication	Historic	Future
ACCESS-CM2	Bi et al. (2020)	250 km	5	all-fires	all-fires
ACCESS-ESM1-5	Ziehn et al. (2020)	250 km	40	all-fires	all-fires
AWI-CM-1-1-MR	Semmler et al. (2020)	100 km	1	all-fires	all-fires
BCC-CSM2-MR	Wu et al. (2021)	100 km	1	all-fires	all-fires
CAMS-CSM1-0	Hao-Ming et al. (2019)	100 km	2	all-fires	all-fires
CAS-ESM2-0	Zhou et al. (2020)	100 km	2	all-fires	all-fires
CESM2	Danabasoglu et al. (2020)	100 km	3	anthro-fires	all-fires
CESM2-WACCM	Danabasoglu et al. (2020)	100 km	5	no-fires	no-fires
CMCC-CM2-SR5	Cherchi et al. (2019)	100 km	1	all-fires	all-fires
CMCC-ESM2	Lovato et al. (2022)	100 km	1	no-fires	no-fires
CNRM-CM6-1	Voldoire et al. (2019)	250 km	10	all-fires	all-fires
CNRM-CM6-1-HR	Voldoire et al. (2019)	100 km	1	all-fires	all-fires
CNRM-ESM2-1	Seférian et al. (2019a)	250 km	10	anthro-fires	anthro-fires
EC-Earth3	Döscher et al. (2022a)	100 km	97	all-fires	all-fires
EC-Earth3-Veg	Döscher et al. (2022a)	100 km	8	anthro-fires	anthro-fires
EC-Earth3-Veg-LR	Döscher et al. (2022a)	250 km	3	anthro-fires	anthro-fires
FGOALS-f3-L	He et al. (2019)	100 km	1	all-fires	all-fires
FGOALS-g3	Pu et al. (2020)	250 km	5	all-fires	all-fires
GFDL-ESM4	Dunne et al. (2020)	100 km	3	no-fires	no-fires
GISS-E2-1-G	Kelley et al. (2020)	250 km	36	all-fires	all-fires
GISS-E2-1-H	Kelley et al. (2020)	250 km	10	all-fires	all-fires
GISS-E2-2-G	Rind et al. (2020)	250 km	5	all-fires	all-fires
IITM-ESM	Krishnan et al. (2021)	250 km	1	all-fires	all-fires
INM-CM4-8	Volodin et al. (2018)	100 km	1	all-fires	all-fires
INM-CM5-0	Volodin and Gritsun (2018)	100 km	5	all-fires	all-fires
IPSL-CM6A-LR	Boucher et al. (2020)	250 km	11	all-fires	all-fires
KACE-1-0-G	Lee et al. (2020)	250 km	3	all-fires	all-fires
MCM-UA-1-0	Stouffer (2019)	250 km	1	all-fires	all-fires
MIROC6	Tatebe et al. (2019)	250 km	50	all-fires	all-fires
MPI-ESM1-2-HR	Gutjahr et al. (2019)	100 km	10	all-fires	all-fires
MPI-ESM1-2-LR	Mauritsen et al. (2019)	250 km	30	anthro-fires	anthro-fires
MRI-ESM2-0	Yukimoto et al. (2019)	100 km	10	anthro-fires	all-fires
NorESM2-LM	Seland et al. (2020)	250 km	13	no-fires	no-fires
NorESM2-MM	Seland et al. (2020)	100 km	2	no-fires	no-fires
TaiESM1	Wang et al. (2021)	100 km	1	anthro-fires	all-fires
UKESM1-0-LL	Sellar et al. (2019)	250 km	17	all-fires	all-fires
Table 1: Climate models included in ClimateSet with related source and original nominal resolution. The ensemble members are the maximum number of ensemble members available for one scenario, i.e. one scenario does not always contain all ensemble members. Similarly, one ensemble member often contains only a subset of the scenarios. The Input4MIPs files column refers to the specific input files needed for a climate model. These input files provided by ClimateSet are representing the corresponding climate forcing emission fields, separated for different fire models (see Section 2.1.2).
Features	CMIP6	Input4MIPs
Variables	temperature, precipitation	CO2, CH4, BC, SO2
Scenarios	historical, SSP1-2.6, SSP2-4.5, SSP3-7.0, SSP5-8.5	historical, SSP1-2.6, SSP2-4.5, SSP3-7.0, SSP5-8.5
Frequency	monthly	monthly, every 10 years
Time length	2015 — 2100	2015 — 2100
Spatial area	global	global
Levels	1 (surface)	1 – 25 (AIR)
Table 2: Features shared among the two original datasets, CMIP6 and Input4MIPs. The features are only representative for SSP scenario data. For historical data, the time length ranges from 1750 - 2015 however, some datasets only provide the subset 1850 - 2014. In terms of frequency, the historical Input4MIPs data has monthly data for every year. The levels for Input4MIPs data noted here refer to the different height levels of the anthropogenic AIR emission fields.
Figure 3: Climate model predictions. Absolute temperature projections for January 2100 under SSP3-7.0 by A) EC-Earth3-Veg, B) CESM2-WACCM, C) MPI-ESM1-2-HR, and D) TaiESM1.
available in Input4MIPs we have chosen the mentioned four because they are part of the five most important SSP scenarios for policy making (Arias et al., 2021). We did not include SSP1-1.9 – the lowest and most optimistic forcing scenario – because not all climate models include SSP1-1.9 and this would have narrowed ClimateSet's CMIP6 dataset significantly. The climate forcers we have chosen are the same used in Watson-Parris et al. (2022)'s ClimateBench.  $\mathrm{CO}_{2}$  is due to its cumulative character (see Appendix C) and high concentration considered as the most important climate forcing factor, followed by  $\mathrm{CH}_4$  with a long lifecycle and high radiative forcing potential (Fig. SPM2 in on Climate Change (IPCC) (2023)).  $\mathrm{SO}_2$  provides a cooling effect, and damages vegetation, while BC decreases the Earth's albedo when deposited and has negative effects on human health. Overall, this selection represents both long-lived GHG  $(\mathrm{CO}_{2}$  and  $\mathrm{CH}_4)$ , and short-lived aerosol and aerosol precursors  $(\mathrm{SO}_2,$  BC). The Feng et al. (2020) dataset provides for each scenario and climate forcer three data-files that represent (1) "anthropogenic", (2) "anthropogenic aircraft", and (3) "open burning" emissions. Since the "open burning" emissions are not available for the historic case in Feng et al. (2020), we supplemented this data with Van Marle et al. (2017)'s dataset. ClimateSet provides all emissions in  $\mathrm{kg~m}^{-2}\mathrm{s}^{-1}$ . Table 2 lists the shared features of the Input4MIPs datasets. Appendix D describes how the historical open burning data should be handled and its dependence on the fire model of climate models (Appendix E). In our preprocessing pipeline the mentioned Input4MIPs datasets are combined appropriately given those considerations, i.e. ClimateSet provides summed up and ready-to-load input emission data.
Extensions. ClimateSet can be extended by additional scenarios (e.g. SSP1-1.9, SSP2-4.5-covid, SSP4-6.0) and climate forcers (e.g. CO,  $\mathrm{H}_2$ ,  $\mathrm{NH}_3$ ). Note, that for additional control (e.g. piControl) and CO2 scenarios (abrupt-4xCO2, 1pctCO2), no additional Input4MIPs data is required. Those scenarios are set "internally" in the climate models and can be retrieved from CMIP6. When extending the ClimateSet's input data, note that the historical data of the desired climate forcer must be available both in Feng et al. (2020)'s and Van Marle et al. (2017)'s dataset.
All data requested through ClimateSet is directly downloaded from ESGF  $^{6}$  (Appendix G) and run through ClimateSet's preprocessing pipeline. The original data from ESGF is not consistent across
$^{6}$ https://esgf-node.llnl.gov/projects/esgf-llnl/
Figure 4: (A) Scenario variance across climate models, and (B) climate model variance across scenarios. The figures show the projected global temperature increase compared to the historical mean (1960 - 1990). For each scenario or model the mean temperature increase is represented as colored line and the standard deviation as area. Note the differences in temperature projections.
different datasets and climate models, and must be preprocessed. The ClimateSet preprocessor is built modularly, as described below, and can be reused and recycled for related datasets.
The Checker uncovers some inconsistencies across different climate models or input datasets. It checks for corruptness of files, variable naming, units, temporal and spatial resolution, and longitude-latitude structure. Based on its output, some of the preprocessing steps can be skipped if not needed.
The Raw Processor for CMIP6 data syncs time-axis, calendars, and height levels. For the Input4MIPs data, it additionally handles the special case of biomass burning data, corrects units, sums over sectors, and creates loadable data files. The raw processor is using Climate Data Operators (CDO) Schulzweida (2022), a command line tool optimized for processing large climate datasets. To our knowledge, this is the fastest way to process the data we have at hand (see also Appendix J).
The Resolution Processor creates the desired spatial and temporal resolution across all data. The spatial remapping can be used to increase or decrease the resolution. The chosen remapping algorithm can be adapted for each variable. On the temporal axis, the processor can be used to aggregate (e.g. from "months" to "years"), interpolate (e.g. from "years" to "months"), and interpolate between "time jumps" (e.g. "monthly data every 10 years" to "monthly data every year"). To make the resolution processor as efficient as possible, we implemented it with CDO. It can also be used separately from the other processors, see Appendix I for further information.
The Structure Processor is mainly used for CMIP6 data to make sure that all files are using the same longitude-latitude structure, vertices and bounds, the same names for variables and dimensions, and to correct units where necessary. The structure processor was implemented with xmip (Busecke and Abernathey, 2020) and follows their guidelines. Note that it is significantly slower than the CDO-implemented modules.
More details and visualizations of ClimateSet's preprocessing-pipelines can be found in Appendix H.
2.3 Usage
Access ClimateSet. Instructions to access and download the core-data can be found on https://climateset.github.io. We provide both the raw data and the final processed data. The latter can be directly used for climate emulation and other climate prediction related tasks.
Extend ClimateSet. To extend ClimateSet, you first use the downloader to retrieve the desired data from ESGF. Then, you build the desired preprocessing pipeline by adapting the configuration files or by stacking the desired modules. Every processing step can be switched on or off.