SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Another case where CO2 data needs additional preprocessing is when control scenarios rather than SSP scenarios are included in the dataset. Example for control scenarios are abrupt-4xCO2 (abrupt quadrupling of CO2 in the atmosphere), piControl (pre-industrial CO2 kept constant). Those scenarios are especially interesting when interventional data is needed since one variable is changed (intervened on) while the others stay constant. For those scenarios, the internal CO2 mass balances of the climate models must be used: The abrupt-4xCO2 scenario is the scenario of quadrupling the current climate-model-internal CO2 amount. The CO2 mass balances can be downloaded for each climate model from Earth System Grid Federation system (ESGF).
C.1 CO2 Emission Map
An example of a CO2 emission map is shown in Fig. 8.
D Open Burning and Biomass Burning Data
The emission datasets usually contain three different types of data: 1) Anthropogenic emissions, 2) Aircraft emissions, and 3) "fire" emissions. This data is collected across two different datasets, one for the historical emissions (Van Marle et al., 2017) and one for the future emissions (Feng et al., 2020). Due to different naming conventions in Van Marle et al. (2017) and Feng et al. (2020), the "fire emissions" are called "openburning emissions" for the future case, and "biomassburning emissions" in the past case. We use both names here to distinguish between the two datasets, because - in contrast to the anthropogenic and aircraft emissions - the openburning and biomassburning data need different types of treatment.
All three data types (1-3) have a spatial and temporal dimension, and all three of them separate the emissions on another dimension: (1) The anthropogenic emission files contain different sectors, (2) the aircraft emissions files have different height levels, and (3) the "fire" emissions consist of different types of fires. However, in contrast to (1) and (2), the "fire" emissions in the datasets contain only one level, i.e. the different fire types are not directly encoded in the datasets. While the sectors and levels of the anthropogenic and the aircraft emission files are simply summed up, we actually need to separate between the different types of "fire emissions" depending on the fire model of each climate model (see Appendix E).
To retrieve individual numbers for the different types of fire emissions, the percentage file that are provided alongside with the original datasets can be used. Those percentage files are available on the
Figure 8: Input4MIP's projected global anthropogenic and aircraft CO2 emissions for January 2100 in the SSP3-7.0. The emissions are summed over all sectors. Transportation paths become visible in the CO2 emission map as well as CO2 emissions across land masses.
Input4MIPs archive on ESGF. The following sectors exist for those files, and are different for the historical and future data:
Biomassburning:
- Savanna, grassland, and shrubland fires
- Boreal forest fires
Temperate forest fires
- Deforestation and degradation
- Peatland fires
- Agricultural waste burning
Openburning:
- Agricultural waste burning on fields
- Forest burning
- Grassland burning
- Peat burning
The preprocessing pipeline provided here computes the overall fire emission maps and separates between three final cases: (1) anthropogenic fire emissions only, (2) all fire emission sources, (3) no fire emissions at all. Note that for CO2 only one type exists (no "fire" emissions), as further described in Appendix E. Which of those three files should be used in the other GHG cases, depends on the fire model of the climate model and is listed in Table 3. The subsequent section analyses in detail which types need to be included for each climate models.
E Fire Models
The internal fire model of each climate model determines which GHG emission fields (Lasslop et al., 2020) should be used as input for the climate model. For example, if a climate model contains an extensive fire model that already models peatland burning, we cannot provide peatland burning emissions as input to the model – the emissions would be “counted” twice. Hence, it is necessary to evaluate for each climate model which fire model it is using, what the model is capable of, and which fire emissions should consequently be provided as input. Table 3 provides an overview of all the models, which fire model they are using and which types of fires need to be included in their input data. In the preprocessing pipeline this data is summarized to anthropogenic fire emissions (“anthro-fires”), all fire emissions (“all-fires”), and no fire emissions (“no-fires”). The latter can be used if a fire model is capable of modeling all fires such as the NorESM2. Matching the right GHG emission inputs to the climate models output is definitely good practice, however, we want to mention that in the single-emulator case, ML models are expected to be adequate for mapping from simple emission files (i.e. ignoring fire emissions) to their climatic response.
Model	Land Model	Historical fire emission types	Future emission types	Type of fire model
No fire model	All other ESMs	None	All	All	No fire model
Historic + future fire model	CESM2-WACCM	CLM5	None	None	w/ anthro.
CNRM-ESM2-1	SURFEXv8.0 (ISBA)	Deforestation + Agricultural	Deforestation + Agriculture	w/o anthro.
CMCC-ESM2	CLM4.5	None	None	w/ anthro.
EC-Earth3-Veg	LPJ-GUESSv4	Deforestation + Agricultural	Deforestation + Agriculture	w/o anthro.
EC-Earth3-Veg-LR	LPJ-GUESSv4	Deforestation + Agricultural	Deforestation + Agriculture	w/o anthro.
MPI-ESM1-2-LR	JSBACH3.20	Deforestation + Agricultural	Deforestation + Agriculture	w/o anthro.
NorESM2-LM	CLM5	None	None	w/ anthro.
NorESM2-MM	CLM5	None	None	w/ anthro.
GFDL-ESM4	LM4.1	None	None	w/ anthro.
Historic fire model	TaiESM1	CLM4	Deforestation + Agricultural	All	w/o anthro.
CESM2	CLM5	None	All	w/ anthro.
MRI-ESM (2.0)	HAL1.0	Deforestation + Agricultural	All	w/o anthro.
Table 3: Fire Models with their corresponding Historical and Future emissions type. The type of fire model used by each climate model is listed in the last column. The historic fire models must use the "all" fire emission types and future emission types.
We retrieved the information about the fire models from a combination of different sources. Yu et al. (2022) describes the fire models of most the fire models we are using, while the TaiESM stems from Lasslop et al. (2020). However, the information provided there is not explicit about which fire types exactly must be include from the openburning / biomassburning files. Teckentrup et al. (2019); Rabin et al. (2017) explicitly state how different fire models treat cropland, pasture, and deforestation fire. Additionally, we referred to Seférian et al. (2019b) for CNRM-ESM2-1, Ward et al. (2018) for GFDL-ESM4, Yukimoto et al. (2019) for MRI-ESM2, Lindeskog et al. (2013) and Spessa et al. (2013) and Döscher et al. (2022b) for the EC-Earth3-Veg models. We could not find any information about the land (HAL 1.0) or fire model of the MRI-ESM2 model. However, in the meta information of their data (e.g. on esgf), they describe the "Carbon Mass Flux into Atmosphere Due to CO2 emissions from Fire Excluding Land-Use Change [kgC m-2 s-1]". From that we infer that no anthropogenic fire is modelled here. A detailed report on how we compiled the fire model information is available on request.
F Similarities Within and Across Climate Models
Similarities within climate models. When training on multiple – instead of single – ensemble member of climate models, weighting between the different ensemble members must be considered: Some models such as EC-Earth3-Veg contain up to 97 ensemble members while others contain only 1 ensemble member. Training an ML model with the complete dataset without weighting the ensemble members would skew the results heavily towards those climate models with many ensemble members. Furthermore, some ensemble members are closer to each other than others and providing this information to the ML models could potentially improve their performance.
Similarities across climate models. The following similarities can occur:
(A) Climate models from different institutes share the same sub-models
(e.g. ACCESS-CM2 uses the same atmospheric model as HadGEM3 models);
(B) Climate models are newer versions of themselves, stemming from the same institute
(e.g. NorESM1 (CMIP5) and NorESM2 (CMIP6));
(C) Climate models are implemented for different resolutions, with differing physics implementations to ensure resolving the relevant processes
(e.g. NorESM2-LM (100 km) and NorESM2-MM (250 km)).
Similar climate model names (e.g. GISS-E2-1-G, GISS-E2-1-H, GISS-E2-2-G) usually indicate that the same model is run with different sub-models for atmosphere, ocean, land, etc. (case (A)).
In future work, we would like to encode similarities within and across climate models in the super-emulator, for example by providing an additional input vector encoding the submodels of the climate model.
G Downloader
We present a ready-to use downloader class that forms the first step of creating a custom climate data dataset in an intuitive and script-based fashion.
The Downloader class can be prompted with a set of properties / heuristic to select them (for an example, see Fig. 9. Acting on those, the Downloader will automatically interact with the ESGF nodes to narrow down the search space and obtain the data if available.
Variables and experiments have to be fixed for all data. The version can be nailed down to a specific version e.g. '20170519' or alternatively, can be set to 'latest' in which case always the latest available version of the specified data files will be selected for download. For data coming from the CMIP6 project, the model ID has to be set additionally. For CMIP6, either all available ensemble members can be considered, alternatively, the user can the ensemble member ID or restrict the number of members to be considered.
The resulting search space will then be searched and constraint for the remaining properties. If the specified default nominal resolution, frequency or grid label are not available, the Downloader will download the first available alternative.
The Downloader communicates with the data nodes of the Earth System Grid Federation system (ESGF) via ESGF PyClient.
G.1 Earth System Grid Federation (ESGF)
The Earth System Grid Federation (ESGF) is a partnership of climate modelling centres dedicated to supporting climate research by making an effort to provide access to the distributed climate model data collected by the CMIP projects, which exceeds hundreds of petabytes. The data, hosted on several servers all around the world, can be searched over a web-based platform.
Further documentation on ESGF's web presence can be found here: ESGF User Support
G.2 ESGF Pyclient
As the ESGF's web presence proves to be quite slow and inflexible when it comes to broadly searching the database in order to craft custom datasets, and more overly cannot be easily incorporated in an automatic pipeline as it is dependent on manual selection, we resolve to make use of a python package that hooks onto the search nodes and interfaces with the ESGF Search API. This open-source package, named ESGF-Pyclient Documentation, allows querying the database via python scripting.
G.3 Data Structure
The Downloader will download and store the raw data files separately for each unique pair of specifications, whereas the resulting files span over one year each, with the number of data points per file determined by the chosen temporal resolutions. See Fig. 10 for an illustration of the folder
class Downloader:
    class handling the downloading of the data. It communicates with the esgf nodes to search and download the specified data.
    def __init__(self, model: str = "NorESM2-LM", # default as in ClimateBench experiments: List[str] = [
        "historical",
        "ssp370",
        "hist-GHG",
        "piControl",
        "ssp434",
        "ssp126",
    ], # sub-selection of ClimateBench default
    vars: List[str] = ["tas", "pr", "S02", "BC"]
    data_dir: str = ".../tmp/data/",
    max_ensemble_members: int = 10, #max ensemble members
    ensemble_members: List[str] = None #preferred ensemble members used, if None not considered
):
loader  $=$  Downloader  $(\ast \ast$  kwargs) downloader download_from_model() downloader download_raw_input()
Figure 9: The Downloader class: The Downloader can be instantiated with the default arguments or be prompted with a new set of specifications. Downloading CMIP data from a model and downloading input4mips data are handled by separate functions, each of which can also be prompted with a new set of specifications to allow for flexibility.
structure that will be created in the data directory pointed to by the 'data_dir' variable that the Downloader was initialized with.
H Processing Pipelines
In the following, the specific pipelines are described that were used here to create the core dataset of ClimateSet. The different modules of the processing pipelines can be shuffled around and adapted as needed. The three main modules are:
- The checker (checks for corruptness, inconsistencies in units, variable namings, and similar);
- The resolution processor (spatial and temporal aggregations and interpolations, see Appendix I)
- The raw processor (corrects names, units, calendars, time axis, etc.)
The modules are partially different for the Input4MIPs and CMIP6 dataset, e.g. the raw processor of the CMIP6 dataset contains both an internal processor and a xmip Busecke and Abernathey (2020) processor, whereas the Input4MIPs pipeline can only use the processing tools provided by us so far. The details are described further in Fig. 11 and Fig. 12.
H.1 Input4MIPs Processing Pipeline
The flowchart of the processing pipeline is as shown in Fig.11.
Figure 10: Raw Data Structure: Raw data will be stored separately for each unique pair of specifications, with one file per year.
H.2 CMIP6 Processing Pipeline
The flowchart of the processing pipeline is as shown in Fig. 12.
I Resolution Preprocessing
The resolution preprocessing covers both the spatial and the temporal domain, and the user can both interpolate or aggregate the data at hand. The resolution preprocessor was implemented with CDO (Schulzweida, 2022) since - to our knowledge - it is the fastest tool for remapping netcdf files. Both the resolution preprocessor and the raw preprocessor can also be extended with "National Center of Atmospheric Research Command Language" (NCL by NCAR (2019)) commands if CDO does not provide a desired functionality since NCL is more expressive than CDO. Python tools require loading the data, e.g. with xarray which takes a considerable amount of time, and CDO profits from multiprocessing and pre-weight calculations. In the following the different types of resolution processing are described that we implemented in ClimateSet with CDO. Our python code directly calls the CDO commands via "subprocess". The resolution processor can also be used separately from the rest of ClimateSet if considered helpful.
I.1 Spatial Resolution Preprocessing
The spatial resolution preprocessor uses the CDO command "remap" which can be used for both aggregation and interpolation. Essentially, it regrids a given netcdf file. The file can be either remapped according to a given example (e.g. the NorESM grid), or according to a target gridfile. Please refer to our GitHub repo for an example of a grid or to the CDO documentation (Schulzweida, 2022). Usually, such a file contains the variables "gridtype", "xsize", "ysize", "xfirst", "yfirst", "xinc", and "yinc". If an example file is given, the spatial resolution preprocessor will pick a random file from the directory that should be processed, assuming that all files in that directory have the same grid. It will then calculate weights for this file, and the weights can then be re-used for the rest of the directory. This accelerates the resolution processing a lot. The "remap" function of CDO works bidirectional, i.e. both aggregation and interpolation is handled this way. Beware, that also climate models with the right resolution, still need to be remapped since many models have different grids
Figure 11: Flowchart of the Input4MIPs processing pipeline. The input is first repaired  $\mathrm{(CO_2}$  files,  $\mathrm{CH}_4$  files, correct biomassburning names). Subsequently, the data is checked. Anthropogenic fire emission files are created. Historical biomassburning files are aggregated to  $50~\mathrm{km}$ . Raw preprocessing is applied (correcting calendar, time-axis, and units, and summing levels/sectors). For the future files a temporal interpolation is applied to retrieve annual files (instead of decadal files). The different emission files are summed up (anthropogenic, aircraft, fire emissions) - with three options (anthropogenic fires, all fires, no fires). All emission maps are then aggregated to  $250~\mathrm{km}$  (same as climate model's resolutions). In the end loadable data is created, with a choice of receiving a spatial map, global totals, monthly or annual data.
Figure 12: Flowchart of the CMIP6 processing pipeline. The checker evaluates if files are corrupt, units, names, or similar are inconsistent. The "corrections module" makes sure time-axis, calendars, and levels are aligned across all models. Afterwards, the model is aggregated to  $250\mathrm{km}$  or the grid is adapted to  $250\mathrm{km}$  grid of the NorESM2 model. Lastly, the xmip processor is applied to get the same structured, numpy-friendly, format across all climate models.
and different version of the same resolution. Remapping is necessary for all climate models to make sure that they are operating on the same grid.
Different algorithms can be used to interpolate to a different resolution. Those algorithms can be set in "res_PROCESSing.params.json". For example, we choose bilinear interpolation for temperature files, and first order conservative interpolation for precipitation files.
I.2 Temporal Resolution Preprocessing
Temporal Aggregation A given file can be aggregated along the temporal axis to a desired time unit. So far the units "hour", "day", "month", and "year" are supported. It requires that all timesteps are set to the first of the original time unit. Alternatively, the aggregation can happen over a given number of timesteps.
Temporal Interpolation The temporal interpolation can have two different forms. 1) Files can be interpolated that have e.g. monthly data, but only every 10 years. In that case the "interpolate_jump" function interpolates bilinearly the missing years. 2) Files can be interpolated to different time units. This is currently supported for "seconds", "minutes", "hours", "days", "months", and "years". The interpolation can either happen to another unit or by choosing how many timesteps should be interpolated (i.e. how many new timesteps should be created between two given timesteps). All temporal interpolation is linear. For further information refer to CDOs documentation (https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf).
J Accelerate ClimateSet
The following measures can be applied to accelerate ClimateSet's preprocessing:
1. Using the multi-thread function of the CDO-implemented processors (resolution & raw preprocessor);
2. Waiving the checker and/or the structure processor;
3. Supplying the resolution processor with an example resolution map.
(1) only works on machines that allow multi-threading for CDO, e.g. on HPCs.
(2) is accelerating the process because the checker and the structure processor (xmip)(Busecke and Abernathey, 2020) are not implemented with CDO. Since python tools such as xmip(Busecke and Abernathey, 2020) are computationally not as optimized as CDO is, those modules are significantly slower. While the checker can be waived, the structural preprocessor can only be waived if the selected climate models use the same naming and structure across their dimensions. We would like to implement multi-threading for the checker and structure processor later on, so those two processors are not a bottleneck for the overall acceleration of the preprocessing.
(3) Is already the default. For ClimateSet a  $250\mathrm{km}$  NorESM2-LM spatial map is used which can be found in the "meta" directory of the dataset. Using an example map accelerates the process by re-using the weights needed for remapping: Each file with the same longitude-latitude grid can use the same weights file for computing the new grid.
K ML Models' Implementation Details
K.1 ClimaX
ClimaX stands for a foundation model that has recently emerged for tackling a wide range of weather and climate modelling tasks. With its remarkable capability to handle diverse spatial and temporal resolutions, as well as high-dimensional data, ClimaX sets itself apart by offering forecasting abilities for various lead-times and settings. It has not only achieved state-of-the-art (SOTA) or near-SOTA performance but has also demonstrated computational feasibility across multiple forecasting and climate modelling domains, including ClimateBench (Nguyen et al., 2023).
One of CimaX's key attributes is its capacity to train on heterogeneous datasets encompassing different variables with distinct physical grounding and spatio-temporal coverage. As a foundation model, CimaX undergoes a pre-training phase on CMIP6 data, employing a self-supervised approach. The pre-training process incorporates historical projections drawn from five different climate models, including a set of both surface and atmospheric (i.e. across several pressure layers) variables, such as wind speeds, temperature, and humidity. This foundational pre-training enables subsequent finetuning for various specific tasks.
Architecture: ClimaX extends the vision transformer architecture (Dosovitskiy et al., 2021) with novel encoding and aggregation blocks to accommodate the intricacies of weather and climate modelling and weather forecasting. Notably, it employs variable tokenization, treating each variable in the input as image-like data represented by a spatial map. These variables are then considered as separate tokens, enabling comprehensive analysis.
To tackle the challenge of scaling computations with an increasing number of input variables, CimaX implements a variable aggregation technique. It performs a cross-attention operation for each spatial position across all variables, resulting in a unified token sequence that captures the holistic representation of the input data. The transformed data is then processed by a vision transformer in conjunction with a prediction head.
Distinguishing Features from the Original Implementation: In adapting CimaX to ClimateBench, the authors introduced several modifications to their model differing from the weather forecasting stream. Firstly, only specific parts of the pre-trained weights from the foundation model were retained. This involved preserving and either freezing or finetuning the attention layers. Additionally, the ClimateBench implementation included a temporal aggregation model. This model processed each year of the 10-year input sequence through tokenization, aggregation, and attention layers, followed by global average pooling over the spatial map and cross-attention across time. The resulting embedded output was then fed into a linear attention head, mapping it to a spatial grid for a single variable and single time step only.
In contrast, our approach to climate projection diverges as we designed our task in a sequence-to-sequence manner. Thus, we maintain the same selection of pre-trained weights but exclude spatial map pooling and temporal aggregation, as our objective is to obtain predictions for each time-step in the input sequence. Additionally, we are modelling several output variables, i.e. temperature and precipitation, at the same time. Therefore, we utilize the original nonlinear decoder with a prediction head that maps the output to several variables across the spatial grid.