Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a large dataset that facilitates comparisons between research works that propose different machine learning models to replace subroutines in numerical simulations of weather and climate models (specifically, radiative transfer calculations). The goal is to standardize benchmarking across different techniques, which have hitherto been evaluated based on different datasets, setups, and metrics. Secondarily, the authors propose new models not studied in related work, as well as limitations of previously used models and datasets. Thirdly, the authors also identify multiple out-of-distribution test sets, the possibility of building physics-informed models, and the challenge of trading off accuracy versus inference speed.
strengths: Preface: As an emergency reviewer, I have to say this field (i.e. climate/weather) is well outside my domain expertise.

1. *Significance*: To the best of my knowledge subsequent to a brief literature review, it appears that the authors' contributions are as stated---that is, there is indeed an apparent need for standardization across multiple threads of research to better gauge progress and compare different approaches to deploying machine learning models within RT and weather/climate modeling more broadly. This appears in line with several recent such "call-to-arms" works in medicine and reinforcement learning, where having any amount of such standardization is much better than individual works picking their own convenient settings for study.

2. *OOD Test Sets*: Another strength is the presence of explicitly identified OOD test sets, which appear selected to handle (1) distribution shift in atmospheric opacity, (2) forward distribution shift in time, and (3) backward distribution shift in time.
weaknesses: Two questions:

1. *Benchmark Models*: Are there not more interesting models that have been proposed, than the described vanilla MLPs, GCNs, graph networks, and 1D CNNs? In the related work, has anything else been done beyond these, that can also be benchmarked?

2. *Downstream Use*: RMSE and MBE are used. Could the authors comment on how these models are typically used (i.e. is the typical scenario that one or two specific endpoints need to be predicted using a pre-defined metric, or would the models need to "simulate" a wider range of variables for multiple purposes)?
correctness: As an emergency reviewer, the background knowledge (i.e. climate/weather) required to issue a confident assessment of this lies outside my domain expertise.
clarity: The paper is quite well-written; I had no issues with understanding the aims, procedures, contributions, and results.
relation to prior work: To the best of my judgment, the Related Work section appears to span an appropriate range of relatively recent work in this area.
additional feedback: Please provide some clarity into the above questions.
review_title: Emergency Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a new dataset for estimating the atmospheric radiative transfer (RT) with over 10 million samples from a collection of climate conditions spanning over 40 years of duration. The authors divided the dataset into training, validation and test sets and included out-of-distribution test sets to cover special cases: an anomaly (mount Pinatubo eruption), future climate (based on simulations), pre-industrial (1850-1852). In addition, the paper demonstrates the performance of a set of typical manifold regression baselines and discusses their performance.
strengths: - The domain of the data is highly relevant in the current era of climate changes and global warming.
- The authors have done a good job of providing out-of-distribution test sets for testing the robustness of ML regression techniques.
- The paper is very nicely written.
weaknesses: - The documentation is suboptimal and the only available link is to a Google Drive file with a shell script that downloads the data. Thorough documentation, README, tutorials, etc., are naturally expected at this level.
correctness: - The claims are mostly correct. I cannot judge whether the task of the dataset is really the most important challenge for climate science, but my assessment is that the problem is a relevant one.
clarity: The paper is very clearly written.
relation to prior work: Discussed clearly.
additional feedback: N/A
review_title: Interesting Dataset on an Important Real-World Challenge

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors made a new weather/climate dataset by the name of ClimART.
ClimART is a super-large dataset, which has over 10M samples (present, pre-industrial, and future climate conditions). ClimART is based on the Canadian Earth System Model.

Since both climate change and weather prediction are critical problems facing our society, I think the authors studied a very important issue. It will have a great research impact if we can use approximate machine learning (ML) to achieve similar performance as explicitly compute physical processes. A good dataset is very important for this kind of research.

As mentioned by the authors, this dataset also has some interesting properties such as multiple out-of-distribution test sets, underlying domain physics, and a trade-off between accuracy and inference speed.

As mentioned by the authors, their targets do not include radiation output under all-sky conditions (which, besides aerosols, includes clouds). That should be improved. Another problem is how to build consistent interfaces between different climate models and their parameterizations.

strengths: Since both climate change and weather prediction are critical problems facing our society, I think the authors studied a very important issue. It will have a great research impact if we can use approximate machine learning (ML) to achieve similar performance as explicitly compute physical processes. A good dataset is very important for this kind of research.

As mentioned by the authors, this dataset also has some interesting properties such as multiple out-of-distribution test sets, underlying domain physics, and a trade-off between accuracy and inference speed.

weaknesses: As mentioned by the authors, their targets do not include radiation output under all-sky conditions (which, besides aerosols, includes clouds). That should be improved. Another problem is how to build consistent interfaces between different climate models and their parameterizations.
correctness: Good
clarity: Good
relation to prior work: Good
additional feedback: N/A
review_title: A good climate dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposed a large (10M) dataset that addresses the numerical simulations emulation using Machine Learning dealing with the atmospheric radiative transfer calculation. The documentation of the dataset seems very complete and detailed. Provided three subsets of data for the out-of-distribution generalization comparison. 
strengths: Detailed documentation of the dataset in the appendix, detailed description of the dataset and good accessibility of the dataset. Provides a large dataset for the weather and climate model emulation community. Compared a couple of different NN architectures and provided baseline results for this dataset.
weaknesses: I have two major concerns about this work:
1. Clarity and the concise presentation of the dataset details: Although there are lots of different pieces in the generation process of this dataset using the Canadian Earth System Model, simpler language should be used to give an overview of how the data was simulated to the audience who are not well-versed in the numerical simulation in atmospheric radiative transfer community. A lot of insider jargon is used and therefore I highly doubt an average reader in the wider machine learning community would understand this manuscript easily. I have to go deep into the appendix to start gaining an understanding of what the dataset is really about.

2. Contribution and significance to the wider machine learning community: Maybe influence by the above part, the presentation of this manuscript did not show its significance to the wider ML community and how this dataset can help researchers outside this specific realm of numerical simulation of atmospheric radiation to advance their work. For the contributions summarized by the authors in Section 6, none of them strike me as a great use case for this dataset for the broader community. 

A couple of other small questions/concerns:
1. Necessity of using neural emulators: Table 2 shows the speed comparison. If I understood it correctly, the Physics-RT is the numerical ground truth and it actually is the 2nd fastest method if no GPUs are used. As the authors pointed out in the caption, Physics-RT is "not GPU compatible YET", which sort of implies that this has the potential to be parallelized in the future. If that is the case, questions would raise for whether emulators are even useful since the ground truth simulations take a similar magnitude of time.

2. The out-of-distribution testing conditions:  author provided three different out-of-domain testing conditions. I am not sure what the implications are for models to behave differently on these test sets given that regularizations are not 

3. In Figure 3 the error distribution is not visible as they overlap with each other. Also if there is no temporal nor logical reason (for x-axis) for Figure 3 to use a connected line plot. Maybe consider using a bar plot for that?

correctness: I belive the claims made in the submissions are generally correct, the dataset construction is through large simulation physics-based model and are sound.
clarity: As noted in the weaknesses sections, the clarity of this manuscript in the main paper needs some significant improvement, a general ML audience who is not well-versed in the numerical simulation in atmospheric radiation.
relation to prior work: The relationship to prior work is properly discussed. However, some more specific and concrete descriptions of the other publicly available datasets are desired, instead of a single sentence listed their existence and their drawbacks compared to this work.
additional feedback: *Post dicussion period

First of all, thanks for the crafted rebuttal in such a short time! Sorry for the part-missing question as I didn't check thoroughly the content.

After viewing the author's rebuttal (and modified version), I am happy to change my review score to "6: Marginally above acceptance threshold" as they addressed my major concern of necessity of ML emulators. Although I still think improving the simulation code (to make it GPU-friendly) is a more promising goal, as authors pointed out, this dataset is still valuable before that actually happens.
review_title: I am not sure about the impact to the wider community

Meta-review: This paper describes a large scale (>10 mln) dataset of atmospheric radiative transfer data spanning a large time period (40 years) and has various out of distribution test conditions. It also provides a good set of benchmark results. The reviewers agree that this dataset is an important contribution to enable the development of ML methods that help tackle some of the problems with climate change and weather. The paper is well written, and the dataset is well documented. For this reason I recommend acceptance,
Decision: Accept