Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends the previous Shifts Dataset with two datasets sourced from industrial, high-risk applications of high societal importance which feature ubiquitous distributional shift.  The authors also benchmark some baseline algorithms about model accuracy and robustness with these two different datasets.
strength_and_weaknesses: Strength:
1. Building a benchmark to study distribution shift is an important topic and introducing the medical image and real industrial problem into this benchmark is useful. 
2. The authors propose to evaluate both the robustness and uncertainty is reasonable for me. 

Weaknesses:
1. After reading this paper, I am not very clear about the contribution of this paper (including both empirical novelty and technical novelty), especially compared with the previous work shifts Datasets. For me, the two new things are introducing two new datasets and introducing the concept of "both evaluating accuracy and robustness." However, the medical image dataset is a public dataset, and there are also several similar works to study "domain generalization" for medical images, like cardiac image segmentation (https://www.ub.edu/mnms/ and https://www.ub.edu/mnms-2/). For the second one, as the authors mentioned, this dataset is very similar to the weather prediction one in the original shifts Datasets.  It seems that there are not too many "new things" in this paper.

2. The authors want to benchmark the current ML algorithms assessing robustness and uncertainty, while it seems that the author only evaluates some baseline methods, not comprehensively comparing them with various methods. The major contribution is limited to introducing these two different datasets.

3. For the evaluation of uncertainty, I am not clear whether it is really "uncertainty." If my understanding is correct, it is to evaluate the prediction probability (or something like that). I am not sure if it is proper to mention that as "uncertainty."

4. Some previous uncertainty estimations works may need to be mentioned as related work.

clarity,_quality,_novelty_and_reproducibility: The clarity of this paper is clear and the authors clearly present the motivation of this work, the datasets, and the evaluation procedures. While the novelty (empirical novelty) is limited compared with previous shifts datasets work.
summary_of_the_review: In summary, although the studied problem is important, the current version may not have enough contribution (including both empirical and technical novelty) to be accepted at ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces two new datasets having distribution shifts to an existing collection of datasets. The two new datasets are white matter Multiple Sclerosis lesions in 3D magnetic resonance brain images and the estimation of power consumption in marine cargo vessels. They also have baseline results for each dataset which helps to evaluate the model's performance in terms of robustness and uncertainty estimation.
strength_and_weaknesses: Strength(s):
1. The reasons for creating data splits are sound and well-grounded. They also serve the purpose of demonstrating a realistic distributional shift.
2. The motivation for creating and releasing the datasets is clear and sound. The paper also discusses the potential applications/use cases of the datasets.
3. The datasets are well documented with all the necessary details including license and user agreements. 

Weakness(es)/Suggestion(s):
1. It would be nice to compare the brain MRI dataset with existing MS lesion dataset(s) (like MICCAI's MS dataset). Similarly, if available, please do the same for the power consumption tabular dataset.
2. Although it is good to have an exhaustive appendix, it would be a little hard for the readers to switch back and forth between the main text and the appendix. I'm not sure if the appendix can be part of the main paper. If it can, it would ease reading as the hyperlink to appendices would work.
3. Since the authors are hosting a Grande Challenge with a leaderboard, would the evaluation criteria be the same as the one's mentioned in Table 2 (for the segmentation dataset) and Table 3 (for the tabular dataset)?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written with all the necessary dataset details mentioned in the appendices. In terms of reproducibility, the authors have not released code for the baseline models. They are encouraged to release the code. But, the appendices have enough details to reproduce the results.
summary_of_the_review: In summary, since the paper introduces two well-documented datasets which help to evaluate ML algorithms' robustness and uncertainty estimation capabilities, I think it would be a valuable asset to the community. Therefore, my recommendation would be marginally above the acceptance threshold (6).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper expands the recently proposed Shifts Dataset with two datasets sourced from high-risk applications. They consider the tasks of segmentation of white matter Multiple Sclerosis lesions in 3D magnetic resonance brain images and the estimation of power consumption in marine cargo vessels. 
strength_and_weaknesses: **Strength:**
- Any well maintained dataset is very welcome and needed by the community. 
- The dataset/task that has been suggested are also important and relevant,  as the importance of dataset shift in higher-risk application is becoming more clear for the AI community.
- The paper is clearly written, and provided a detailed appendix.

**Weaknesses:**
- The author suggests a permanent leaderboard on grand-challenge as the evaluation set for the brain MRI task, and mentioned it will be possible to evaluate models on it via dockers. No details for further exploration or readiness of such a system are provided to be reviewed by reviewers.
- Using a docker for evaluation can become challenging very fast. I personally prefer .csv submission and I think their are more convenient as it has been used by WILDS leaderboard (https://wilds.stanford.edu/submit/#making-a-submission).
- All of the MRI data is collected in Europe so, there is neither ethnicity related nor disease prevalence shift in the proposed data, and major acquisition or protocol shift would not be observed. Only the scanner type has been changed from one setting to another. However, at least for developing an effective medical image analysis system disease prevalence shift and shifts related to ethnicity are far more important and less explored. 
- There is no discussion around patient attribution and metadata related to fairness. Specifically for the medical applications, and in a new setting, one major concern is methods fairness and running the proper sub-group analysis is required. Without any relevant meta-data such as ethnicity this is not feasible. [There is a limited discussion around gender/age and tumor size change between sets which is not sufficient and the paper does not provide any effective insight or evaluation.]

clarity,_quality,_novelty_and_reproducibility: - The paper is very well-written and clear.
- The application data provided  is not necessarily novel and not the first.
- There is no reproducibility concern as they (will) provide open access data and leader-board.

summary_of_the_review: Contributions around creating of well-maintained and large-scale distribution shift dataset are important and largely welcomed within the community, however, at least for the medical application side, it lacks further evaluation and expansion to provide subgroup meta-data details and addresses concerns related to fairness. Indeed most of the concerns in such a novel and shifted setup are around fairness and performance of this method in sub-groups and long-tails.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose two new datasets for measuring distribution shift. This works builds upon the earlier work of "Shifts 1.0". One of the datasets is a medical dataset, which the authors took a painstaking amount of time to gather (although the data itself was already public); while the other is vessel power estimation. The reviewers noted that creating benchmarks is an important problem; however, they noted the more modest contribution compared to "Shifts 1.0".

The deciding factor, however, was that the reviewers were unable to look at even at a sample of the dataset. While such datasets could be a boon to the community, but unfortunately, without reviewer scrutiny it's hard to know if the dataset contained errors.
justification_for_why_not_higher_score: Limited number of datasets; the medical dataset based on public data; limited contributions relative to the original Shifts paper.
Decision: Reject