Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose a new benchmark for uncertainty estimates and robustness to "in-the-wild" distribution shifts, called Shifts. The Shifts benchmark includes three datasets from different modalities and tasks, which are weather prediction on tabular data, machine translation, and vehicle motion prediction on self-driving car data. 
strengths: - The benchmark includes large-scale datasets with "in-the-wild" distribution shifts, and it includes various modalities (tabular, text, self-driving car scenes) and tasks (weather prediction, machine translation, and motion prediction). These modalities and tasks are different from what's typically studied in a benchmark for robustness, so I think the benchmark will be a useful resource for the community.
- The paper is clearly written, with a detailed appendix. 
weaknesses: A few of the datasets consider fairly drastic shifts, which makes me wonder whether the distribution is "in-the-wild" as well as tractable:
- The weather prediction dataset studies a distribution shift from tropical, dry, and temperate climates to snow and polar climates, in addition to shift over time. This distribution shift across the climates seems poorly motivated to me (you'd train on all climates typically), and it also seems very challenging for a weather prediction model to generalize to vastly different climates. Could you explain? In addition, do you have a sense of how much of the difference in performance between eval-in and eval-out is explained by the changes over climates, as opposed to changes over time?
- The translation dataset studies generalizing from "standard" language to atypical language such as slangs. I understand the general motivation, but I have concerns about the tractability of generalizing to atypical language from just standard language. 

Baseline algorithms for improving robustness would be nice to benchmark. The current paper only reports performance of standard models.

#### Minor comments
In the first paragraph of section 3, which introduces the tabular weather prediction dataset, the authors seem to claim that the proposed dataset is appropriate for measuring robustness on tabular data in general, rather than just specifically for the distribution shift, task, and data studied on the dataset, by arguing that there are similar distribution shifts in other tabular data such as financial and medical data. I think this over-claims the applicability of the dataset since it's unclear whether robustness for a certain distribution shift and a dataset transfers to robustness for a different distribution shift and a dataset, even if the modality is the same.

The machine translation dataset is a combination of standard corpora and the MTNT dataset, which is the shifted test set with atypical language. The MTNT paper studies basically this setting and demonstrates that models perform poorly on the MTNT dataset. The contributions for this particular dataset seem somewhat limited. 
correctness: The paper is correct, although there are some parts that could be toned down or be more nuanced as described in other parts of the review.
clarity: The paper was written clearly.
relation to prior work: The authors provide a thorough comparison with related work, both on methods and benchmarks. The authors also claim that their benchmark is novel because it considers a range of modalities and tasks, whereas previous benchmarks consider images and classification tasks. The diversity in modality and tasks are definitely a strength of the Shifts dataset as authors point out, but I think the characterization of prior work could use more nuance since existing benchmarks for distribution shifts also consider modalities and tasks beyond image classification. For example, many standard domain adaptation benchmark consider detection tasks. 
additional feedback: Typo in the Dataset paragraph of Section 4: "evaluation (eva) data"
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper propose a dataset for evaluation of uncertainty estimates and robustness to naturally curated distributional shift. The dataset is collected from an industrial source for three different tasks, namely tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction.
All tasks are affected by real distributional shifts. 
A description of the dataset and the baseline results for all tasks are provided.
strengths: I agree with the authors on the necessity of a large scale dataset in the literature for evaluation of uncertainty estimates and robustness to distributional shift.
Some sections of this paper are very well-written, while some are surprisingly explained very poorly. The data collection process is explained in detail and the data descriptions are very clear.

weaknesses: ## Major Comments:
1) I think literature on domain adaptation/transfer learning/multi-task learning is missing in the current related work review. Including the related literature on these fields would improve first accessibility of the paper and second would make it much easier for researchers in different fields to relate.
2) I am not sure about the ensemble-based approach to uncertainty. What does it mean to train ensemble of ten models on the train data? Please provide more explains on "Single" and "Ens" that appears in Table 1, Table 2, and Table 4.
3) How does the distributional shift in the weather data for example relate to concept drift?
4) What does a normal laptop mean? Please provide a description for it.
5) I do not understand assessment metrics explained in Appendix A. This section in particular is very badly written and it has several grammar mistakes. I cannot indicate all of the mistakes because the submitted template does not have the line numbers.

## Minor Comments: 
1) This paper is submitted in wrong template. The correct template is NeurIPS 2021 Datasets and Benchmarks Track template that is available in https://neurips.cc/Conferences/2021/CallForDatasetsBenchmarks.
2) Page 4-Line 9: “A” should be capital in appendix C.4.
3) Section 4-Dataset-1st Line- "(eval)" instead of "(eva)".
4) Note that the font style of eval/dev is not consistent throughout the paper. Is it on purpose?
5) Typo in Appendix A: 'This represents a a hybrid hu...'. 
6) Figure labels are not matching in font sizes. For example see, Figure 5 (a) and Figure 5 (b).
correctness: Based on my understanding the data is constructed in a sound way. However, the evaluation methods are not clearly explained in the paper. Therefore, I am not sure if performances of the baseline models are evaluated reasonably. 
clarity: The data collection process is clearly explained in detail.
Unfortunately, I do not think the evaluation metrics are explained well. Especially, error-retention curves are not explained clearly. To understand the concept one should read [11] and even there the concept of error-retention curves does not appear directly. I would suggest the authors to directly indicate the definition of error-retention curves in the reference or explain it in detail within the paper for the sake of completeness.
relation to prior work: I think literature on domain adaptation/transfer learning/multi-task learning is missing in the current related work review. Including the related literature on these fields would improve first accessibility of the paper and second would make it much easier for researchers in different fields to relate to.

additional feedback: Please see my comments above.
review_title: Review of Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a series of datasets called *Shifts Dataset*, aiming to evaluate the ML models’ ability in terms of uncertainty estimation and robustness. The main idea is to construct the dataset by shifting the test data from the distribution of training/in-domain. Specifically, the data sources come from the industry services and consist of three tasks in order to fit the real-world distribution including multiple modalities: tabular, text and computer vision. For evaluation of distributional shift, the authors also propose a more general evaluation paradigm considering the correlation between uncertainty estimation and the degree of error. For verifying the effectiveness/correctness of the collected data, the authors also conducted the experiments on *Shifts Dataset* using the trained ensemble baseline models. The results evaluated by the error-retention curves show that the datasets are reasonable.
strengths: 1. The authors release a large-scale dataset for promoting the development of uncertainty and robustness verification of ML models. The data collected from the various application scenarios can support the motivation of this paper, i.e., verify the ML model in the “wild applications”. I think the contribution and impact are enough.


2. The proposed evaluation paradigm jointly assesses the robustness and uncertainty. It alleviates the problems that the previous evaluation methods focus on the specific data partitions instead of examining the correct relationship between error and uncertainty estimation.

weaknesses: 1. The authors did not provide the comparison of the related works. What are the differences in the data characteristics between *Shifts Dataset.* and other data sets?


2. The authors did not give some preliminary analysis of each task. It is unclear how the distributional shift is truly examined by using *Shifts Dataset*. 


3. The authors did not evaluate the different methods of overcoming the distributional shifts using *Shifts Dataset*. Only providing the results of ensemble baselines may not be sound, because this kind of dataset should also distinguish the best/better methods that alleviate the distributional shifts.


4. It is better to provide the human evaluation results for text generation tasks, especially for proposing the new benchmark or evaluation metrics for machine translation. It can help examine the correlation results between human ratings and automated evaluation results.
correctness: The construction of the dataset and the evaluation method look sound. But the caption of Figure 2 is obviously wrong. And I am also curious about why the two curves shown in Figure 2 are very close. Does it mean that the proposed evaluation metric cannot work well under this situation?
clarity: The paper is well written with clear logic. The author has claimed the motivation and the evaluation metric. Then the application results can support the effectiveness of this dataset. But I found some typos in this paper, see the “Additional Feedback” part. 
relation to prior work: The paper adequately cited the prior work. For other concerns regarding the previous work, please refer to the “Weakness” part.
additional feedback: I have some questions about the Section.4 Machine Translation.


1. Do you have plans to support more languages? But I fully understand that collecting the professional translation may be expensive (as depicted in the Appendix), and it is hard to support other languages. 

2. Since the pre-trained model is the most popular paradigm in the current NLP field, do you think *Shifts Dataset* can be used to examine the robustness of pre-trained language models?

3. BLEU cannot evaluate the semantic-level features. Why not introduce more evaluation metrics that correlate well with human judgements, e.g., COMET, BLEURT, BERTScore?

The PDF should not be generated with preprint option. There are typos in this paper. For example, check them:

- Abstract & Introduction: In this work we -> In this work, we
- Section.3.Baselines: appendix -> Appendix
- Section.5.Dataset: appendix -> Appendix
- Section.5.Baselines: appendix -> Appendix
review_title: Great contribution of creating the large-scale distributional shifts dataset. But it lacks further analysis. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents an "in-the-wild" benchmark for evaluating uncertainty and robustness to distribution shift.  The benchmark contains realistic tasks and distribution shifts taken from industrial sources. The benchmark has three parts each part relating to a specific modality: Tabular data, machine translation and self-driving cars.
Instead of looking at robustness and uncertainty measures independently, the paper considers a general paradigm that " uncertainty estimates should correlate with the degree of distributional shift.
strengths: - I very much appreciate considering the correlation between uncertainty measures and robustness. This is something that has been under-looked. 
- Diverse and realistic datasets with real world challenges and distribution shifts. 
- many evaluation metrics have been considered. 
weaknesses: - While there are a lot of details in the supplementary materials, the data splits (dev, eval, dev-in, dev-out, eval-in, eval-out) are not well explained in the main manuscript. I think these splits should be better explained. 
- I appreciate the authors consider a multiple assessment metrics, however, the datasets themselves are not discussed thoroughly as they should have been in the main script. For example what makes in-domain and out-domain is not described for each dataset in the main text. I had to find those explanations in the supplementary material. 
- For each dataset only one baseline and in variants of single and ensemble is presented. I think more baselines and a discussion on the performance of these baselines is needed. 
correctness: The dataset is constructed in a sound way. 
clarity: The paper is generally well written and easy to understand. I believe some parts need to be made more clear which I have described above.
relation to prior work: Yes. The paper mentions that in the presented work the focus is on evaluation of the correlation between uncertainty measure and robustness as opposed to those measures individually. 
additional feedback: All feedbacks have been discussed above.
review_title: A realistic in the wild dataset with diverse tasks

Meta-review: The paper presents a welcome addition of distribution shifts beyond the image classification domain. The reviewers are generally favorable of the paper, hence I recommend acceptance.

For a more comprehensive overview of robustness research in the image domain, I recommend the authors refer the reader to the following robustness evaluations: https://arxiv.org/abs/2007.01434 , https://arxiv.org/abs/2007.00644 , and https://arxiv.org/abs/2007.08558
Decision: Accept