Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a taxonomy of typical failure modes (e.g., implementation variations, incorrect baselines, human baselines, etc) when producing benchmark studies, classifies those failure modes, provides descriptions for each one of the failure modes, and lists and summarizes extensive related work (in supplemental material) of more than one hundred other references as examples of those failure modes. While one would expect that all listed failure modes would be "obvious" for practitioners, the extensive list of related work with benchmark issues shows that, in practice, these failure modes are not obvious.
strengths: - The main strengths of the paper are:
    - taxonomy of benchmark evaluation failure modes
    - extensive summary and classification of related work with those failure modes
    - can be used as a checklist of issues to avoid and, if followed, has the potencial to have large, positive impact in the community.
weaknesses: - The main weaknesses of the paper are:
    - The taxonomy is not complete. For example, a common issue is label leakage (although it could be argued that it fits under "errors in test set construction", none of the three examples there relate to label leakage) and another common issue is incorrect "test setup" (e.g., doing cross-validation on a time-sensitive dataset).
    - The paper "appears" too simple and obvious. But it is not!
correctness: It is hard to accept that the taxonomy is complete/correct. 
clarity: The paper is very clearly written.
relation to prior work: The relation with prior work is clear and a very exhaustive list of related work is mentioned.
additional feedback: No additional feedback
review_title: This paper looks like simple common sense, except it is not simple nor common

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors perform a thorough meta-review to develop a taxonomy of failure modes for evaluating machine learning. They present a categorization and description of numerous failure modes along with many warnings and future directions.
strengths: There is no doubt the paper addresses an important topic with broad impact. The authors have done a lot of work surveying plenty of literature and organizing the results into a usable taxonomy. I especially appreciate the nuanced descriptions of failure modes which sometimes identify an “obvious” problem while also noting that serious issues related to that problem have not been observed yet. Interesting open problems and future directions are proposed throughout.
weaknesses: The paper is extensive, but still feels incomplete at times. Some descriptions of failure modes feel too brief and shallow, which is presumably due to space constraints. In addition, the taxonomy could be refined further. Some minor related comments are given in the additional feedback section.
correctness: The claims are correct and supported by a thorough literature review.
clarity: Yes. The paper is well-written and mostly easy to follow.
relation to prior work: Prior work is clearly discussed.
additional feedback: The descriptions of some failure modes could be clearer and/or more rigorous. I was sometimes relying primarily on the examples illustrating the failure mode to understand it’s definition better.

For 3.1 Algorithms, how do pressures from conferences to release code and consider reproducibility affect this failure mode?

Also 3.1, I would like to see more on the issue undocumented details. So many papers lack key details on every aspect of how experiments are conducted.

3.2 Label Errors is very brief. It may be worth discussing additional issues here such as how label errors relate to bias and fairness.

3.2 starting on Line 153: “Relatedly, [72] find that the removal of poor data from machine translation test sets over time has significantly contributed to the observed performance improvement in scores over the past decade.”
This seems like it should be a separate issue from “Contaminated Data” as it is more specifically related to the benchmark drifting and becoming easier over time.

Where does the following failure mode sit in your taxonomy? Sometimes algorithms perform so well on a benchmark that it appears the benchmark is just too easy (this can also lead to the inability to detect performance differences if multiple algorithms all do really well). One scenario where this occurs is when an old benchmark dataset is repurposed for a newly introduced problem variation. Does this fit somewhere in 3.4 Comparison to inadequate baselines? Or is it more similar to some test set size issue?

Could you speak more to the benefits and drawbacks of the choices made in this paper regarding how the taxonomy is structured (e.g., separating internal and external at the highest level)? In other words, what benefits and drawbacks come with the specific perspective of this work?

I would like to see Section 5 expanded if extra space is given to accepted papers.

I like Figure 2 a lot, but I don’t know if it justifies the space it occupies. Maybe it could just be smaller.
review_title: Thorough analysis that could have more detail in the writeup

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a critical perspective into the mechanism of evaluating machine learning methods. The authors highlight two modes of failure, in terms of internal and external classifications. 
strengths: - The paper raises important and common-sense conclusions on evaluating the performance of algorithms.
- The paper has the potential to ignite a stream of concrete steps in terms of more realistic evaluation metrics for ML models.
- The paper is nicely written.
weaknesses: - My main criticism is that the paper proposes common issues with ML evaluations, that are somehow well known by community experts. This raises some doubts about the impact of this paper.
- Secondly, the authors describe the issue of validity, but the manuscript falls short of giving concrete solutions to concrete problems. 
- Thirdly, the paper is in large sections written with vague arguments with a limited scientific justification of claims. To give an illustration, sec 3.2 "test set size", when saying "use sample sizes too small to detect ...", what exactly is considered "too small", and how do you come up with that conventional term.  
- Finally, the paper has some contradictory sections. While on one hand, the authors criticize the use of simple evaluations based on a single metric, on the other hand, they construct arguments based on single-metric results. E.g. In the paper that claims vanilla matrix factorization outperforms deep models, the results are compared using single-values too, concretely Recall, or NDCG. Same for other examples in 3.4.1.

correctness: The claims are rather well-known and unsurprising. 
clarity: The paper is clearly written.
relation to prior work: I am not well aware of a stream of research that resembles the manuscript, therefore, it is hard to judge how comprehensive is the coverage of prior work.
additional feedback: ---- UPDATE ----

I raised the score after the authors' comments.
review_title: Limited contribution

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper presents a useful conceptual framework for understanding different types of model evaluation failures. The terminology and conceptual distinctions will be helpful in guiding the community's ideas of what evaluation should be.
strengths: This is the first contribution I know of that does the following:
- Clearly define the difference between learning problems and tasks
- Identify evaluation failure types separately for learning problems and for tasks
- For each failure mode, identifies examples of these failures in the ML research literature

It will be a good reference for the ML research community about what failure modes to look out for and how to define their research's contribution (as either an advance on a learning problem or a task). The appendix also offers some guidance on how to avoid some failure modes.

The ethical implications of the paper are positive as it is providing guidance on how to improve research integrity. It will hopefully help to prevent some deployments of badly-performing AI that may have been evaluated inadequately.
weaknesses: N/A
correctness: To my knowledge, yes.
clarity: Yes, it was easy to read.
relation to prior work: In the appendix, the authors have a comprehensive review of prior work relating to evaluation. It's clear that their contribution is a meta-review of all this work that synthesizes lessons across all of them.
additional feedback: I have a minor comment about section 4.4.1 ("Reliance on simple heuristics").

I think the section title doesn't adequately describe the problem. Presumably, if a task could genuinely be solved with simple heuristics, that's not a problem (and maybe it's even good news). The problem that the authors describe in that section is rather that task performance was evaluated on inappropriate datasets, resulting in simple heuristics appearing to succeed when actually, when evaluated on appropriate datasets, they don't. Maybe the failure mode could be re-characterized as reliance on 'inappropriate' heuristics (something along those lines).

This raises a bigger issue about whether a task definition should include notions of what the 'appropriate' heuristics or datasets for it should be. I understand why the authors chose to leave out details like these from the definition of a task. However, it seems like specifying these details could eventually be important if we want to understand what, in the context of a specific task, counts as a "transfer" or a "distribution shift". I get that there's no room to address this in this paper, so I'm offering it as a suggestion for future work.
review_title: Good framework for understanding model evaluation failures

Meta-review: This paper provides a useful taxonomy and list of seemingly obvious failures in ML evaluations. After the author discussion period, all authors gave acceptance scores, and I agree with this suggestion. I encourage the authors to address the criticisms given in the individual reviews, using appendices for any details they did not spell out due to space constraints or other details they would like to add based on the reviewer comments.
Decision: Accept