Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper argues that the fundamental nature of benchmarking is a limited approach to understanding general problems, even when solving these general problems are the desired outcome for the technologies undergoing benchmark evaluation. The critique offered is strong, but misses some key fundamental criticisms available and avoids taking certain positions which would make its argument stronger (see main review).

I want to love this paper. I really do. However, I worry that, as presented, it runs the risk of being ignored as irrelevant criticism by precisely the audience it should be reaching. So the authors (and other reviewers) should take anything I say which is critical as a suggestion to improve the paper. I'd really like to see it sharpened into the pointed argument I think it can be, and my score reflects my desire to see it so strengthened prior to archival publication.

Finally, I note that a version of this paper was part of the NeurIPS 2020 workshop ML-RSA (ML Retrospectives, Surveys, and Meta-analyses). That non-archival venue doesn't preclude publication in this track, but I'd be curious to see a response from the authors about what they learned workshopping that paper over the past year as a way of validating some of my concerns about reaching the target audience or missing key existing lines of criticism on this topic.
strengths: The core argument, that benchmarks are an imperfect and epistemically limited way to track performance on (especially, but not only) open-world problems, is important, and the paper does a good job presenting it and summarizing both recent and historical literature in the area. In particular, I am pleased to see the concept of construct validity deployed to make this argument. The paper presents and organizes many ideas supporting this argument I had not previously considered or at least had not reduced to words as pointedly as this paper does.

Focusing on a small number of well-understood benchmarks accepted as "leaderboard problems" for open-world ML tasks (object recognition and natural language processing) provides a useful lens through which to bring the presented theory back "down to earth" and connect the arguments to the target audience of benchmark developers and users.
weaknesses: To me, the paper's biggest weakness is that it buys too much into exactly the current ML discourse/hype it means to debunk. This will be a theme across the specific weaknesses I describe below, and I think it can be avoided entirely with some small changes (again, described below).

First, the paper critiques the idea of benchmarks as valid evaluations of progress on general problems. But this is obvious, or at least should be - only the current ML hype makes it non-obvious (that doesn't make the critique unimportant! Only it makes the need for a precise strike at the core problem all the more critical). Benchmarks were always proxies for the performance criterion of interest. Even in the system performance benchmarks of the 1970-1980s, standard benchmarks were not "axiomatic" or descriptors of how a system would handle an untested workload (think, e.g., about the difficulty of interpreting performance benchmarks during Apple's recent transition from Intel processors to Apple-designed ARM-based processors). But they were better proxies for this than current ML benchmarks might be for the generalization problems of interest. So it's all about whether a benchmark is or isn't a good operationalization of the performance measure of interest. I think the paper could pretty easily reframe its critique of benchmarks in this language, claiming and then supporting the claim that existing benchmarks not only _aren't_ good operationalizations of the kind of generalization performance benchmark authors and leaderboard participants say they are, but _can't_ be for the reasons already offered. This provides a nice tie-in to the idea of construct validity used to make the argument already, so I think the subtle shift in language only strengthens the paper's argument and structure. It might be worth at least mentioning (briefly - 1 sentence, perhaps) the related management and political economy issues - why do we use certain benchmarks as proxies? Who chooses this and who benefits from those choices? Does the performance construct of interest make sense at a technical/construct validity level, or is it merely a construct of convenience (as opposed to an operationalization of convenience, as the paper already discusses). The argument at the very bottom of p5 is important, and deserves some expansion - since ML tasks are not necessarily related hierarchically, it is only by seeing the distinction between the benchmark and the performance goal of interest that we can even interpret what benchmark performance means.

Second, I am surprised to see an argument against the use of benchmarks for understanding progress towards general-purpose tasks which does not explicitly mention either the body of literature on generalization error and robustness or the body of literature on understanding epistemic error. Indeed, as far as I can see, neither the term "generalization" nor the term "epistemic error" appears in the paper. But controlling these is something we (at least ought to) teach students to do in month 1 of a machine learning class when we talk about overfitting and stopping criteria for training. Indeed, I kept expecting the paper to say something about whether benchmark performance leads to over-optimistic performance judgements about what happens when a model comes in contact with "real-world" data (and the related problem of creating/managing post-deployment "ground truth" or even just labeled data). It is particularly jarring not to see any discussion of concept drift or other forms of practical drift which are well studied. If anything is going to resonate with what I think the target audience of this paper is, I think it's this argument. A related argument surrounds what gets hidden by "headline numbers": accuracy on its own is a poor proxy for performance - the paper gets near this topic, discussing some recent work about bias and inequitable outcomes in specific cases, but without actually saying the important core idea: that high accuracy might belie highly unbalanced error rates for small classes or otherwise under-report performance for rare cases. Indeed, it would be useful to have one sentence somewhere that just indicates how ML tends toward the mean. This issue is particularly foregrounded for me in Section 4.1.3, where I think a critique of the concept on technical merits would support the already-strong critique offered on a philosophy-of-science basis (this does not diminish the validity of the proffered critique! But also: you don't want your audience to miss the trees for the forest).

Less importantly, I can't tell where the authors stand on the idea of general performance abstractly. Is the argument against benchmarks also an argument against general-purpose models and AGI? It could be, and perhaps the hesitancy to take this position is justifiable. But I think the paper needs to be clearer on this point. Either general performance is possible and benchmarks aren't telling us whether we're making progress on it (I _think_ this is the position, minus explicitly stating the first part?) or general performance is a misguided goal for the reasons stated in the paper (in which case, the paper should more clearly make that case). This may be a difficult suggestion to take action on, since the very concept of "general intelligence" is actually quite slippery for many of the reasons described in the paper. But I think distinguishing between these two cases will help the argument by focusing the criticism either on the goal or on the way the goal is operationalized. This fits into my overarching "theme" in that the task example of "natural language understanding" could be poorly represented by benchmarks like GLUE/SuperGLUE either because the benchmarks do a poor job of testing knowledge about the underlying world (as the paper mentions) or because the task of "understanding" is difficult to operationalize or because it's poorly formed as a problem. Does the paper accept this framing or is it critiquing it? How does the benchmark relate to other approaches that attempt to test this aspect of natural language models, like the Winograd Schema Challenge?

Minor but related to the specific point just made about NLP vs. NLU, the paper is less than clear about whether the relevant task is language _modeling_ or performance on some specified _task_ (I may not know enough about GLUE/SuperGLUE and possibly the answer is obvious if one does, but if so the relevant background needs to be presented!). These are not the same, and one does not imply the other automatically. For example, even if we accept a functional operationalization of "intelligence", a parser which identifies the parse/derivation tree of a sentence in Chomsky Normal Form cannot be said to be intelligent (although it might help as a component of a system that performs some NLP task well). I noted this issue specifically in the first paragraph of 4.1.2

An incredibly minor weakness - I was surprised not to see TREC mentioned in the history of field-defining benchmarks. I recognize that it's not directly related to the problems the paper is attacking head-on, but information retrieval is not _unrelated_ to natural language processing.

Finally, the conceit in the title and conclusion is fantastic and I'm in favor of keeping it. But I think you need to build the bridge for the reader from the metaphor to your arugment. That's easy - you just need to say more explicitly that benchmarks are inside the museum but the real work of the evaluated models is outside (or something to that effect).
correctness: As noted in the "weaknesses" section, my concerns are more about completeness than correctness with two clear exceptions:
1. On p3, the paper notes how to apply construct validity to understand the core critique. Specifically, the paper says "we can think of benchmarks as mirroring the role of the _experimental setting_ in the evaluation process". But there is no experiment in ML, in the sense that there is no "hypothesis" being tested. Thus, any epistemic error is not introduced by the benchmarking process, but by the problem formulation and dataset generation processes, which have already taken place by the time a benchmark is produced or pursued. A good treatment of this issue at both the technical and philosophy-of-science level can be found in Malik, 2020 ("A Hierarchy of Limitations in Machine Learning"). The paper also says here that construct validity should be considered as an external validity issue (which is absolutely correct), but in fact there are important internal validity issues that must also be considered, such as the propensity of benchmark leaderboard chasing to lead to overfitting and poorer generalized performance (an easy technical argument here is to focus on well understood problems with boosting combined with the fact that the winning models in many leaderboard contests - e.g., the Netflix prize and many Kaggle contests - are boosting-based or were before deep learning ate the mind of the community). Malik does a good job grounding the way that standard ML workflows will lead to optimistic performance in such scenarios and relating it back to the difficult problem of bounding epistemic error. Also, a discussion of construct validity could/should engage with the broader idea of the Theory of Measurement (see, for example, Jacobs & Wallach, 2020 ("Measurement and Fairness"); Jacobs, 2021 ("Measurement as Governance of and for Responsible AI"); and a number of recent works on problem formulation in ML such as Amironesi et al., 2021, which has many overlapping authors with this paper! This is obviously an incomplete list of potential citations).

2. On p5, the paper notes the distinction between intensional and extensional specification, a longstanding and important distinction in the domain of formal program specification for the purpose of verification (bounding the specification-implementation gap which always exists for any computational process in at least some form). This is excellent, but I think the translational value could be stronger (e.g., instead of only citing a recent ML work, the authors could also identify that this is an old idea in a related field and gesture broadly in the direction of references on this, such as (I didn't look very hard here, so there might be better cites to be found) Carnap's original 1947 formulation in formal logic or the summary of how it applies in software engineering given by Eden and Kazman, 2003. Additionally, the paper goes on to state that "[i]n machine learning, the tendency is for [extensional specification]". This is absolutely incorrect, but only because it needs to be strengthened. ML is a purely extensional paradigm. Malik, 2020 does a good job explicating this more deeply, but to the extent that there is intensional specification happening in approaching an ML benchmark, it happens during the problem formulation and dataset development stages. I think this needs to be pointed out, at a minimum.
clarity: Small typo-level nitpicks:
* on p1: "due to its" -> "due to their" (agreement)
* The sentence in the last paragraph of 4.1.1 is confusing - I initially read it to mean the opposite of what it says.
relation to prior work: In general, the work is well contextualized. I've offered a few specific citations I think would be useful in places, most notably Malik's 2020 preprint "A Hierarchy of Limitations in Machine Learning", which has a lot to speak to across my suggestions since it covers both the philosophy of science issues the paper engages with and the technical issues (e.g., the natural optimism of cross-validation-based performance measures) in extreme detail.
additional feedback: I reiterate that my goal in writing this review is not to nitpick or critique, but to offer places where the argument can be improved because it is important and deserving of a platform. But I want it to be more definitively able to occupy the space it is creating when it gets there.
review_title: Great, but improvements needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This position paper surveys the history of influential machine learning benchmarks, highlighting their limitations.  It notes that many benchmarks were influenced by the DARPA Common Task Framework, which focused on narrow practical tasks like ASR and MT, but has now evolved to include less practical and harder to measure general objectives.  
strengths: * Extensive and insightful survey of the history of machine learning benchmarks and the dangers of applying them to increasingly general tasks.
* Examines the implications of benchmark design choices, such as the attempts to improve diversity of imagenet through query translation into other languages.

weaknesses: * The paper points out multiple problems, but doesn’t take a position.  

Should the field move back to narrow and practical tasks for unsolved problems or continue to pursue more general objectives?  

How should RecSys benchmarks make it simple for evaluation while being large enough to be representative?

Should an enhanced ImageNet improve diversity?  Along which dimensions?

How can benchmarks be realigned with real world tasks?  
correctness: It is historically accurate as far as I can tell.

clarity: The paper is well written with helpful analogies and memorable stories. 
relation to prior work: It surveys benchmarking across multiple fields.
additional feedback: If the field shouldn't benchmark general capabilities, what should it benchmark?
review_title: History of the benchmarks we use today

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This position paper argues that the standard evaluation practice of "benchmarking" within ML fields encourages over-generalization, loss of (scientific) focus, and broader harm. Its argument involves historical context, which helps situate and support its claims. The paper is very well written. The core contribution is a well-argued call to re-examine how ML fields handle evaluating progress.
strengths: The main strength is that for the points it makes, it does a very thorough job of supporting them. It's an important topic, and one that affects (nearly) anyone who works within ML research.
weaknesses: While the paper does a very good job outlining its core arguments about what is wrong with the current benchmarking paradigm, and it *does* include some suggestions in the appendix, at multiple times the lack of these suggestions in the main paper cause a feeling of "what would be an appropriate alternative?" The inclusion of this in the appendix makes me think it could be included in the main paper, especially if a camera ready has extra pages. However, a reorganization, where some suggestions are provided in each of 4.{1,2,3} would help the forward-looking part of the argument. Sometimes this may be somewhat straightforward based on the appendix, though in other cases it can be difficult. For example, a case study example for both ImageNet and GLUE in 4.1.1 suggesting fixes for the identified limitations would help.
correctness: Yes: see previous box(es).
clarity: Yes: see previous box(es).
relation to prior work: Yes: see previous box(es).
additional feedback: N/A
review_title: Well argued paper, with some reorganization suggested

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In "AI and the Everything in the Whole Wide World Benchmark", the authors assert that AI researchers should not advertise benchmarks as machine learning concepts that can move AI towards AGI nor should people buy into the idea that a benchmark can help achieve general intelligence. Starting with the concept of Grover and the Everything in the World Special, the authors support their thesis by providing a sociohistorical analysis of benchmarks in which the authors argue that the introduction of "general purpose benchmarks" that are a far cry from the original narrow AI task benchmarks with practical real-world tasks impede meaningful progress in the AI field. The authors' apparent purpose is demonstrate that benchmarks are not indicative of general artificial intelligence in order to convince AI researchers to change their framing in benchmarks as general purpose. The intended audience are artificial intelligence researchers at large, and the authors establish a relationship with the audience through their retelling of the history of AI and associated benchmarks as well as the Sesame Street concept. The authors' contribution is not only providing this intervention, but providing ways forward for the ML community.
strengths: The historical background is very well done and very informative. One can tell it is very well researched, and everybody can learn something new from this paper.

The intervention of construct validity is important, and I think it is something that will hopefully be studied more on the AI + society side.

Very good job with all the suggestions for improving benchmarks. :)

weaknesses: The first paragraph should be more tied together.
"He opens the door, only to find himself in the outside world." -- How do the authors of this paper interpret this?
"As a children’s story, Grover’s described situation is meant to be absurd." -- The word absurd doesn't 100% tie in to the afore sentences.
"However, in this paper, we discuss how a similar faulty logic is inherent to recent trends in artificial intelligence" -- You never stated what the logic is. 
In sum, there needs to be more explication and tying together of these concepts.

To strengthen the paper, be careful about making claims that a) have no literature you can currently cite or b) are not supported through argumentation.
correctness: My unsolicited opinions/thoughts:
Computer vision/NLP benchmarks are being tagged as general benchmarks when really the AI community would probably consider that narrow AI. So then, it is really a "general" intelligence in one "cognitive" area which I understand is somewhat wonky to reconcile.

I think it's easier to say that AI researchers should avoid hyping up their benchmark as moving towards a "generalized" intelligence in a specific area (e.g., computer vision) than it is to convince them to stop doing it. In reality, A LOT of structures actually support AI researchers providing more hype than outcome. In particular, a) professors have to fund their research labs, b) industry researchers who have to justify company expenditure, and c) the U.S. government wielding AI in a scientific battle with China. Therefore, there is a lot of incentive to make false promises or put up facades.

I think the philosophy of artificial intelligence from a machine learning researcher's perspective is actually richer than the sole presentation that the whole AI field thinks benchmarks get the field closer to general intelligence. For example, there are papers on how machine learning alone will not get us to general artificial intelligence alone, the misaligned representation of cognitive science in current machine learning that impacts general intelligence, and many researchers will state how dumb AI can actually be and how far away it is from general intelligence. I suppose there are equally more papers hyping up a facade of machine learning as better than it is, but I maintain that these are mostly a result of economic incentives. Even more so, I think there may be AI researchers for whom general intelligence is not a goal at all. Anyway, I mention all that to say that the artificial intelligence community is has varying opinions on this topic and there are varying opinions.

One criticism I have in general about everybody's claim that AGI is the goal of the AI community is that a lot of business would be happy with ML as it stands since it can be used for business analytics to make money, so who would be the stakeholders of AGI and what do they hope to achieve with AGI?

I know there is some discussion in the paper about general purpose NLP benchmarks, but some in the NLP community, particularly Dan Jurafsky, have expressed concerns about the state of benchmarks in NLP (e.g., inadequacy), so I don't think the NLP community has a universal opinion on NLP benchmarks being general purpose about as far as I can tell.
clarity: The paper is not often clear at times. The introduction has a lot of jargon which is not immediately explicated although I respect the page count limitations. 

In 2.1 paragraph 2, I am unsure if the authors are making the claim that the AI and ML fields collectively have the goal to move toward AGI because the following sentence states it is a subset of the community. If there is a collective goal, I would cite literature supporting this claim, so that it does not appear as a broad sweeping statement.

If I'm not mistaken, the argument in the paper asserts that more progress can be made if, among other things (e.g., see section 4 writ large), we stop glamorizing the false promises of general purpose benchmarks, which will allow AI researchers to understand where we really are and potentially how we should proceed given this knowledge of our current state of affairs.
relation to prior work: This paper provides an original contribution by rethinking the way benchmarks are portrayed and how they are approached. It not only thinks through the fallacies of modern benchmarks, but provides ways to potentially improve the state of benchmarking
additional feedback: Overall, the paper is really good, and the recommendations for moving forward with benchmarks provide a lot of pizzazz. Where I'm not 100% sold is the idea that the mentioned benchmarks are "general purpose" in the sense of the overarching goal of AGI rather than general purpose in a narrow AI specialty, even the authors mention that AGI is underspecified. For example, is computer vision one piece of AGI, or do we get to an intelligent agent that can understand its environment another way? Another reason I'm not sold is that the authors could have made the claim that the promise of current benchmarks is overhyped and not rooted in practical task specifications without making the claim that these are general purpose benchmarks and tying them to AGI. The AGI conversation somewhat seems to fall apart very easily (to me because I also read a lot on AGI although I don't care to touch it with a ten foot pole because it raises more questions than it answers) when considering benchmarks for "narrow AI".
review_title: The Fallacy of Benchmarks for General Artificial Intelligence

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This position paper argues that large-scale “general” benchmarks are in fact deceptively limited and criticizes the widespread reliance on such benchmarks as monolithic measures of progress in their respective domains. The paper centers its argument on the ImageNet and GLUE benchmarks. The paper begins with background on research striving for artificial general intelligence and of benchmarking more broadly (which can be traced back to benchmarking computer performance) and goes on to discuss how benchmarking in machine learning began as standardized ways of evaluating performance on narrowly scoped tasks. The authors then argue that benchmarks for general-purpose language understanding and general-purpose object recognition are fundamentally flawed because they are no longer narrowly scoped and falsely claim to be real-world-esque proxies for generality beyond the data provided in said benchmark (which is, by definition, fixed). These claims are supported by arguments that any fixed dataset contains subjective bias, which has been previously shown to be true of the benchmarks in question – ImageNet and GLUE. Many other problems surrounding ImageNet and GLUE are discussed, including limitations in task design, problems with task/dataset collection, a lack of domain knowledge, a lack of context regarding tasks/datasets, limitations in scope, misuse by the community, inappropriate SoTA-chasing, and the problem of influencing the direction of the field away from grounded applications/the popularity of certain types of algorithms. 
strengths: The paper provides many very compelling arguments that monolithic benchmarks should not be so heavily relied upon as measures of progress within subfields of machine learning, and highlights many of the problems with chasing “SoTA” performance on these benchmarks. These contributions are valuable to the machine learning community and should be taken seriously. The authors argue that it’s actually impossible for specific benchmarks to act as perfect proxies for general intelligence (more specifically, as it pertains to vision and language understanding problems), and it's clear that this perspective is very often neglected in the community - particularly, I think, as it pertains more concretely to the publication process. 

The two benchmarks which are at the center of the discussion are two of the most widely adopted means of evaluation for vision and NLP, which are themselves two of the most widely studied domains in machine learning - which makes the scope of the argument in the paper immediately relevant to a very large portion of the machine learning community, and likely indirectly relevant in various ways to the entire machine learning community as new benchmarks are developed for new domains. The authors provide a number of novel arguments while highlighting arguments and analyses made in other works which support their claims. I would like to see the computer vision and NLP communities take these arguments seriously, as I think they are a valuable contribution toward making more meaningful progress in these fields beyond chasing the state-of-the-art. 

Note that while the study provides no quantitative analysis or concrete dataset or benchmark contribution (which would probably be orthogonal to the arguments by the authors anyway), the work is within the scope of the Datasets and Benchmarks track (i.e., “identifying significant problems with existing datasets and their use”). 

weaknesses: While I agree with virtually all of the arguments presented in the paper, after an initial read, I was left somewhat unsure as to what actions an individual researcher proposing a new method could take to mitigate reliance on popular benchmarks falsely claiming generality. It is often the case that researchers are systematically pressured into relying on benchmarks such as ImageNet and GLUE as part of the publication process. 
correctness: The claims in the submission seem correct. 
clarity: This paper was a good read as it is very well-written. 
relation to prior work: The paper clearly identifies and discusses prior work analyzing problems in benchmarking NLP and vision while making broader (novel) points about the use of “general” benchmarks, which separates this paper from prior work. 
additional feedback: -	Please use the latex template designed for the Datasets and Benchmarks track (page 1: “NeurIPS 2020 Workshop: ML Retrospectives, Surveys & Meta-analyses (ML-RSA)” seems to be a holdover from a previous submission). 
-	Missing space in section 3.2 “well-defined kind of generalizability.Furthermore,” 
-	Same in 4.1.1. “are not much more carefully selected.The benchmark”, “(lacking in particular any exploration of pragmatics[Qiu et al., 2020], or the proper handling of negation[Ettinger, 2020])”
review_title: A compelling argument against "general" monolithic benchmarks in vision and NLP

Meta-review: The paper is very clear in its arguments against the over-reliance of machine learning on benchmarking, which it claims as used as inadequate proxies for generalized learning. This core argument is supported by historical background and suggestions for improvement, although some of these suggestions could be moved to the main body of the paper or otherwise reorganized for increased visibility.

The reviewers provided very thorough reviews. They are in agreement that this paper is well done and covers an important topic that is relevant to the track. The authors' responses to the reviews were detailed and the resulting conversation makes it clear that the paper has the potential to spark important discussions in the community.
Decision: Accept