What’s the Meaning of Superhuman Performance in Today’s NLU?
Year: 2023

Abstract:
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.

heading: 1 Introduction
text: In recent years, research in the field of Natural Language Processing (NLP) has been driven by a frantic race to reach the top spot in popular benchmarks (Wang et al., 2018, 2019; Lai et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019). Typically the race takes the shape of a rapid cycle of parameter tuning updates by several teams, communicating their results using a shared leaderboard. Not infrequently, systems achieve better-than-human performance on several tasks (see Figure 1). Yet what does this level of performance really mean for NLP? The impressive capabilities of ChatGPT make this question even more urgent.
It is relatively easy to outperform humans with simple procedural tasks like arithmetic and extreme memory-intensive tasks involving vast amounts of data. But most tasks involving natural language typically require knowledge and inference. Do high-performing NLP algorithms really have (super)human capabilities? Or are the metrics that deliver these scores suspect?
Given the impact of claiming superhuman performance, it is important for researchers to understand exactly what is going on. As many in NLP have experienced, the false sense of accomplishment of superhuman performance often leads to an abrupt disappointment when a supposedly superb system is applied to realistic data in a real-world situation. By propounding unrealistic claims, NLP researchers harm themselves and the field as a whole.
12471
Some problems result from the metrics used to assess systems, which are invariably automated, and the data these metrics employ, which may be skewed in various ways. The metrics might give incomplete or biased reports of performance, or simply not apply in certain situations.
Other problems arise from the ‘boundary parameters’ that shape the task, which are usually not adequately reflected in the evaluation metric, very seldom in the kinds of automated metrics used in leaderboards. Specifically, the correctness of a task setup and its dataset instances should not be taken for granted. Also, humans and machines are often evaluated under different conditions, such as the level and type of knowledge provided to perform the task and the test items used to compute performance.
Yet other problems result from the nature of leaderboard-based evaluation. Despite the obvious benefit of driving development through competition with little human effort, these evaluations typically do not foster understanding. Teams driven by a rapid evaluation turnaround cycle in a competitive mode tend to focus more on quantitative results than on error analyses which aim at improving awareness of their problem. As currently set up, benchmarks and comparisons do not incentivize a deeper understanding of the systems’ performance, nor do they foster research geared towards producing automatic explanations: it is one thing to produce a numerical system performance score, but quite another to rate the adequacy and understandability of an explanation.
In this paper, we explore the interpretation of the superhuman performance and the utility of leaderboards, discuss how human performance is actually computed in a range of tasks, and how requirements differ for humans and automatic systems across tasks. We hope to encourage leaderboard creators to be more circumspect when setting up their challenges and provide clear ‘boundary conditions’ and descriptions of the limitations of their evaluations.

heading: 2 Popular Leaderboards Are Saturated
text: Leaderboard-based evaluation has become a popular practice in NLP1 (Wang et al., 2018, 2019; Lai
1We mainly focus on leaderboards for Natural Language Understanding (NLU) due to their commonalities, but note that Natural Language Generation (NLG) tasks such as machine translation, where the most accepted ‘leaderboards’ are based on human scoring, have their own set of issues (e.g. Läubli et al., 2020).
et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019). The goal of these leaderboards is to encourage the development of systems capable of solving certain tasks and to measure their progress by comparing the best systems against humans. Their great success has led many researchers to focus on just the proposed tasks, resulting in a rapid saturation of the scores which, in many tasks, are equal to or greater than those obtained by humans. As a consequence, many have attributed superhuman performance to such systems, and some tasks have been deemed solved. However, while systems in some areas of AI are compared with the best possible humans, e.g. IBM Deep Blue vs. Garry Kasparov in chess2 or IBM Watson vs. Ken Jennings and Brad Rutter in the Jeopardy! quiz show3, NLP researchers often naively or vaguely estimate the “human baseline”, assuming it is a uniform and accepted term of comparison, an established level that systems need to simply beat. In this section we provide a broad overview of existing NLP benchmarks, with a particular focus on NLU leaderboards where human baselines are outperformed by systems, and then show that the construction of such benchmarks is fraught with inconsistencies.
The SuperGLUE benchmark (Wang et al., 2019) is a well-known framework for evaluating research towards general-purpose language understanding models for English. It is a collection of 10 language understanding tasks built on existing public datasets, together with private test data, an evaluation server, and a single-number performance metric. In many tasks, humans are outperformed by the best-scoring systems, often by a large margin, ranking 8th in the current overall leaderboard. Likewise, the SuperGLUE predecessor, i.e. GLUE (Wang et al., 2018), was built to measure advances in NLU, and the systems’ scores quickly saturated the benchmark, thereby sliding the human baseline down to the 23rd position in the ranking.
The RACE benchmark (Lai et al., 2017) was designed specifically for evaluating NLP models on a set of challenging reading comprehension tasks, such as Question Answering (QA) and text summarization. It consists of a large dataset of more than 28,000 multiple-choice questions, which are drawn from middle and high school problems extracted from English examinations in China. These
2https://en.wikipedia.org/wiki/Deep_Blue_ versus_Garry_Kasparov
3https://en.wikipedia.org/wiki/IBM_Watson# Jeopardy!
questions cover a wide range of topics and require the ability to reason, understand context, and make inferences based on the provided text. Human baselines rank 21st on the public leaderboard, with a gap of almost 20 points compared to the bestscoring system. Similarly, the SQuAD2.0 benchmark (Rajpurkar et al., 2018) is another popular collection of reading comprehension questions and answers based on Wikipedia articles. The questions are created by crowdworkers and the answers are a portion of text from the corresponding article. The peculiar difference of this benchmark compared to SQuAD1.1 (Rajpurkar et al., 2016) is that some of the questions may not have answers, hence systems are required to learn to abstain as well. Again, the human baseline is placed in low positions of the ranking, reaching just the 30th place. Another notable, related benchmark is CoQA (Reddy et al., 2019), a large-scale dataset focused on Conversational QA systems. In this task, humans rank 7th, with a gap of 2 points from the top system.
Quite different results are observed when moving to a cross-lingual scenario or when systems are required to perform mathematical and logical reasoning. In particular, XTREME (Hu et al., 2020) is a benchmark for cross-lingual transfer evaluation that covers dozens of languages spanning 12 language families, and includes 9 tasks that require reasoning about different levels of syntax or semantics. In this case, the human baselines beat the systems in all tasks, with an overall score 8 points higher than that of the best-performing system. XTREME has been succeeded by XTREME-R (Ruder et al., 2021), a more challenging multilingual benchmark that covers 14 language families and includes 10 tasks, and similar results have been observed. Furthermore, when systems are evaluated over MathQA (Amini et al., 2019) inputs, i.e. mathematical questions in the form of natural language, systems perform poorly compared to humans. Indeed, humans achieve an accuracy of 87% while systems only reach 54.2%. Since systems are still far from human-level performance in these benchmarks, they are out of the scope of our study. However, the highlighted gaps should encourage further research in these areas.
An alternative view on system evaluation is presented by the adversarial evaluation framework (Nie et al., 2020; Kiela et al., 2021), where the evaluation is performed through an iterative “humanand-model-in-the-loop” annotation process. Hu-
mans are asked to inspect the model output and produce adversarial examples that target specific model weaknesses. The evaluation target is thus a moving goalpost, as opposed to the static targets of most other benchmarks, which saturate quickly. The Dynabench benchmark (Kiela et al., 2021) embraces this approach, incorporating tasks such as NLI, QA, sentiment analysis and hate speech detection. It provides a platform for the annotators to examine model output and create adversarial examples. At the time of writing, most of the tasks within Dynabench do not report human performance, however. Exceptions include the adversarial visual QA task (Sheng et al., 2021), where the proposed adversarial examples are solved by other humans and agreement is computed in terms of accuracy. Model performance in this setting falls far below the human performance.
Using more challenging examples for model evaluation, and possibly subsequent re-training, is an appealing approach, likely to strengthen the models with respect to the aspects that the examples target. The caveat is, however, that special care needs to be taken to avoid loss of generality. The annotation of adversarial examples directly depends on the behavior of the model (or set of models) under consideration; the addition of a large number of adversarial examples will likely change the data distribution by potentially overemphasizing rare events; finally, the annotators may focus on a small number of properties of the model, thus “overfitting” the models.
Although there are many other popular NLP benchmarks to be investigated, e.g. XGLUE (Liang et al., 2020) and SentiBench (Ribeiro et al., 2016), we limit our review to those benchmarks in which human performance is provided and that can therefore help us answer the main question of this paper concerning the meaning of superhuman performance.

heading: 3 Human Baselines Are Not Reliable
text: As discussed above, many NLU benchmarks are saturated (cf. Figure 1). Here we dive deeper into some of them, identify the reasons for their quick saturation, and discuss whether it is fair to claim superhuman performance of state-of-the-art models. In particular, we study SuperGLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2016, 2018), as the representatives for general language understanding and reading comprehension, respectively.

heading: 3.1 SuperGLUE
text: For each of the ten tasks in SuperGLUE, human performance is provided and systems are compared against it. Specifically, for four of these tasks – Word in Context (WiC, Pilehvar and CamachoCollados, 2019), Multi-Sentence Reading Comprehension (MultiRC, Khashabi et al., 2018), Recognizing Textual Entailment (RTE, Nangia and Bowman, 2019), Reading Comprehension with Commonsense Knowledge (ReCoRD, Zhang et al., 2018) – human performance is computed by the authors of the corresponding papers, while for the remaining tasks4 humans are evaluated by the creators of the SuperGLUE benchmark.
WiC For this lexical-semantic task, four sets of 100 instances with an overlap of 50 instances between two of the annotators were randomly sampled from the test set. Each set was then assigned to an annotator, resulting in a total of 300 annotated instances. The annotators were not lexicographers and were not provided with sense distinctions to resemble the more difficult scenario for unsupervised models (cf. Appendix C). A final score of 80% was then obtained by averaging the individual scores achieved by the humans on the 4 sets (between 79% and 82%).
MultiRC In the Multi-Sentence Reading Comprehension task, four native-speaker annotators tagged the entire test set of 166 instances. Human performance was obtained by combining the individual predictions of the different annotators via majority voting.
RTE To establish the human performance on the RTE task, annotators were hired through the Hybrid data collection platform. Each annotator first completed a short training procedure, during which they were provided with task-specific guidelines and annotated 20 random examples from the dev set. Only annotators with ≥ 65% accuracy qualified for the main task. 500 examples were randomly taken from the test set and, for each instance, the final label was obtained by combining 5 different annotations via majority voting, reporting a final accuracy of 93.6%. The average pay rate was $17/hr for the main task, and $7.6/hr for training.
4Boolean Questions (BoolQ), Commitment Bank (CB), Choice of Plausible Alternatives (COPA), Winograd Schema Challenge (WSC), Broadcoverage Diagnostics (AX-b), Winogender Schema Diagnostics (AX-g).
ReCoRD For the Reading Comprehension with Commonsense Knowledge task, 2,257 crowdworkers were hired through the Amazon Mechanical Turk platform (AMT). For first-time workers, the HIT5 assignments were accompanied with guidelines. Crowdworkers were required to have ≥ 50 HITs with a 95% HIT acceptance rate and to be located in the USA, Canada, or UK. The average pay rate was $3.6/hr.
Other SuperGLUE Tasks For the six remaining tasks, the SuperGLUE authors hired crowdworkers through AMT: the annotators first completed a short training phase where 30 random development set examples were provided for each task. Only workers who completed 5 HITs during training with performance at, or above, the median across all workers were admitted to the main task. Human performance was estimated on a random set of 100 test samples from each task, by applying majority voting on the annotations of 5 workers. During both phases, workers had access to task-specific guidelines, with a pay rate of $23.75/hr.

heading: 3.2 SQuAD
text: In SQuAD1.1 (Rajpurkar et al., 2016), the researchers obtained ≥ 3 answers from human workers for each question in the dev and test sets, and estimated human performance by using only one of the answers as the “human prediction” and the remaining answers as “ground truth” for comparison. Specifically, workers were shown the questions and relevant paragraphs of an article and were asked to select the shortest paragraph span that answered the question. They were advised to complete 5 questions in 2 minutes with a $9/hr pay rate.
In SQuAD2.0 (Rajpurkar et al., 2018), instead, the authors collected multiple answers for each question (i.e. 4.8 answers, on average) and selected the final human prediction by majority voting. The answers were collected by providing annotators with a paragraph and its associated questions – unanswerable and answerable ones shuffled together – and asking them either to highlight the answer in the paragraph or to mark the question as unanswerable. They were asked to spend one minute per question with a $10.50/hr pay rate.
5A Human Intelligence Task, or HIT, is a question that needs an answer. A HIT represents a single, self-contained, virtual task that a worker can work on, submit an answer, and collect a reward for completing.

heading: 3.3 Issues
text: Comparing the performance of the five best systems against humans on SuperGLUE (Table 1), it is immediately apparent that the machines outperform humans on 6 out of 10 tasks, and often by a large margin (e.g. 7.8 F1 points on MultiRC). Similarly, best systems substantially outperform humans on SQuAD1.1 and SQuAD2.0, with a margin of 8.3 and 4.1 points in exact match accuracy, respectively. Interestingly, (zero-shot) ChatGPT performs poorly compared to both human baselines and best-performing (fine-tuned) systems. Indeed, compared to the scores reported in Table 1, it achieves just 86.8 on BoolQ, 89.3 on CB, 58.0 on COPA, 85.2 on RTE and 64.6 on WiC as measured by Qin et al. (2023) and Kocoń et al. (2023). Additionally, Kocoń et al. (2023) also showed that ChatGPT performs 20% worse than state-of-theart systems on the SQuAD2.0 benchmark, and demonstrated that it is, on average, 25% worse than specialized ML systems on a wide array of tasks. Hence it is not relevant for our study as its performance is still far from human-level.
What does appear relevant, instead, are the extremely high, often superhuman, scores achieved by specialized systems. Nevertheless, notwithstanding such scores, in the above-mentioned benchmarks multiple factors make human-tosystem comparisons unfair because they limit human performance while facilitating systems. We list them in the remainder of this section.
Apples and oranges The most glaring problem is that, on almost all SuperGLUE tasks, humans and systems are evaluated on different test sets (i.e. on a small subset vs. the full test set). Specifically, in the WiC and RTE tasks, humans are assessed on 21.4% and 16.6% of the test set (i.e. 300 out of 1400 and 500 out of 3000 instances), respectively. Similarly, in the other SuperGLUE tasks humans are evaluated on a subset of 100 instances per task, which – in the worst case of the BoolQ dataset –
amounts to just 3% of the test set. We provide more details in Appendix B.
Human evaluation metrics Different metrics are used to assess humans across tasks. While most of the tasks employ majority voting, WiC merely averages the scores achieved by humans on 4 small distinct subsets. In SQuAD1.1, humans are evaluated by comparing the tags of an arbitrary annotator against those of two other “ground truth” annotators, thereby likely underestimating the final score.
Heterogeneous and unknown pay rates Pay rates varied considerably across the various tasks, ranging from undeclared pay rates to $23.75/hr. Low and mediocre wages, as in ReCoRD and SQuAD, may have contributed to suboptimal human performance: the $3.6/hr pay rate on ReCoRD could be one of the reasons for the large gap between systems and humans, while the unknown pay rate for MultiRC might explain the 18.2% human error rate on this binary classification task.
Ground-truth data quality We identified several errors and ambiguous instances in the goldstandard datasets, some of which we report in Table 2. Importantly, we note that, while systems can find spurious correlations between training and evaluation instances, and therefore provide the correct answer without clear evidence, humans cannot find such correlations, or otherwise may genuinely disagree on what the correct answer is. We elaborate on this point in Appendix A, by analyzing several examples per task, as well as in Appendix C, where we report the results of an ad hoc study concerning the WiC dataset.
Information about annotators and instructions Details of the annotator pool (e.g. the number of annotators, their background and nationality, etc.) are often omitted. Similarly, the absence of training instructions and task guidelines raises questions about the quality of the training phase, if any.

heading: 4 Setups Favor Misleading Comparisons
text: Summarizing the above observations, we find four main sources of human-to-system comparison error. These correspond to the following key aspects of the evaluation process: system performance, the evaluation data, the measurement process, and humans themselves. We discuss each in turn.

heading: 4.1 Systems: Right for the Wrong Reasons
text: Across a variety of tasks, Søgaard et al. (2021) report that random train-test splits consistently overestimate model performance: randomization at the sentence level reduces discrepancies between training and test sets as sentences from the same documents occur in both. Non-random standard splits also bring the danger of inadvertent, communitywide overfitting (Gorman and Bedrick, 2019)6.
In natural language inference (NLI), multiple authors have found that BERT achieves what looks like near-human accuracy by exploiting idiosyncrasies of the data: they are “right for the wrong reasons” (McCoy et al., 2019; Niven and Kao, 2019). Here much of BERT’s success is attributed to its ability to learn syntactic and lexical cues for inference, which happen to be mostly correct on the original test data. However, these cues do not actually support such inferences on adversarial datasets, taking BERT’s accuracy to chance level or below.
Poliak et al. (2018) report an even more extreme
6Even with hidden test sets, such overfitting could happen via publication bias or just the faster spread of methods used in “state-of-the-art” systems (as measured on a static test set).
case of being “right for the wrong reason”: several NLI datasets support what they call hypothesisonly models, which perform surprisingly well without exposure to the premise (Gururangan et al., 2018), e.g. outperforming the majority-class baseline. Poliak et al. (2018) attribute this to statistical irregularities in the data (often single words indicating negation), caused by obvious annotation strategies chosen by crowdworkers who were not stimulated enough to come up with more creative ways to produce contradictions or entailments. Along the same lines, Parmar et al. (2023) recently identified instruction bias in 14 NLU benchmarks. Specifically, they found that this phenomenon is evident in most of these datasets, showing that ∼73% of instruction examples, on average, share a few clear bias patterns, and that models often fail to generalize beyond such patterns.

heading: 4.2 Data: Monolithicity Obscures Details
text: A further cause of systematic performance overestimation is that test sets include instances with varied, often unfathomable, levels of difficulty, so the exact reported accuracy will be a weighted average that depends directly on the mixture of easy and hard instances in the test data. The composition of train-test splits can thus make a big difference (Swayamdipta et al., 2020).
In QA, Lewis et al. (2021) investigated the traintest splits of several popular datasets. They found that there can be substantial overlap between the answers and even the questions of the training and test sets. The evaluation results differed greatly be-
tween seen and unseen questions and answers; for instance, the exact-match accuracy of BART as a closed-book QA system on WebQuestions dropped from 76% to below 2% when neither the question nor the answer were ever seen during training.
In semantic parsing, seq2seq models such as BART and T5 are very accurate when evaluated in-domain on broad-coverage parsing tasks, e.g. Bevilacqua et al. (2021a). Yao and Koller (2022) report that their accuracy drops to close to zero on test subsets that require them to generalize to language that is structurally more complex than the training data. This is corroborated when constructing hard sets, i.e. train-test splits based on compositional generalization, forcing the accuracy of seq2seq models below 20% (Bogin et al., 2022).

heading: 4.3 Measurement: Automation Is Limiting
text: A third key limitation of current evaluations, and especially existing leaderboards, is that they assume that the performance of a model can be measured automatically. While this has not been discussed very much in NLU, in other communities it has long been recognized that automatic evaluations are imperfect proxies of human judgments (Novikova et al., 2017). Machine translation papers report BLEU scores because they are drastically cheaper to calculate than the cost to collect human judgments about the fluency and adequacy of text; but one system that outperforms another on BLEU is not necessarily judged better by humans (CallisonBurch et al., 2006; Popel et al., 2020). While recent automatic metrics correlate better with human judgments (Kocmi et al., 2021), automatic evaluation has consistently been found problematic when comparing top-performing systems (Ma et al., 2019). Similarly, Byron et al. (2009) recommend crowdsourced evaluations to counter the inadequacy of automated evaluation for NLG.
The deeper issue with our reliance on automated evaluations is that they constrain the tasks on which we can evaluate systems. New shared tasks and datasets are specifically designed to make automated evaluations possible. However, many skills that show competent language use cannot easily be approximated by automatic measures (Dunietz et al., 2020): there are entire facets of language competence that are systematically out of scope for the tasks we design. One might argue that these are the most interesting parts of the actual mastery of language. Therefore, human-level performance
on automatically-evaluated tasks does not equate to human-level performance on real language use.

heading: 4.4 Humans: They Often Disagree
text: The final and possibly most problematic issue with system evaluation lies in the creation of the evaluation data itself. Common evaluation methodology assumes that there exists a single ground-truth for evaluation. This is a great oversimplification. We argue that evaluation should be conducted with reference to different groups of annotators to go beyond a one-dimensional performance score, to reflect multiple possible ‘truths’.
A great deal depends on how annotators are instructed to produce the data. It is well-known that human annotation quality may suffer from errors resulting from lack of attention given to the task, both by annotators themselves and by the annotation managers, often resulting from the need to drive annotation costs down (Mishra and Gorana, 2021). Importantly, however, human label variation does not always reflect poor annotation. Label variation can also result from stimulus characteristics or the context in which annotation occurs, including factors like the identity of the annotators, their background, and world knowledge. Plank (2022) identifies three main reasons for human label variation, namely annotator disagreement, subjectivity (multiple possible perspectives) and underspecfication (multiple plausible answers). While subjectivity (e.g., due to cultural differences) is a clear issue in tasks like hate speech detection (Davani et al., 2021), inherent disagreements, ambiguous sentence meaning, underspecification in guidelines and annotator behavior have been identified not only in fine-grained Word Sense Disambiguation tasks (Navigli, 2009), but even in NLI (Pavlick and Kwiatkowski, 2019; Zhang and de Marneffe, 2021; Jiang and de Marneffe, 2022).
While the standard approach for training and evaluating NLP systems is to use a single gold label for each example, a growing body of work deals with multiple labels by varying model training in various ways: different aggregation methods (Paun et al., 2018), training on the distributional labels (Potts et al., 2020), learning from agreement signals (Plank et al., 2014), or modeling the annotators (Geva et al., 2019; Sap et al., 2022; Gordon et al., 2022). Recently, Basile et al. (2021) proposed extending this approach to evaluation. Fully benefiting from this extension requires releasing
annotator characteristics labels (Prabhakaran et al., 2021), including socio-demographic information, and carefully documenting the annotation process (Gebru et al., 2018; Bender and Friedman, 2018; Geiger et al., 2020).
Annotator disagreement often results from differences across individuals – not just in NLP but also in fields such as cognitive science (Levinson, 2012) and psycholinguistics (Kidd et al., 2018). This phenomenon is often underestimated, since experiments tend to focus on a homogeneous sub-sample of the human population (Henrich et al., 2010). Annotators have different natural biases (Reidsma and op den Akker, 2008), and models often learn annotator-specific signals that are not generalizable (Geva et al., 2019), including opinion, personality (Sap et al., 2022) and culture (Hershcovich et al., 2022), but also different interpretation of guidelines (Hansen and Søgaard, 2021; Parmar et al., 2022). To deal with subjectivity, Rottger et al. (2022) recently introduced two contrasting data annotation paradigms: the descriptive and prescriptive ones. While the former encourages annotator subjectivity by capturing and modelling different beliefs, the latter, instead, discourages it and enforces annotators to encode one specific belief, formulated in the annotation guidelines. Depending on the downstream application of the dataset, one paradigm can be more suitable than the other, but neither paradigm is inherently superior. However, dataset annotators should explicitly aim for one of the two paradigms to facilitate the intended downstream use of their dataset, and to document, for the benefit of others, how exactly their dataset was annotated.
In conclusion, without more attention to the “science of annotation”, the methodological laxity in today’s dataset creation will continue to foster inaccurate estimations of human performance.

heading: 5 Humans Can Explain Their Answers
text: When performing language tasks, humans are capable of explaining why they provided a given answer. Thus, when models are claimed to attain human-level language understanding, we can reasonably expect to be able to elicit explanations from them. This has proven highly challenging, however, which casts further doubts on such claims.
Why do we need explanations? At the level of an individual problem instance, explanations can help users assess whether to trust a given answer. At the level of a system, they help regulators and
the general public to assess whether, or in what contexts, a system is safe to use, e.g. by uncovering unwanted biases or by revealing that the system relies on outdated knowledge. In the context of this paper, explanations can help NLP researchers understand the behaviour of their systems, e.g. to make sure that models are right for the right reasons (McCoy et al., 2019; Niven and Kao, 2019), or to uncover some of the shortcuts that the model may have learned (Geirhos et al., 2020), as discussed in §4.1. Indeed, the absence of explanations can lead researchers astray. For example, in a prize-winning paper, Kaushik and Lipton (2018) analysed several state-of-the-art QA systems and found that they simply classified the best matching answer using their pre-stored knowledge about each question candidate, without performing any ‘reading’. None of the papers in which these QA systems were introduced had considered this possibility.
What are the challenges? While the importance of explanations is well-understood, progress has been hampered by various issues. One issue is that the evaluation of system-generated explanations is hard to automate (§4.3). Another issue is that it is not always clear what form the explanations should take. For tasks such as sentiment classification, it may be sufficient to highlight which words from the input text have mostly affected a given prediction. However, for NLI and QA, providing informative explanations can be challenging, even for humans. This can be observed by inspecting datasets that include human explanations (Camburu et al., 2018; Rajani et al., 2019; Aggarwal et al., 2021). Finally, system-generated explanations are typically not faithful, i.e. they do not necessarily reflect the process used by the model. For instance, Camburu et al. (2020) found that models can generate contradictory explanations for a given input.

heading: 6 Recommendations
text: Based on the findings of the previous sections, we argue that current claims regarding superhuman performance are not adequately grounded, leading to unjustified hype. Here we provide a set of recommendations aimed at making comparisons between humans and machines fairer and more reliable.
Do not favor machines against humans Various actions can be taken to set a level playing field between humans and machines, so as to provide a more realistic sense of their actual performance:
1. Avoid using the same documents for training and evaluation (§4.1): in fact, using the same documents inherently reduces discrepancies across splits (Gorman and Bedrick, 2019), encouraging models to learn specific idiosyncrasies that appear in both (McCoy et al., 2019).

heading: 2. Balance easy and hard test set items (§4.2),
text: so as to report accuracies and enable analyses based on their difficulty level.
3. Occasionally refresh test sets (§2), as suggested by recent trends in adversarial evaluation (Kiela et al., 2021).
4. Complement automatic evaluations with human judgements (§4.3), so as to compare systems with humans on facets of language use that cannot be evaluated automatically.
5. Adequately train and motivate humans (§3.3), aiming to increase the quality of human annotations through a solid training process and higher pay, in a sense mimicking the effort taken in improving systems.
Make human performance evaluation transparent and reproducible We suggest carrying out an effort similar to systems’ reproducibility for evaluating humans as well, including:
1. Document the annotator pool composition (§3.3), by explicitly answering the following questions: how many annotators were hired? Following what process? What is their cultural background, nationality, languages and areas of expertise? What is their hourly pay rate?
2. Specify the annotation process (§3.3 and §4.4): it is important to state how many annotators were assigned to each instance, the training process they underwent, the guidelines they received (and how such guidelines were fine-tuned), and the metrics used to compute the overall human performance (averaging individual scores, majority voting, etc.).
3. Provide individual annotations (§4.4): this allows recalculation of overall human performance whenever new metrics are tried, identifying the best metrics, calculating the scores of the best and worst annotators, the gap between the two, and the correlation between
metrics and individual annotators – all aspects that the annotation community has long advocated. Importantly, the availability of individual answers, combined with the annotators’ profiles, opens the door to deeper investigations about why and when humans disagree.
Increase annotation accountability Multiple measures can be implemented to make both systems and benchmarks more reliable, transparent and informative:
1. Include explanations in your benchmark (§5): requiring annotators to provide the rationale behind their choices implicitly enforces them to devote more attention to the annotation task, thus yielding higher-quality and more consistent annotations. Moreover, annotators’ explanations can be used to study subjectivity, and discover (and mark) ambiguous instances.
2. Let systems produce explanations (§5): before claiming superhuman performance, it is important that, similarly to humans, systems can explain the inferences behind their predictions. This is key both for increasing systems’ credibility and for discovering their limitations. However, it is not impossible that a system will produce the right answer with the wrong explanation, or vice versa. For this reason, we believe that a system must be able to provide explanations that support its answers without knowing that answer a priori, inferring the answer based on its knowledge.

heading: 7 Conclusions
text: We have discussed the distressing tendency of many NLP researchers to claim superhuman performance for their systems, and outlined why such claims are not (yet) grounded. We identified problems with evaluation data, evaluation measures and methodology, system understandability, and the human creation of data, all of which contribute to our conclusion.
As a final remark, with this paper we hope to make the reader more suspicious and rigorous when claims about “superhuman” performance are made, and, more importantly, to incentivize benchmark creators to address current limitations and design more solid and transparent benchmarks that will advance our scientific understanding of NLP systems and humans.

heading: 8 Limitations
text: In this paper, we have unearthed a variety of problems present in current evaluation benchmarks that favor systems over humans, or that simply make such comparisons unfair. We conclude that there is no real evidence to claim that today’s language models possess superhuman performance. However, without empirical results obtained under the right setups, we cannot even claim the opposite, namely that humans are still better than systems. We leave such demonstrations for future work.
Additionally, while a good portion of the NLP research effort is devoted to natural language generation (NLG) tasks (which includes MT), here we provide only some pointers to NLG/MT. Indeed, as discussed in Section 4.3, these problems exist in the NLG universe as well, but, due to space constraints, we limit our analysis to NLU tasks.

heading: Acknowledgments
text: This work grew out of a brainstorming session held at the Rome Workshop on Ten Years of BabelNet in July 2022.7
We gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union’s Horizon 2020 research and innovation programme, and the support of the PNRR MUR project PE0000013-FAIR.
This work has been carried out while Simone Tedeschi was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome. The contribution of Jan Hajič has been supported by the LUSyD project No. GX20-16819X, funded by the Czech Science Foundation, and has used resources provided by the LRI LINDAT/CLARIAH-CZ, project LM2023062 funded by the MŠMT CR. The DFKI contribution to this work is supported by the LT-BRIDGE project, which has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 952194. Rico Sennrich was funded by the Swiss National Science Foundation (grant no. 176727).
7http://mousse-project.org/events/ event-a5f3r5.html

heading: A Ground Truth Data Quality
text: In Table 2, we reported one problematic example for each of the most crucial tasks in the SuperGLUE benchmark. Here we comment on those examples and provide additional problematic cases which we identified by manually inspecting the datasets. Some of these cases appear recurrently.
BoolQ The example in Table 2 is blatantly wrong, as it explicitly says that shower gel is an effective and perfectly acceptable substitute to shampoo, hence the label should be FALSE. We provide more errors in Table 3. Specifically, we believe that some of these examples are wrongly annotated, ambiguous, or highly misleading. In the first example, from the premise, it seems that some scientists and ornithologists differentiate between doves and pigeons, so the answer might be subjective, and therefore ambiguous. In the second example, instead, it seems there is no evidence that a red back spider bite can kill a human being, but the answer is TRUE. Similarly to the first case, in the third example the premise states that in most prisons possession of mobile phones is not allowed, thus
the answer might change depending on the prison. In the fourth example, the fact that all the fingers have a similar vein structure does not mean that the ring finger is not connected to the heart, on the contrary, this reinforces the hypothesis. Finally, while two or more players can be substituted in a football game, the same player cannot be substituted twice.
CommitmentBank In the CB example reported in Table 2 we have that A does not know whether they’ve ever made a movie and, indeed, asks if B thinks they have. Therefore, we cannot conclude that the movie was never made, and the answer should be NEUTRAL. By inspecting the dataset, we discovered that its instances follow standard patterns that can be easily learned by machines, but,
at the same time, confuse humans. Indeed, most of the time, the entailment is TRUE when a fragment of the hypothesis appears (as an exact match) in the premise (see the second and third examples in Table 5). To the contrary, the entailment is FALSE when the same text fragment appears negated either in the premise or in the hypothesis, e.g. preceded by don’t think or similar, standard constructs (see the first, fourth and fifth examples in Table 5). However, as argued before, the mere fact of not thinking that a thing is true does not necessarily imply that thing is not true.
MultiRC Regarding the MultiRC example of Table 2, in this case, the error is in the candidate answers. Specifically, two candidate answers are
equivalent, i.e. Mass of the object and The object’s mass, but they are labeled differently, namely with TRUE and FALSE tags, respectively. In Table 4 we provide additional errors for this task. Specifically, in the first example, the question explicitly asks “Who are the two that Guty and Bruno are planning to murder?”, but the possible answers are i) Miriam and Bruno’s father, ii) Bruno’s father and iii) Guy’s wife. Although, by design, MultiRC creators explicitly state that multiple answers can be correct, answers are judged independently, so it would not be valid to form a correct answer by combining ii) and iii). These cases are very frequent in MultiRC and might have negatively affected human performance. Furthermore, typos in the questions and/or paragraphs (i.e. Guty, in this case) might have further limited their scores. In the second example, the ground truth answers are “2002” and
“2007”. However, while “2007” can be inferred by adding 82 years (i.e. the age at which Albert Bandura received the Grawemeyer award) to his birth date (i.e. 1925), “2002” is a wrong answer. Indeed, the paragraph says that “A 2002 survey ranked Bandura as the fourth most-frequently cited psychologist of all time”, but there is no evidence that he received the award in 2002.
Finally, in the third example, from the paragraph, it is clear that the German art collector Cornelius Gurlitt passed away at the age of 81. However, there are three errors in the possible answers for this entry. First, “At the age of 81” and “81” are labeled as TRUE and FALSE, respectively. Second, “80 years old” is labeled as TRUE, hence contradicting the first answer. Finally, “80” is labeled as FALSE further contradicting the penultimate answer.
RTE In the RTE example (Table 2), the specific premise regarding Pacific countries is not sufficient to entail the general hypothesis, thus the answer should be FALSE. We provide more examples in Table 6. In particular, in some of them, we believe that the label is incorrect (examples 1, 3 and 5), or at least highly misleading, while in some others we
think that not enough information is provided to entail the hypothesis (examples 2 and 4).
WiC In the WiC example provided in Table 2, the word criticism is used with the same meaning in the two contexts, namely disapproval expressed by pointing out faults or shortcomings according to WordNet. We provide additional ambiguous or wrongly annotated examples in Table 7.
By inspecting the WiC dataset, it is immediately apparent that, in many negative examples, the semantic gap between the meanings of the same lemma in the two contexts is very narrow. Although
such cases are difficult even for machines, we posit that for humans (especially if sense distinctions are not provided and annotators are not lexicographers, as in WiC) they are way more difficult.
SQuAD For the SQuAD dataset, studies about errors in the annotations have already been performed by Rodriguez et al. (2021) through automatic error detection methods. Specifically, they annotated SQuAD items by discriminability, difficulty, and Item Response Theory (IRT) prediction errors, and discovered that items with negative discriminability, or where IRT’s prediction is wrong, have a much higher rate of annotation error, i.e. they are often “flawed” or “wrong”. We believe that tools for error detection (Klie et al., 2022) should play a key role in the improvement of existing benchmarks and in the creation of new ones.
Finally, still related to the topic of wronglyannotated or ambiguous instances in the datasets, Nangia and Bowman (2019) performed an interesting study on the GLUE benchmark. In order to investigate the effect of these instances, they looked at human performance when there is 5-way annotator agreement. Using unanimous agreement has the effect of filtering out examples for which: i) the annotation guidelines supplied do not provide clear advice, and ii) humans understand the expectations of the task but find the example genuinely difficult or uncertain. They discovered that this widened the gap between humans and systems by more than 3 points on average, hence confirming the hypothesis that humans were often penalized by unclear guidelines or other factors. Even more interestingly, they found that in some tasks, when systems are evaluated on the unanimous subsets they obtain lower scores compared to those obtained on the entire test sets containing wrong or ambiguous instances, hence suggesting that systems had learned specific idiosyncrasies appearing in both training and test sets (Section 4.1).

heading: B Apples and Oranges
text: In Section 3.3, the first issue that we pointed out was that, on almost all SuperGLUE tasks, humans and machines are evaluated on different test sets (i.e. on a small subset vs. the full test set). Here, we provide more details (see Table 8). Specifically, it can be observed that only 3 out of 10 tasks are fully annotated by humans, while for the remaining 7 tasks only a small portion is annotated, ranging from 3% to 40% of the full dataset size.

heading: C The Copenhagen Experiment
text: In this Section, we report on an experiment that was conducted in Copenhagen at the Danish Language Technology Conference in 20228. The main goal was to verify the claim that contextual sentence examples from open lexical resources, such as those used to create the WiC dataset, i.e. WordNet, VerbNet and Wiktionary, “constitute a reliable base for the construction of the dataset, as they are curated in a way to be clearly distinguishable across different senses of the word” (Pilehvar and CamachoCollados, 2019). Based on this assumption, “the [WiC dataset] annotators were not provided with knowledge from any external lexical resource”, and were asked to label each instance solely based on whether they thought the two occurrences of the word referred to the same meaning or not.
We repeated the above annotation task by asking 25 conference participants to provide “true” (T) and “false” (F) answers for the six instances in Table 9. The results show a high degree of disagreement, suggesting that the above claim is not always valid, especially when subtle sense distinctions are involved. We posit that the presence of a certain amount of intrinsically debatable items hampers fair comparisons between humans and systems.
Indeed, in WSD evaluation tasks, where the granularity of senses is a key concern (Bevilacqua et al., 2021b), we advocate that the starting point for the task design should consist either of the warning made by many lexicographers that “there is very little agreement about what word senses are or how broad their scope should be, and no definitive way
8https://cst.ku.dk/kalender/ sprogteknologisk-konference-2022/
of knowing when one sense ends and another begins” (Atkins and Rundell, 2008), or the one from the famous lexicographer, James Murray, that “the best any lexicographer could hope for would be that readers would feel, on scanning a multisense dictionary entry, that this is not an unreasonable way of exhibiting the facts”. With large corpora and the latest advances in language modeling, we now have the possibility to measure differences between contexts in which words are used, and we need not and should not rely on made-up sentences from the times when corpora were not available at all. This is corroborated, for instance, by the inter-tagger agreement and the systems’ results of the multilingual version of the WiC task, where sentences come from real text and dictionary definitions are used as a help for annotators (Martelli et al., 2021).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and Appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 2, 3, 4, and Appendix
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 2 and 3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 2 and 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sections 2, 3, and Appendix
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave such demonstrations for future work. Additionally, while a good portion of the NLP research effort is devoted to natural language generation (NLG) tasks (which includes MT), here we provide only some pointers to NLG/MT. Indeed, as discussed in Section 4. 3, these problems exist in the NLG universe as well, but, due to space constraints, we limit our analysis to NLU tasks