How not to Lie with a Benchmark: Rearranging NLP Learderboards
Authors: Tatiana Shavrina
Valentin Malykh
Venue: NeurIPS 2021 Submitted (2021)
Keywords: benchmarking
human evaluation
NLP benchmarks
GLUE
average
model evaluation

Abstract: Proper model ranking and comparison with a human level is an essential requirement for every benchmark to be a reliable measurement of the model quality. Nevertheless, the methods for model comparison could have a fundamental flaw - the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets.
In this paper, we examine popular NLP benchmarks' overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results. We analyze several popular benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME. The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.

How not to Lie with a Benchmark: Rearranging NLP Learderboards
Anonymous Author(s)
Affiliation
Address
email
Abstract
Comparison with human is an essential requirement for benchmark for it be a reliable measurement of model capabilities. Nevertheless, the methods for model comparison could have fundamental flaw the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets.
6
In this paper, we examine popular NLP benchmarks' overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results. We analyze several popular benchmarks including GLUE, SuperGLUUE, XGLUE, and XTREME. The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.
11
Introduction
13
The benchmarking approach has a rich history throughout computer science and is now the leading method in machine learning progress validation. the field of Natural Language Processing (NLP), 15 there exist at least 898 benchmarks" the most prominent being GLUE, SuperGLUE, XGLUE, etc., created within single paradigm.
16
17 The increase in the number of publications and developments in the field of machine learning has led to the need for methodological development of standards for describing models and all stages of the experiment, including the collection and processing of preliminary data for training, reproducibility of results, testing conditions, and most importantly, the creation of common measurable criteria for evaluating intelligent systems, both natural and artificial (Chollet) 2019). Modern NLP benchmarks substantively inherit Turing test, i.e. test the model abilities with various intellectual tasks expressed with texts, and methodologically inherit the benchmark approach for measuring the computing performance, like SPECB
19
21
22
23
24
25 The delicate question of a general assessment of the model results on all the tasks is often solved by method that is unforgivably simple for such a responsible task the arithmetic mean for all 27 the tasks. This does not take into account the scatter of results on different tasks, the different size of the task test sets (e.g. in SuperGLUE they differ hundred times, compare 146 test samples in et al.) 2019a)), different susceptibility to leaks (Elangovan al. (02T). including year of creation (Recognizing Textual Entailment data
29 Winograd Schema and 10'000 test samples ReCoRd(Wand
31 was collected in 2005 (Dagan et al. 2005), while BoolQ or CommitmentBank data was collected in 2019 et al. 2019), (De Marneffe al. 2019)).
32
sicondingo/httpp:/pppprrsstttttd com/ http//rraa/aaarraaaaaaa taaguugge-procesiing
http:p/s/www...e...rgr/bebccccce html
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

33 In this article we present an analysis of the NLP benchmarks' results, using not the arithmetic mean, but the other metries: geometric mean and harmonic mean. As F1 (harmonic mean) is frequently used to normalize Precision and Recall as they are fractions, optimizing the classifier threshold to maximize F1 leads to more balanced balance between metrics than the arithmetic mean because penalizes systems more for the smaller values(Sasaki et al.) (007). The geometric mean, as noted in the (Fleming and Wallace, 1986), is the preferred metric to the arithmetic mean in computing performance benchmarks when it comes to normalized values and percentages. The results change the usual idea of the models' order on leaderboards: humans still occupy the first place the intellectual task solving. and the best results (1.5-2% worse than humans) belong DeBerta
34
35
36
37
38
40
al. 2020), T5+Meena (Raffel et al. 2020) and McA. model Thus, the contribution of this paper is two-fold: 1) we present the reviewed approach model evaluation on multiple tasks 2) re-arrange the currently existing leaderboards of most popular benchmarks.
42
44
45
This work is organized as follows: section 2 presents previous work on the topic, it is followed by 46 section [3 with a description of the general methodology of cross-checking the results, including the scores for each benchmark. Next, analysis of the results and discussion are presented in section 4
48 as well as a conclusion in section
2 Previous Work
49
Evaluation and comparison of NLP models beget a rich history, rising with the Turing test (Turing 2009). The question-answering approach then evolved into the SQuAD task (Rajpurkar et al. (016). 51
comparing systems and annotator results by their ability find answers informative questions.
59
The next step in the development and assessment of intelligent systems belongs to the benchmark methodology, which aims to bring the solution of the Natural Language Understanding problem
55
closer- General Language Understanding Evaluation Wang et al. 2019b).
56
The General Language Understanding Evaluation (GLUE) methodology includes:
57
1. a benchmark from N (11 in original GLUE) intellectual tasks of understanding natural language, with fixed division into training, validation and test data;
58
59
2. a set of diagnostic data designed exclusively for testing and analyzing the results of trained systems in relation to a wide range of categories found at various levels of natural language (morphological, lexical, syntactic, semantic);
80
81
3. averaged human performance evaluation the tasks;
65
4. publicly available rating system and codebase to quickly reproduce results from publicly available systems and self-evaluate models.
64
85
GLUE has developed, by no means, a prolific method to model evaluation, and has already been reproduced several times in new language material: in Chinese (Xu et al. 2020). Korean (Park)
86
67
et al., 2021), Russian (Shavrina et al., 2020), Polish (Rybak et al.) 2020). and French (L.e et al.
68
2019 languages, and also jumpstarted two multilingual projects: XGLUE (Liang al., (0200 and
69
XTREME (Hu et al. 2020)
70
As stated in (Wang al. 2019a). "Lacking a fair criterion with which to weight the contributions of each task to the overall score, we opt for the simple approach of weighing each task equally, and for
71
72 tasks with multiple metrics, first averaging those metrics to get a task score." " All the GLUE-based benchmarks follow this methodology.
73
However, apart from the GLUE format, other benchmarks have provided several alternatives to evaluate the overall model contribution.
75
76 KILT, Benchmark for Knowledge Intensive Language Tasks (Petroni et al. 2020). avoids calculating
77 the overall result, and also do not compare the result with the human level, but only provides metrics for individual tasks.
79
DecaNLP (MeCann al. 2018) makes rating using not the average, but the sum of points for all tasks. This approach allows balancing the contributions of different tasks to the overall metric.
80
https://www hwt.::.fleek...mm httpyytokk..cm///////1/111

---

3
Method
81
82
We arrange the NLP benchmark results using the publicly available model scores for all the tasks to calculate new overall scores.
83
Other available options from Pythagorean means- harmonic mean and geometric mean can also be considered: we have centered our research around 2 simple statistics that are widely used for averaging fractions (lun Chou 1969) normalized values (Fleming and Wallace (986) among the possible alternatives. We did not consider other measures central tendency, like median and mode, the averaged samples more often consist of about 10 measurements, and them such metrics can give the same results for competing systems.
85
87
88
89
The arithmetic mean (AM) desribed in eq. is caleulated as the sum of the task scores (Xs) divided by the total number of tasks, referred to as N.
91
92
The geometric mean (GM) desribed in eq. is calculated as the N-th root of the product of all task scores (with the above conditions), where N is the number of values.
93
94
The harmonic mean (HM) desribed in eq. is calculated as the number of values N divided by the sum of the reciprocal of the values
1
I +
22 +
AM
T1
n
i=1
98
GM
Tj
V 21 ... In
(2)
N
HM
(3)
97
The sections below present the results of the leaderboard re-weighting as of May 2021.
98
3.1
Reevaluating the Benchmarks
99
The GLUE (11 tasks for English), SuperGLUE (10 tasks for English), XGLUE (/I tasks for 19 languages), and XTREME (4 tasks for 40 languages) provide different scoring metrics for each task, including Accuracy, F1, Matthew's ' correlation coefficient, Exact Match, while the overall score is calculated by their simple average. In cases like these, the geometric mean is appropriate when the data contains values with different units of measure (lun Chou 1969).
101
102
103
The harmonic mean of the task results better overall metric has the same grounding as introduction of the F-1 measure over precision and recall (Sasaki et al.) 2007): the harmonic mean is more intuitive than the arithmetic mean when computing mean of ratios. Given the set of metrics with large scatter, the harmonic mean will be less than the arithmetic mean, penalizing the system more for the errors made.
105
107
As stated in (Dittmann and Maug. 2008). "error measures are inherently subjective as they are determined by the loss function of the researcher or analyst who needs to choose a valuation procedure. Therefore, our analysis cannot establish which error measure should be used. Instead, our objective is to highlight the effects of the choice of error measure, that researchers and analysts alike can draw their own conclusions about the error measure and, eventually, about the valuation methods they wish to use." Nevertheless, the arithmetic, geometric and harmonic mean 113
109
111
115 are all differently subjected to outliers in a data sample. The arithmetic mean always results in a higher values than the geometric mean or the harmonic mean, and the harmonic mean always results in a lower estimate than the geometric mean (Xia et al. 1999). thus, the harmonic and geometric mean tend more strongly toward the least values, and tend to mitigate the impact of large outliers and aggravate the impact of small ones.
117
119
The harmonic mean is the appropriate mean if the data is comprised of rates, while the geometric mean is used as an unbiased estimation when working with normalized ratios, for example, in finance (Dittmann and Maug 2008) or computing benchmarks (Fleming and Wallace 1986).
121
3

---

123
However, their applicability to a better summarization of the model performance to single number has been widely discussed, see (Smith 1988). discussing performance computing:
124
125
the harmonic mean is considered the appropriate metric to summarize benchmark results expressed as rates,
127
while geometric mean is applicable in case of the use of performance numbers that are normalized with respect to one of the results being compared (see[4).
129
and arithmetic mean should not be used as a summarizing metric with rates, making it the worst choice for results accumulation.
When measuring the geometric and harmonic mean, the following assumptions were used:
132
the geometric mean does not accept zero values (also negative ones), we filled the cases of metric lacking (for example, Sentence Retrieval in XTREME) with 0.00001 values;
133
134
tasks with more than one metric measured (e.g. Accuracy and FI), are subjected to the aver- aging operation when measuring the total score, as in the standard GLUE methodology: the arithmetic mean of all metrics is taken for each task. Another modification of measurement is potentially possible: for tasks with several metrics, take all of them at once and add them as separate independent results the averaging. Then tasks with separate high metries will have less weight on the total;
135
136
137
138
139
140
among the best benchmark results, there were no negative values (MCC metric), but in theory they could be, and that would prevent the calculation of the harmonic mean.
142
the GLUE diagnostic dataset does not have human evaluation score on the leaderboard and therefore, is considered 0 (0.00 (0.00001), though in the SuperGLUE benchmark the same dataset has obtained its evaluation.
144
145
3.2
GLUE
146
GLUE benchmark (Wang et al. 2019b) combines 11 tasks in various text classification and question answering.
148
Overall score: average of all the task results. If task has main metrics, these metrics are averaged, then added to the overall average.
149
150
collected on reported human performance numbers from original datasets, not exceeding 200 examples (heavily criticised in (Nangia and Bowman 2019)). The human baseline performance on the diagnostic set was provided by the project authors with the help of six NLP researchers annotating 50 randomly selected sentence pairs.
Human evaluation:
151
152
153
Rearranging the scores: the results of geometrie and harmonic mean rearrangement are presented in Tab. D GLUE benchmark seem to be the most reordered of all the ratings considered: the best result by geometric and harmonic means belongs to humans, DeBerta and McA got point demotion, and the other models got severely rearranged their places.
155
157
N
Name
AM
HM
GM
CoLAA
SST-2
MRPC
STS-B
QUP
MNLI
MNLI
QNLI
Mean
Mean
Mean
mum
Human
37,10
86,16
86,91
83,55
92,65
69.95
92,50
91,20
DeBERTs
90,80
84,78
86,25
71,50
97.30
93,00
92,75
83,50
91,90
91,00
99,20
Mac
Albert
50,70
84,70
86,13
93,55
92,70
82,65
91,30
97,80
-DKM
TS
90,30
84.18
85,92
71.60
97.50
91.60
92,95
82,85
92.20
91.90
96,90
PING-AN
50,60
84,26
85,83
73,50
57,20
93.00
92,70
83,55
91.60
91,30
97,50
ERNIE
90,40
84,27
85.75
74,40
92.45
92,80
83,05
91,40
91.00
96,60
Table 1: Top results of ranking GLUE benchmark with geometric mean. N -original model rank on
the leaderboard. MNLI m and MNLI mm correspond to MultiNLI Matched MultiNLI Mismatched,
other task abbreviations correspond to their GLUE leaderboard designations accordingly.
158 3.3 SuperGLUE
SuperGLUE Wang et al. 2019a is the sophisticated version of the GLUE benchmark, combining 10 180 tasks with a higher demand for higher intellectual abilities. Task data must is available under various licenses that allow use and redistribution for research purposes.

---

162
Overall score: average of all the task results. If task has 2 main metrics, these metrics are averaged, then added the overall average.
163
184
Human evaluation: ready-made estimates for WiC, MultiRC, RTE, and ReCoRD datasets, the other tasks being evaluated by the project creators with the help of crowdworker annotators through Amazon's Mechanical Turk.
186
Re-weighting the results using the geometric mean and harmonic mean again makes significant changes to the original ranking: the top-3 result (human) is ranked top-1, the DeBerta and T5 models are shifted down I position, PAI ALbert and Nezha Plus models swap their
Rearranging the scores:
168
170
places, see Tab. 2
N
Model
AM
HM
GM
BoolQ
Mean
COPA
MRC
RCD
RTE
WIC
WSC
AX-b
mean
AX-g
Human DeRERT
88,73
97.35
66.35
86,00
10000
76.6
90,30
86.89
87,60
90,40
98,40
73.95
94.30
93.20
77,50
95,90
66.7
93.55
TS+
2
Meena
90,20
86,42
87,10
91,30
96,70
97,40
75,65
93.85
92,70
77,90
95,00
66.5
89,35
TS
86,57
656
Alberi
$6.10
85,24
85,78
88,10
94.40
91,80
60.65
88.65
88, 80
74,10
93,20
756
98,75
plus
Necha
82,29
93,60
74,60
80,75
Table 2: Top results of ranking SuperGLUE benchmark with geometric mean, N original model
rank on the leaderboard MRC stands for MultiRC averaged metric, RCD ReCoRD averaged metric.
3.4 XTREME
172
The XTREME benchmark (Hu et al. 2020)covers 40 typologically diverse languages from 12 language families and includes 9 tasks that require analysis of different levels of syntax semantics.
173
174
Overall score: 2-step averaging: I) calculating average for each task on all languages 2) calculating
average all tasks.
178
Human evaluation: 2-step averaging:
1. step I: ready-made estimates from the original datasets taken and extrapolated all unestimated languages; besides, for some datasets there were no original estimates provided (POS) and empirical estimate of 97% taken based on (Manning) 201 estimates for NER and sentence retrieval tasks;
178
180
2. step 2: all the task results averaged together.
182
Rearranging the scores: the results of applying the geometric mean and harmonic mean did not
183
change the current ranking of the models the quality spread between them is high enough for the metrics averaging them to retain the current order.
184
Sentence-pair Classification
Structured Prediction
Question Answering
Senfence
N Model
AM
HM
GM
Retrieval
Human
93.30
93,13
93,21
95,10
97,00
87,80
0.00001
VECO
81.10
81.27
81,70
88,60
75,40
72,40
92,10
ERNIE-M
80,90
81,11
81,52
87,90
75,60
72,30
91,90
T-ULRv2
80.70
80,91
81,25
88,80
75,40
72,90
89,30
Anonymousã
79,90
80,12
80,50
88,20
74,60
71,70
89,00
Polyglot
77,80
78,02
78,56
87,80
72,90
67,40
88,30
Table Top results of ranking XTREME benchmark with geometric mean. N original model rank
on the leaderboard; the averaged task scores are shown by the column markings.
185
3.5 XGLUE
The XGLUE benchmark (Lann et al.) consisss of 11 problems in 19 languages and evaluates 187 the performance of multilingual pre-trained systems in terms of their ability to cross-language understanding and natural language generation.
5

---

189
Overall score: 2-step averaging: 1) calculating average for each task on all languages 2) calculating average on all tasks.
190
191
Human evaluation: not provided.
Rearranging the scores: Since the human level is not measured in the benchmark, we only compare the present models with each other. Tab. | shows the results the difference in the quality of the models is large enough to preserve their ranking on all averaging metrics.
198
Model
AM
HM
GM
NER
POS
NC
MLQQ4
XNLJ
PAWS-X
QADSM
WPR
QAM
PILTER
80,10
79,86
82,60
81,63
83,50
76.30
83,90
53,80
71,40
74.70
73,40
Daseline
76.10
73,45
73,80
79,70
79,60
83,50
66,00
75,30
90.10
68.40
73,90
68.90
N Model
AM
HM
GM
Q6
NIG
Unicoder
10.65
10,60
10,70
MP.Tune
8,70
8,70
7,10
8,10
9,40
Table Top results of ranking XGLUE benchmark with geometric mean. - original model rank
on the leaderboard; the first rows correspond to NLU tasks, the last 3 rows the NLG tasks.
Results and Discussion
195
196
The results show that the ranking of results within a single leaderboard can fluctuate significantly. So, in GLUE, the first place in terms of the harmonic and geometric mean belongs to the result occupying the 16th line in the arithmetic mean. SuperGLUE, the permutation not so striking the third result is on the 1st place. On the XTREME and XGLUE benchmarks system ranking is preserved.
198
Since all three averaging metries considered are subject different biases, we present the statistical 200
measurements of the top-3 SuperGLUE results in Tab. 5) Human results have the highest total points
202
for all tasks (as in the DecaNLP methodology), while the standard deviation and variance are greater
203
than t and top-3 models.
Model Human
AM
GM
HM
Sum
Var
Std
89.8
88,73
87,96
894,40
130.31
11,42
DeBerta
90,3
87,60
86,89
882,55
117,46
10,84
T5 + Meena
90.2
87,10
86,42
877,25
112,39
10,60
Table Measuring the statistics of the top-3
SuperGLUE results. Sum is sum of all the task scores, Var and Std are variance and standard deviation on the task scores respectively. Notable results are highlighted in bold.
204
To further explore the rating results of the GLUE and SuperGLUE benchmarks, we have conducted series of experiments with normalizing the model performance with human scores, fully transferring the standard methodology for computing performance, in which the geometrie mean is the approved metric. The results are presented in Appendix 1. As can be concluded from the table with normalized values, in the case of SuperGLUE, the results obtained by the new ranking method are confirmed. In the case of GLUE, the geometric mean shows that the normalized ratio of models to the human level asserts high level of artificial solutions over the human level.
205
206
207
208
209
210
211
The following topics remain debatable and need special attention of the community:
212
Different metries for obtaining the average value (arithmetic, geometrie, harmonic) have different restrictions on the accepted values (for example, not every one can take negative or zero values). At the same time, metrics that take zero and negative values are actively used in measuring various skills MCC metric on SuperGLUUE diagnostics can be negative, other metrics can be equal to or greater than zero, and they are averaged altogether. Potentially, the issue of fair metric will raise the problem of revising the use of some individual metrics for evaluating tasks. The differences the metric scale for different tasks can pose problems for the computation of the total score and some metrics can be scaled or normalized, We may consider rescaling MCC score so that it is between 0 and
213
214
215
216
218
220
221
2. Correct averaging of the overall score for multilingual benchmarks creates additional prob- lems while performing the averaging operation in 2 stages; for all languages and all tasks.

---

223
As a result, the score, consisting of one number, becomes less and less informative and more prone outliers.
224
225
3.
Nevertheless, the competitive side of benchmarks is the driving force behind the progress in the field of machine learning, and besides all problems, it is still not worth giving up leaderboards with single metric.
226
4. In addition the problem of the main averaging metric, we left outside of the scope the problem that was also discovered within the framework of this study: human benchmark scores on various tasks were obtained in a very different way, and always a smaller sample than the full test set. For a fair comparison of humans and machines, the test results should be normalized by the same number of test items, and it is worth revising the human evaluation and re- re-performing it on all test items using more annotators.
229
231
233
Conclusion
234
235
In this paper we present an alternative methods to arrange the popular NLP benchmark results, elaborating on several task evaluation. We analyze popular benchmark averaging methods and provide new insight into model comparison. Namely, we obtain the following results:
236
237
238
for popular benchmarks GLUE and SuperGLUE we can conclude that their overall score is subject to bias due to outliers; the alternative arrengemend methods end with significantly different ordering of the results;
239
240
241
rebuilding leaderboards using other metrics (geometric or harmonic mean) allows one to conclude that human result is the first in the rankings:
242
in XGLUE leaderboard human result is obtained by extrapolation from one language to others, while in practice the level of problem-solving by native speakers of different languages varies;
244
246
* the last finding could be extended to other multilingual benchmarks also.
247
An unbiased view of the overall score in the benchmarks is a necessity for a community to target language model development on the complex improvement of their quality, not the partial results in narrow tasks. The expansion towards multilingual and multimodal models makes this issue more and more urgent and we hope our help to foster research in this direction.
249
References
252
François Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547
253
Christopher Clark, Kenton Lee, Ming-Was Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv
254
255
preprint ArXiv:1905.1001
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer.
257
258
De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung,
260
volume 23, pages 107-124.
281
Ingolf Dittmann and Ernst Maug. 2008 Biases and error measures: How to compare valuation methods.
262
263
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: quanti-
264
fying data leakage in nlp performance evaluation. arXiv preprint arXiv:2102.018818
265
Philip J. Fleming and John J. Wallace. 1986. How not to lie with statistics: The correct way to
266
summarize benchmark results, Commun. ACM, 29(3):218 221.
267
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint 0335542006.03654.
7

---

289
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross- mooss-l generalisation. In International Conference on Machine Learning, pages 1411-4421. PMLR.
270
272
Hang Le, Loic Vial, Jibril Frej. Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2019. Flaubert: Unsupervised language model pre-training for french. arXiv preprinl
273
275
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401.
276
278
Ya lun Chou. 1969. Statistical analysis: With business and economic applications
279
Christopher D Manning. 2011. Pfrt-of-speech tagging from 97% to 100%: is it time for some
280
linguistics? International conference on intelligent text processing and computational linguistics, pages 171-189. Springer.
282
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint 1006.088000088
285
284
Nikita Nangia and Samuel R Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the glue benchmark. arXiv preprint vrxiiv/905.. 10425.
285
286
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021. Klue: Korean language under- standing evaluation. arXiv preprint arXiv: 100.0005.0000 arXiv:2105.096680 09680.
287
289
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thomee, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020. Kilt: benchmark for knowledge intensive language tasks. arXiv preprini 20Xv:2200.00252..
290
292
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer
293
295
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of texi.
296
297
Piotr Rybak, Robert Mroezkowski, Janusz Traez, and Ireneusz Gawlik. 2020. Klej: comprehensive benchmark for polish language understanding, arXiv preprint arXiv:2005.006330
Yutaka Sasaki et al. 2007. The truth of the f-measure. 2007.
300
Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. arXiv preprint arXiv:2010.15925.
301
302
303
304
James F. Smith. 1988. Characterizing computer performance with a single number. Communications of the ACM, 1220-2220-
305
Alan M Turing. 2009. Computing machinery and intelligence. In Parsing the turing tesl, pages 23-65, Springer.
307
308
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint XiX.:i9905.052:
310
311
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019b. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019.
313
314
Da-Feng Xia, Sen-Lin Xu, and Feng Qi. 1999. A proof of the arithmetic mean-geometric mean- harmonic mean incqualities. RGMIA research report collection, 2(1).
8

---

316
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986.
317
319
1. For all authors...
320
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
321
contributions and scope? [Yes]
322
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
324
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
325
326
2. If you are including theoretical results...
327
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
328
(b) Did you include complete proofs of all theoretical results? [N/A]
329
3. If you ran experiments...
330
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
331
332
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
333
334
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
338
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
337
338
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
339
(a) your work uses existing assets, did you cite the creators? [Yes]
340
(b) Did you mention the license of the assets? [Yes]
341
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
343
344
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
347
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
348
349
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
352
353
355


Reviews:
[{"review_id": "tbWnkxsiOSN", "reviewer": "Reviewer_DNNE", "summary": "This paper presents an analysis of how results are re-ordered on a number of popular NLP benchmarks when the harmonic or geometric mean are used instead of the arithmetic mean. The paper does not contain much analysis of the significance of these results, or a strong argument in favor of any one approach.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A number of poular NLP benchmarks aggregate scores from multiple tasks. This paper presents a criticism of the averaging technique, along with an analysis of how results are re-ordered if the harmonic or geometric mean is used instead. The paper shows that the alternate means lead to significantly different orderings and consistently place human scores above NLP systems.\n\nThe choice of averaging procedure is important. However, this paper does not present much justification for either of the alternatives. There are citations to related discussions on averaging of rate metrics in reporting computer performance, where outliers can be extremely important. But there is no discussion of the nature and importance of outliers in our NLP benchmarks and this paper does not contains strong evidence for preferring either the harmonic or geometric means.\n\nThe paper would be stronger with a thorough analysis of the nature of the outliers that are being highlighted by the harmonic and geometric means, along with a discussion of how the choice affects our abilities to predict performance on arbitrary new tasks. The discussion in this paper is important, but the current presentation does not have enough depth to be presented in NeurIPS.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper doesn't provide a strong argument for any one method of aggregating sub-task scores, and would be a lot stronger with a detailed discussion of what the different means are highlighting in terms of sub-task performance and expected generalization to new tasks.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "s7glzLuOOfh", "reviewer": "Reviewer_Yzmg", "summary": "This paper proposes harmonic and geometric mean as methods of aggregating performance results on different tasks in multi-task benchmarks. It then goes through several multi-task benchmarks and shows how the leaderboards would change under these new global metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really, really like the idea of this paper. It’s a great point that the arithmetic mean has downsides and is probably not entirely appropriate for multi-task benchmarking. It is also heartening that using other aggregation methods can surface new potential for improvement (as it puts humans back on top of the leaderboard). Some theoretical motivation for geometric and harmonic mean measures is provided which is good as well.\n\nI absolutely think this kind of work should be done, but I feel that this particular paper falls short. As is, it seems to me like a missed opportunity to investigate the questions which would be most important for people producing new benchmarks, in particular: which aggregation methods *should* we use, and in what cases? How do different aggregation methods affect the interpretation of the results? And, how can we structure our benchmarks so that we *can* have meaningful aggregate measures over them? The investigation of geometric and harmonic mean with existing multi-task benchmarks provides us an opportunity to really dig into these questions, but I feel that the opportunity is mostly passed up. The theoretical motivation is shallow, just mentioning things like geometric means being appropriate for fractions and harmonic means being appropriate for rates (L120–122). As a minor note, I think a paper like this should go deeper into the background and explain why these things are true instead of just giving references. But more importantly, what I think must be discussed is how these considerations apply to benchmarks in practice. Which common measures in dataset evaluations are meaningfully interpreted as rates versus ratios? How do we deal with benchmarks that have heterogeneous metrics? And for that matter, what even are the desiderata for an aggregate metric in the first place? How do we know when to prefer one over the other, either in general or for any specific benchmark?\n\nSo, yeah. Lots of questions remain, and the paper left plenty of space on the table that could have been used to deeply investigate them (I'm not asking for answers; these are hard questions. But at least dedicated attention and investigation which provides the lay of the land is essential for this paper to be really useful). So I feel this work is very promising and interesting, but incomplete.\n\nAs a “historical” note which relates to the discussion of the goals of a benchmark/metric: In the background, GLUE is presented as a natural point in the progressive development of benchmarking practice in AI, and arithmetic mean is evaluated from that point of view. But I think it’s important to remember the history here: GLUE was presented particularly as a transfer learning benchmark. The different sizes of test set were a result of the different availability of data for each task, and all tasks were equally weighted partly with the intent of assessing a model’s ability to perform well on tasks with less data by leveraging training signal in other tasks with more data. Other domain- and problem-specific concerns may apply to similar multi-task benchmarks (for example, testing crosslingual performance, etc.) which mean the overall concerns about arithmetic mean might be overblown (rather, the problem would be if the benchmark is interpreted in isolation as a singular barometer of AI performance. But with so many distinct multi-task benchmarks out there these days, I’m not sure if this should be a big concern).\n\n* L36: “a more balanced balanced” -> you can just say “a better balance” :)\n* L50–51: the reference to Turing (2009) is a bit strange. His publication “Computing Machinery and Intelligence” appeared in the journal Mind in 1950. Also, it’s not really a good example of an evaluation benchmark. The Turing Test was closer to something between a thought experiment and a grand challenge. Benchmarks are more for measuring incremental progress. A better reference would probably be something on IBM’s Common Task Framework or the Shared Tasks common in the NLP and CV communities.\n* L79–80: For the purposes of ranking, sum and average are equivalent, so I’m not sure what this is saying.\n* L239: arrengemend", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. I don't think there are any significant concerns. This paper argues for improvements to multi-task metric reporting methodology, which seems like an uncontroversial good (i.e., if we are going to use measuring sticks, we should make them accurate).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "mmHVN7fXAbl", "reviewer": "Reviewer_hz3e", "summary": "The paper proposes to replace the arithmetic mean with harmonic and geometric mean in leaderboard result aggregation. It presents reordered leaderboards for GLUE, SuperGLUE, XGLUE and XTREME. \n", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In a paper whose only contribution is to replace arithmetic mean with harmonic and geometric mean, in leaderboard averaging, I would have liked to see the explanation of why these are better than the arithmetic mean, fleshed out in more detail, rather than simply referring to previous work and stating that this is how things are. \n\nThe paper fails to discuss how the proposal is related to previous work on aggregation of benchmark results; see [0-3] for a few examples. \n\nThat benchmarks use ‘wrong averaging’ is not a new observation. See blog posts such as [4-5]. I feel the paper contributes very little compared to what’s out there; in fact it goes into less detail than some of the blog posts. \n\nThe paper comes across as written in haste with poor formatting and grammar, e.g., lines 27-30, and poor language, e.g., \"a more balanced balance”. Consider this sentence for example (lines 104-5): \"The harmonic mean of the task results as a better overall metric has the same grounding as introduction of the F-1 measure over precision and recall.” I simply cannot make sense of it. \n\nThe reasoning is also weak in places. See, for example, lines 79-80: \"DecaNLP (McCann et al., 2018) makes a rating using not the average, but the sum of points for all tasks. This approach allows balancing…” I do not understand why summing is easier to de-aggregate than averaging (but let me know if I’m missing something). Another example, line 89: While I agree mean and mode probably do not make sense for aggregation in benchmarks, I don’t understand why \"on them such metrics can give the same results for competing systems”. \n\n[0] http://users.umiacs.umd.edu/~jbg//docs/2021_acl_leaderboard.pdf\n[1] https://www.aclweb.org/anthology/N13-1068.pdf\n[2] https://www.cambridge.org/core/journals/natural-language-engineering/article/survey-of-25-years-of-evaluation/E4330FAEB9202EC490218E3220DDA291\n[3] https://www.aclweb.org/anthology/2020.emnlp-main.393/\n[4] https://towardsdatascience.com/on-average-youre-using-the-wrong-average-geometric-harmonic-means-in-data-analysis-2a703e21ea0\n[5] https://machinelearningmastery.com/arithmetic-geometric-and-harmonic-means-for-machine-learning/", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I was a bit underwhelmed by the lack of discussion of related work, as well as of a more general discussion of leaderboards. Finally, with three different ways of averaging leaderboard results, we enable cherry-picking. The authors could perhaps recommend reporting all three to prevent this. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "Pe_-71m8I_4", "reviewer": "Reviewer_65MJ", "summary": "The benchmarks and leaderboards such as GLUE, SuperGLUE have been largely driving the progress of developing NLP models these days. A range of  models have already achieved super-human performance on these leaderboards.  This paper makes an argument that taking the arithmetic mean of scores across different tasks is not ideal, and instead, it proposes to use geometric mean and harmonic mean to re-compute the scores on multiple leaderboards. The findings are 1)  The ranking will be changed greatly by using geometric or harmonic mean; 2) The human performance baseline will rank highest on GLUE and SuperGLUE.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper clearly raises an interesting question and I think how to aggregate different scores/metrics is an important topic that is worth a discussion in the research community. This problem has been somewhat neglected at this point AFAIK. (As the paper pointed out, some other recent benchmarks such as KILT don't use an aggregated score).\n\nHowever, on the other hand, this paper clearly doesn’t have substantial content for a NeurIPS paper. It is only 6.5 pages and most of the content is re-calculating the scores for 4-5 leaderboards. The contribution is too slim in that regard. \n\nAlthough the paper proposes to use geometric or harmonic mean, I think it lacks in-depth analysis or explanations for why the metrics should be adopted and used. \n\nAll the numbers in the tables have a formatting issue, e.g., “89,80” → “89.80”.\n\nIt lacks discussion of related work. The issues of leaderboards have been widely discussed in the NLP community recently, e.g., \n- Utility is in the Eye of the User: A Critique of NLP Leaderboards\n- Show your work: Improved reporting of experimental results\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It raises an interesting and important question to the research community for how we compare different models across different metrics/tasks on a leaderboard, although I don't think the paper sufficiently addressed the problem. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that this paper proposes a reasonable small change to the reporting of scores from multitask benchmark, but that the arguments justifying this change are not precise or novel enough to warrant presentation at NeurIPS.
Decision: Reject