How not to Lie with a Benchmark: Rearranging NLP Learderboards

Comparison with a human is an essential requirement for a benchmark for it to1 be a reliable measurement of model capabilities . Nevertheless , the methods for2 model comparison could have a fundamental flaw - the arithmetic mean of separate3 metrics is used for all tasks of different complexity , different size of test and training4 sets.5 In this paper , we examine popular NLP benchmarks ’ overall scoring methods and6 rearrange the models by geometric and harmonic mean ( appropriate for averaging7 rates ) according to their reported results . We analyze several popular benchmarks8 including GLUE , SuperGLUE , XGLUE , and XTREME . The analysis shows that9 e.g . human level on SuperGLUE is still not reached , and there is still room for10 improvement for the current models.11 1 Introduction12 The benchmarking approach has a rich history throughout computer science and is now the leading13 method in machine learning progress validation . In the field of Natural Language Processing ( NLP ) ,14 there exist at least 898 benchmarks1 , the most prominent being GLUE , SuperGLUE , XGLUE , etc.,15 created within a single paradigm.16 The increase in the number of publications and developments in the field of machine learning has led17 to the need for methodological development of standards for describing models and all stages of the18 experiment , including the collection and processing of preliminary data for training , reproducibility19 of results , testing conditions , and most importantly , the creation of common measurable criteria for20 evaluating intelligent systems , both natural and artificial ( Chollet , 2019 ) . Modern NLP benchmarks21 are substantively inherit to Turing test , i.e . test the model abilities with various intellectual tasks22 expressed with texts , and methodologically inherit the benchmark approach for measuring the23 computing performance , like SPEC2.24 The delicate question of a general assessment of the model results on all the tasks is often solved25 by a method that is unforgivably simple for such a responsible task - the arithmetic mean for all26 the tasks . This does not take into account the scatter of results on different tasks , the different size27 of the task test sets ( e.g . in SuperGLUE they differ a hundred times , compare 146 test samples in28 Winograd Schema and 10 ’ 000 test samples in ReCoRd ( Wang et al. , 2019a ) ) , different susceptibility29 to leaks ( Elangovan et al. , 2021 ) , including year of creation ( Recognizing Textual Entailment data30 was collected in 2005 ( Dagan et al. , 2005 ) , while BoolQ or CommitmentBank data was collected in31 2019 ( Clark et al. , 2019 ) , ( De Marneffe et al. , 2019 ) ) .32 1according to https : //paperswithcode.com/area/natural-language-processing 2https : //www.spec.org/benchmarks.html Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . In this article we present an analysis of the NLP benchmarks ’ results , using not the arithmetic mean,33 but the other metrics : geometric mean and harmonic mean . As F1 ( harmonic mean ) is frequently34 used to normalize Precision and Recall as they are fractions , optimizing the classifier threshold to35 maximize F1 leads to a more balanced balance between metrics than the arithmetic mean because it36 penalizes systems more for the smaller values ( Sasaki et al. , 2007 ) . The geometric mean , as noted37 in the ( Fleming and Wallace , 1986 ) , is the preferred metric to the arithmetic mean in computing38 performance benchmarks when it comes to normalized values and percentages . The results change the39 usual idea of the models ’ order on leaderboards : humans still occupy the first place in the intellectual40 task solving , and the best results ( 1.5-2 % worse than humans ) belong to DeBerta ( He et al. , 2020 ) ,41 T5+Meena ( Raffel et al. , 2020 ) and McAlbert+DKM models3 . Thus , the contribution of this paper is42 two-fold : 1 ) we present the reviewed approach model evaluation on multiple tasks 2 ) we re-arrange43 the currently existing leaderboards of most popular benchmarks.44 This work is organized as follows : section 2 presents previous work on the topic , it is followed by45 section 3 with a description of the general methodology of cross-checking the results , including the46 scores for each benchmark . Next , an analysis of the results and discussion are presented in section 4,47 as well as a conclusion in section 5.48 2 Previous Work49 Evaluation and comparison of NLP models beget a rich history , rising with the Turing test ( Turing,50 2009 ) . The question-answering approach then evolved into the SQuAD task ( Rajpurkar et al. , 2016 ) ,51 comparing systems and annotator results by their ability to find answers to informative questions.52 The next step in the development and assessment of intelligent systems belongs to the benchmark53 methodology , which aims to bring the solution of the Natural Language Understanding problem54 closer - General Language Understanding Evaluation ( Wang et al. , 2019b ) .55 The General Language Understanding Evaluation ( GLUE ) methodology includes:56 1. a benchmark from N ( 11 in original GLUE ) intellectual tasks of understanding a natural57 language , with a fixed division into training , validation and test data ; 58 2. a set of diagnostic data designed exclusively for testing and analyzing the results of trained59 systems in relation to a wide range of categories found at various levels of natural language60 ( morphological , lexical , syntactic , semantic ) ; 61 3. averaged human performance evaluation on the tasks ; 62 4. publicly available rating system and codebase to quickly reproduce results from publicly63 available systems and self-evaluate models.64 GLUE has developed , by no means , a prolific method to model evaluation , and has already been65 reproduced several times in new language material : in Chinese ( Xu et al. , 2020 ) , Korean ( Park66 et al. , 2021 ) , Russian ( Shavrina et al. , 2020 ) , Polish ( Rybak et al. , 2020 ) , and French ( Le et al.,67 2019 ) languages , and also jumpstarted two multilingual projects : XGLUE ( Liang et al. , 2020 ) and68 XTREME ( Hu et al. , 2020 ) .69 As stated in ( Wang et al. , 2019a ) , `` Lacking a fair criterion with which to weight the contributions of70 each task to the overall score , we opt for the simple approach of weighing each task equally , and for71 tasks with multiple metrics , first averaging those metrics to get a task score . '' All the GLUE-based72 benchmarks follow this methodology.73 However , apart from the GLUE format , other benchmarks have provided several alternatives to74 evaluate the overall model contribution.75 KILT , a Benchmark for Knowledge Intensive Language Tasks ( Petroni et al. , 2020 ) , avoids calculating76 the overall result , and also do not compare the result with the human level , but only provides metrics77 for individual tasks.78 DecaNLP ( McCann et al. , 2018 ) makes a rating using not the average , but the sum of points for all79 tasks . This approach allows balancing the contributions of different tasks to the overall metric.80 3https : //www.iflytek.com/news/2118 3 Method81 We arrange the NLP benchmark results using the publicly available model scores for all the tasks to82 calculate new overall scores.83 Other available options from Pythagorean means - harmonic mean and geometric mean - can also84 be considered : we have centered our research around 2 simple statistics that are widely used for85 averaging fractions ( lun Chou , 1969 ) or normalized values ( Fleming and Wallace , 1986 ) among the86 possible alternatives . We did not consider other measures of central tendency , like median and mode,87 as the averaged samples more often consist of about 10 measurements , and on them such metrics can88 give the same results for competing systems.89 • The arithmetic mean ( AM ) desribed in eq . 1 is calculated as the sum of the task scores ( Xs ) 90 divided by the total number of tasks , referred to as N.91 • The geometric mean ( GM ) desribed in eq . 2 is calculated as the N-th root of the product of92 all task scores ( with the above conditions ) , where N is the number of values.93 • The harmonic mean ( HM ) desribed in eq . 3 is calculated as the number of values N divided94 by the sum of the reciprocal of the values95 AM = 1 n n∑ i=1 xi = x1 + x2 + · · ·+ xn n ( 1 ) 96 GM = ( n∏ i=1 xi ) 1 n = n √ x1x2 · · ·xn ( 2 ) HM = N 1 x1 + 1x2 + · · ·+ 1 xN ( 3 ) The sections below present the results of the leaderboard re-weighting as of May 2021.97 3.1 Reevaluating the Benchmarks98 The GLUE ( 11 tasks for English ) , SuperGLUE ( 10 tasks for English ) , XGLUE ( 11 tasks for 1999 languages ) , and XTREME ( 4 tasks for 40 languages ) provide different scoring metrics for each task,100 including Accuracy , F1 , Matthew ’ s correlation coefficient , Exact Match , while the overall score is101 calculated by their simple average . In cases like these , the geometric mean is appropriate when the102 data contains values with different units of measure ( lun Chou , 1969 ) .103 The harmonic mean of the task results as a better overall metric has the same grounding as introduction104 of the F-1 measure over precision and recall ( Sasaki et al. , 2007 ) : the harmonic mean is more intuitive105 than the arithmetic mean when computing a mean of ratios . Given the set of metrics with a large106 scatter , the harmonic mean will be less than the arithmetic mean , penalizing the system more for the107 errors made.108 As stated in ( Dittmann and Maug , 2008 ) , `` error measures are inherently subjective as they are109 determined by the loss function of the researcher or analyst who needs to choose a valuation110 procedure . Therefore , our analysis can not establish which error measure should be used . Instead,111 our objective is to highlight the effects of the choice of error measure , so that researchers and112 analysts alike can draw their own conclusions about the error measure and , eventually , about the113 valuation methods they wish to use . '' Nevertheless , the arithmetic , geometric and harmonic mean114 are all differently subjected to outliers in a data sample . The arithmetic mean always results in a115 higher values than the geometric mean or the harmonic mean , and the harmonic mean always results116 in a lower estimate than the geometric mean ( Xia et al. , 1999 ) , thus , the harmonic and geometric117 mean tend more strongly toward the least values , and tend to mitigate the impact of large outliers and118 aggravate the impact of small ones.119 The harmonic mean is the appropriate mean if the data is comprised of rates , while the geometric120 mean is used as an unbiased estimation when working with normalized ratios , for example , in121 finance ( Dittmann and Maug , 2008 ) or computing benchmarks ( Fleming and Wallace , 1986 ) .122 However , their applicability to a better summarization of the model performance to a single number123 has been widely discussed , see ( Smith , 1988 ) , discussing performance computing:124 • the harmonic mean is considered the appropriate metric to summarize benchmark results125 expressed as rates,126 • while geometric mean is applicable in case of the use of performance numbers that are127 normalized with respect to one of the results being compared ( see 4 ) ,128 • and arithmetic mean should not be used as a summarizing metric with rates , making it the129 worst choice for results accumulation.130 When measuring the geometric and harmonic mean , the following assumptions were used:131 • as the geometric mean does not accept zero values ( also negative ones ) , we filled the cases132 of metric lacking ( for example , Sentence Retrieval in XTREME ) with 0.00001 values ; 133 • tasks with more than one metric measured ( e.g . Accuracy and F1 ) , are subjected to the aver-134 aging operation when measuring the total score , as in the standard GLUE methodology : the135 arithmetic mean of all metrics is taken for each task . Another modification of measurement136 is potentially possible : for tasks with several metrics , take all of them at once and add them137 as separate independent results to the averaging . Then tasks with separate high metrics will138 have less weight on the total ; 139 • among the best benchmark results , there were no negative values ( MCC metric ) , but in140 theory they could be , and that would prevent the calculation of the harmonic mean.141 • the GLUE diagnostic dataset does not have a human evaluation score on the leaderboard142 and therefore , is considered 0 ( 0.00001 ) , though in the SuperGLUE benchmark the same143 dataset has obtained its evaluation.144 3.2 GLUE145 GLUE benchmark ( Wang et al. , 2019b ) combines 11 tasks in various text classification and question146 answering.147 Overall score : average of all the task results . If task has 2 main metrics , these metrics are averaged,148 then added to the overall average.149 Human evaluation : collected on reported human performance numbers from original datasets , not150 exceeding 200 examples ( heavily criticised in ( Nangia and Bowman , 2019 ) ) . The human baseline151 performance on the diagnostic set was provided by the project authors with the help of six NLP152 researchers annotating 50 randomly selected sentence pairs.153 Rearranging the scores : the results of geometric and harmonic mean rearrangement are presented154 in Tab . 1 . GLUE benchmark seem to be the most reordered of all the ratings considered : the best155 result by geometric and harmonic means belongs to humans , DeBerta and McAlbert+DKM got a 1156 point demotion , and the other models got severely rearranged their places.157 3.3 SuperGLUE158 SuperGLUE ( Wang et al. , 2019a ) is the sophisticated version of the GLUE benchmark , combining 10159 tasks with a higher demand for higher intellectual abilities . Task data must is available under various160 licenses that allow use and redistribution for research purposes.161 Overall score : average of all the task results . If task has 2 main metrics , these metrics are averaged,162 then added to the overall average.163 Human evaluation : ready-made estimates for WiC , MultiRC , RTE , and ReCoRD datasets , the164 other tasks being evaluated by the project creators with the help of crowdworker annotators through165 Amazon ’ s Mechanical Turk.166 Rearranging the scores : Re-weighting the results using the geometric mean and harmonic mean167 again makes significant changes to the original ranking : the top-3 result ( human ) is ranked top-1 , the168 DeBerta and T5 models are shifted down 1 position , PAI ALbert and Nezha Plus models swap their169 places , see Tab . 2.170 3.4 XTREME171 The XTREME benchmark ( Hu et al. , 2020 ) covers 40 typologically diverse languages from 12172 language families and includes 9 tasks that require analysis of different levels of syntax or semantics.173 Overall score : 2-step averaging : 1 ) calculating average for each task on all languages 2 ) calculating174 average on all tasks.175 Human evaluation : 2-step averaging:176 1. step 1 : ready-made estimates from the original datasets taken and extrapolated to all177 unestimated languages ; besides , for some datasets there were no original estimates provided178 ( POS ) and an empirical estimate of 97 % was taken based on ( Manning , 2011 ) ; no estimates179 for NER and sentence retrieval tasks ; 180 2. step 2 : all the task results averaged together.181 Rearranging the scores : the results of applying the geometric mean and harmonic mean did not182 change the current ranking of the models - the quality spread between them is high enough for the183 metrics averaging them to retain the current order.184 3.5 XGLUE185 The XGLUE benchmark ( Liang et al. , 2020 ) consists of 11 problems in 19 languages and evaluates186 the performance of multilingual pre-trained systems in terms of their ability to cross-language187 understanding and natural language generation.188 Overall score : 2-step averaging : 1 ) calculating average for each task on all languages 2 ) calculating189 average on all tasks.190 Human evaluation : not provided.191 Rearranging the scores : Since the human level is not measured in the benchmark , we can only192 compare the 2 present models with each other . Tab . 4 shows the results - the difference in the quality193 of the models is large enough to preserve their ranking on all averaging metrics.194 4 Results and Discussion195 The results show that the ranking of results within a single leaderboard can fluctuate significantly . So,196 in GLUE , the first place in terms of the harmonic and geometric mean belongs to the result occupying197 the 16th line in the arithmetic mean . In SuperGLUE , the permutation is not so striking - the third198 result is on the 1st place . On the XTREME and XGLUE benchmarks system ranking is preserved.199 Since all three averaging metrics considered are subject to different biases , we present the statistical200 measurements of the top-3 SuperGLUE results in Tab . 5 . Human results have the highest total points201 for all tasks ( as in the DecaNLP methodology ) , while the standard deviation and variance are greater202 than top-2 and top-3 models.203 To further explore the rating results of the GLUE and SuperGLUE benchmarks , we have conducted a204 series of experiments with normalizing the model performance with human scores , fully transferring205 the standard methodology for computing performance , in which the geometric mean is the approved206 metric . The results are presented in Appendix 1 . As can be concluded from the table with normalized207 values , in the case of SuperGLUE , the results obtained by the new ranking method are confirmed . In208 the case of GLUE , the geometric mean shows that the normalized ratio of models to the human level209 asserts a high level of artificial solutions over the human level.210 The following topics remain debatable and need special attention of the community:211 1 . Different metrics for obtaining the average value ( arithmetic , geometric , harmonic ) have212 different restrictions on the accepted values ( for example , not every one can take negative or213 zero values ) . At the same time , metrics that take zero and negative values are actively used214 in measuring various skills - MCC metric on SuperGLUE diagnostics can be negative , other215 metrics can be equal to or greater than zero , and they are averaged altogether . Potentially,216 the issue of a fair metric will raise the problem of revising the use of some individual metrics217 for evaluating tasks . The differences in the metric scale for different tasks can pose problems218 for the computation of the total score and some metrics can be scaled or normalized . We219 may consider rescaling MCC score so that it is between 0 and 1.220 2 . Correct averaging of the overall score for multilingual benchmarks creates additional prob-221 lems while performing the averaging operation in 2 stages : for all languages and all tasks.222 As a result , the score , consisting of one number , becomes less and less informative and more223 prone to outliers.224 3 . Nevertheless , the competitive side of benchmarks is the driving force behind the progress225 in the field of machine learning , and besides all problems , it is still not worth giving up226 leaderboards with a single metric.227 4 . In addition to the problem of the main averaging metric , we left outside of the scope the228 problem that was also discovered within the framework of this study : human benchmark229 scores on various tasks were obtained in a very different way , and always on a smaller230 sample than the full test set . For a fair comparison of humans and machines , the test results231 should be normalized by the same number of test items , and it is worth revising the human232 evaluation and re-performing it on all test items using more annotators.233 5 Conclusion234 In this paper we present an alternative methods to arrange the popular NLP benchmark results,235 elaborating on several task evaluation . We analyze popular benchmark averaging methods and236 provide new insight into model comparison . Namely , we obtain the following results:237 • for popular benchmarks GLUE and SuperGLUE we can conclude that their overall score is238 subject to bias due to outliers ; the alternative arrengemend methods end with significantly239 different ordering of the results ; 240 • rebuilding leaderboards using other metrics ( geometric or harmonic mean ) allows one to241 conclude that human result is the first in the rankings ; 242 • in XGLUE leaderboard human result is obtained by extrapolation from one language243 to others , while in practice the level of problem-solving by native speakers of different244 languages varies ; 245 • the last finding could be extended to other multilingual benchmarks also.246 An unbiased view of the overall score in the benchmarks is a necessity for a community to target247 language model development on the complex improvement of their quality , not the partial results in248 narrow tasks . The expansion towards multilingual and multimodal models makes this issue more and249 more urgent and we hope our help to foster research in this direction.250 References251 François Chollet . 2019 . On the measure of intelligence . arXiv preprint arXiv:1911.01547.252 Christopher Clark , Kenton Lee , Ming-Wei Chang , Tom Kwiatkowski , Michael Collins , and Kristina253 Toutanova . 2019 . Boolq : Exploring the surprising difficulty of natural yes/no questions . arXiv254 preprint arXiv:1905.10044.255 Ido Dagan , Oren Glickman , and Bernardo Magnini . 2005 . The pascal recognising textual entailment256 challenge . In Machine Learning Challenges Workshop , pages 177–190 . Springer.257 Marie-Catherine De Marneffe , Mandy Simons , and Judith Tonhauser . 2019 . The commitmentbank:258 Investigating projection in naturally occurring discourse . In proceedings of Sinn und Bedeutung,259 volume 23 , pages 107–124.260 Ingolf Dittmann and Ernst Maug . 2008 . Biases and error measures : How to compare valuation261 methods.262 Aparna Elangovan , Jiayuan He , and Karin Verspoor . 2021 . Memorization vs. generalization : quanti-263 fying data leakage in nlp performance evaluation . arXiv preprint arXiv:2102.01818.264 Philip J. Fleming and John J. Wallace . 1986 . How not to lie with statistics : The correct way to265 summarize benchmark results . Commun . ACM , 29 ( 3 ) :218–221.266 Pengcheng He , Xiaodong Liu , Jianfeng Gao , and Weizhu Chen . 2020 . Deberta : Decoding-enhanced267 bert with disentangled attention . arXiv preprint arXiv:2006.03654.268 Junjie Hu , Sebastian Ruder , Aditya Siddhant , Graham Neubig , Orhan Firat , and Melvin Johnson . 2020.269 Xtreme : A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.270 In International Conference on Machine Learning , pages 4411–4421 . PMLR.271 Hang Le , Loïc Vial , Jibril Frej , Vincent Segonne , Maximin Coavoux , Benjamin Lecouteux , Alexandre272 Allauzen , Benoît Crabbé , Laurent Besacier , and Didier Schwab . 2019 . Flaubert : Unsupervised273 language model pre-training for french . arXiv preprint arXiv:1912.05372.274 Yaobo Liang , Nan Duan , Yeyun Gong , Ning Wu , Fenfei Guo , Weizhen Qi , Ming Gong , Linjun275 Shou , Daxin Jiang , Guihong Cao , et al . 2020 . Xglue : A new benchmark dataset for cross-lingual276 pre-training , understanding and generation . arXiv preprint arXiv:2004.01401.277 Ya lun Chou . 1969 . Statistical analysis : With business and economic applications.278 Christopher D Manning . 2011 . Part-of-speech tagging from 97 % to 100 % : is it time for some279 linguistics ? In International conference on intelligent text processing and computational linguistics,280 pages 171–189 . Springer.281 Bryan McCann , Nitish Shirish Keskar , Caiming Xiong , and Richard Socher . 2018 . The natural282 language decathlon : Multitask learning as question answering . arXiv preprint arXiv:1806.08730.283 Nikita Nangia and Samuel R Bowman . 2019 . Human vs. muppet : A conservative estimate of human284 performance on the glue benchmark . arXiv preprint arXiv:1905.10425.285 Sungjoon Park , Jihyung Moon , Sungdong Kim , Won Ik Cho , Jiyoon Han , Jangwon Park , Chisung286 Song , Junseong Kim , Yongsook Song , Taehwan Oh , et al . 2021 . Klue : Korean language under-287 standing evaluation . arXiv preprint arXiv:2105.09680.288 Fabio Petroni , Aleksandra Piktus , Angela Fan , Patrick Lewis , Majid Yazdani , Nicola De Cao , James289 Thorne , Yacine Jernite , Vladimir Karpukhin , Jean Maillard , et al . 2020 . Kilt : a benchmark for290 knowledge intensive language tasks . arXiv preprint arXiv:2009.02252.291 Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi292 Zhou , Wei Li , and Peter J. Liu . 2020 . Exploring the limits of transfer learning with a unified293 text-to-text transformer.294 Pranav Rajpurkar , Jian Zhang , Konstantin Lopyrev , and Percy Liang . 2016 . Squad : 100,000+295 questions for machine comprehension of text.296 Piotr Rybak , Robert Mroczkowski , Janusz Tracz , and Ireneusz Gawlik . 2020 . Klej : comprehensive297 benchmark for polish language understanding . arXiv preprint arXiv:2005.00630.298 Yutaka Sasaki et al . 2007 . The truth of the f-measure . 2007.299 Tatiana Shavrina , Alena Fenogenova , Anton Emelyanov , Denis Shevelev , Ekaterina Artemova,300 Valentin Malykh , Vladislav Mikhailov , Maria Tikhonova , Andrey Chertok , and Andrey Evlampiev.301 2020 . Russiansuperglue : A russian language understanding evaluation benchmark . arXiv preprint302 arXiv:2010.15925.303 James E. Smith . 1988 . Characterizing computer performance with a single number . Communications304 of the ACM , 31 ( 10 ) :1202–1206.305 Alan M Turing . 2009 . Computing machinery and intelligence . In Parsing the turing test , pages306 23–65 . Springer.307 Alex Wang , Yada Pruksachatkun , Nikita Nangia , Amanpreet Singh , Julian Michael , Felix Hill , Omer308 Levy , and Samuel R Bowman . 2019a . Superglue : A stickier benchmark for general-purpose309 language understanding systems . arXiv preprint arXiv:1905.00537.310 Alex Wang , Amanpreet Singh , Julian Michael , Felix Hill , Omer Levy , and Samuel R Bowman . 2019b.311 Glue : A multi-task benchmark and analysis platform for natural language understanding . In 7th312 International Conference on Learning Representations , ICLR 2019.313 Da-Feng Xia , Sen-Lin Xu , and Feng Qi . 1999 . A proof of the arithmetic mean-geometric mean-314 harmonic mean inequalities . RGMIA research report collection , 2 ( 1 ) .315 Liang Xu , Hai Hu , Xuanwei Zhang , Lu Li , Chenjie Cao , Yudong Li , Yechen Xu , Kai Sun , Dian316 Yu , Cong Yu , et al . 2020 . Clue : A chinese language understanding evaluation benchmark . arXiv317 preprint arXiv:2004.05986.318 1 . For all authors ... 319 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s320 contributions and scope ? [ Yes ] 321 ( b ) Did you describe the limitations of your work ? [ Yes ] 322 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] 323 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to324 them ? [ Yes ] 325 2 . If you are including theoretical results ... 326 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 327 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 328 3 . If you ran experiments ... 329 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-330 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] 331 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they332 were chosen ) ? [ Yes ] 333 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-334 ments multiple times ) ? [ N/A ] 335 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type336 of GPUs , internal cluster , or cloud provider ) ? [ N/A ] 337 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 338 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 339 ( b ) Did you mention the license of the assets ? [ Yes ] 340 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 341 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re342 using/curating ? [ N/A ] 343 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable344 information or offensive content ? [ N/A ] 345 5 . If you used crowdsourcing or conducted research with human subjects ... 346 ( a ) Did you include the full text of instructions given to participants and screenshots , if347 applicable ? [ N/A ] 348 ( b ) Did you describe any potential participant risks , with links to Institutional Review349 Board ( IRB ) approvals , if applicable ? [ N/A ] 350 ( c ) Did you include the estimated hourly wage paid to participants and the total amount351 spent on participant compensation ? [ N/A ] 352 _353 354 355

Summary: The benchmarks and leaderboards such as GLUE, SuperGLUE have been largely driving the progress of developing NLP models these days. A range of  models have already achieved super-human performance on these leaderboards.  This paper makes an argument that taking the arithmetic mean of scores across different tasks is not ideal, and instead, it proposes to use geometric mean and harmonic mean to re-compute the scores on multiple leaderboards. The findings are 1)  The ranking will be changed greatly by using geometric or harmonic mean; 2) The human performance baseline will rank highest on GLUE and SuperGLUE.