What are the best Systems? New Perspectives on NLP Benchmarking
Authors: Pierre Colombo
Nathan Noiry
Ekhine Irurozki
Stephan CLEMENCON
Venue: NeurIPS 2022 Accept (2022)

Abstract: In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in {\it (i)}  assessing the progress of new methods along different axes and {\it (ii)} selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (\textit{e.g.} GPT, BERT) that are expected to generalize well on a variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions. This paper proposes a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (\textit{e.g.} GLUE, EXTREM, SEVAL, TAC, FLICKR). In particular, we show that our method yields different conclusions on state-of-the-art systems than the mean-aggregation procedure while being both more reliable and robust.


What are the best systems? New perspectives on NLP Benchmarking
Pierre Colombo L2S
Nathan Noiry
S2A Telecom Paris, France
CentraleSupelec, France
pierre. hea.rrecccoobbbeeeerrreeeeeeeee.
nathan nathan. Srthaannnnrryyeellcoo---rrrrrsse
Ekhine Irurozki S2A
Stephan Clémençon
S2A Telecom Paris, France
Telecom Paris, France
ekhine. irurozki@@teeecoomppaaa fr
stephan.. clomenggon@tsleecmmppaiis
Abstract
In Machine Learning, a benchmark refers to an ensemble of datasets associated with one multiple metrics together with way aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (e.g. GPT, BERT) that are expected generalize well on variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on different scale, which may lead to spurious conclusions. This paper proposes new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (e.g. GLUE, EXTREM, SEVAL, TAC, FLICKR). In particular, we sate--of-the-ard
show that method yields different conclusions state-of- th-tthh-rtt systems than the mean-aggnn ssean-aggeegttin procedure while being both more reliable and robust.
Introduction
This paper is about improving current practices regarding benchmarks of NLP systems. As pointed out by [83], benchmarks are made of datasets, metrics, and way to aggregate performance. Our point that, if the bulk of the NLP community efforts on this domain about collecting new datasets and introducing new metrics, little work is concerned with the third part, namely how to aggregate various performances.
Why are benchmarks vital? Research advances in Machine Learning (ML) are crucially fueled by reliable evaluation procedures [34, 78]. The latter indeed mandatory to fairly compare new methods and systems. Usually, one relies well-chosen metric that reflects the ability to perform on task e.g. accuracy for classification, -quarrd error for regression.
The multi-tasks evaluation setting, If single-tasks problems are quite common, to best understand weakness and model in real-world scenario, the community is heading towards more complex
36th Conference on Neural Information Processing Systems (NcurIPS 2022),, New Orleans.

---

evaluations involving fine- thee-graied evaluation [61] across several metrics (or criteria [46, 103]) and several tasks [36, 47, 63, 89, 91, 108]. This is due to the increasing performance of deep neural networks, which are nowadays designed to generalize in great variety of situations and to solve complex tasks [86]. One is typically seeking for models with good transfer learning properties, meaning an ability to generalize well under distribution shift and/or task shift [51]-
How to aggregate performances? The multi-tasks setting has been investigated in recent works that provide benchmark of state-of-the-ar models across a great variety of tasks [28, 62, 80, 90, 108], sometimes with more than fifty [2, 84, 85, 94]. These papers provide tables of scores across the considered tasks, but the only non-qualitative way to compare systems consists in averaging the performances across tasks and then ranking systems according their mean score values. This is, for instance, done with the GLUE benchmark [91] and its derivatives [92]. However, taking the mean is seriously flawed since the different metrics are usually not on the same scales and can even
be unbounded [23, 102]. Even a pre-processing renormalization scheme would fail to capture the
intrinsic difficulty of the tasks.
Contribution 1. Our first contribution is to provide a reliable tool to rank systems in a multi-tasks setting. We rely on ranking aggregation procedure which, from a set of rankings induced by each criterion, returns a single ranking that somehow aggregates the former. This procedure, called the Kemeny consensus [52], can be seen as a voting rule and stems from the social choice theory [66].
Aggregation when instance-level information is available. As illustrated by Ruder [83], Zhong et al. [109], fine-grained understanding of the model performance should include instance-level scores. If taking the mean is quite natural in the classification setting, this is not always the case, as recently pointed out by [73] in the NLG setting. In this article, the authors investigate pairwise comparison of NLG systems for a single metric (e.g. BLEU [71], ROUGE [59], METEOR [5, 35, 49], CHRF [76, 77]. BertScore [105]). They prove that comparison based on the mean or the median of the scores across test utterances can be highly flawed. They rather advise to rely on the Bradley- Terry [10] pairwise comparison method, which consists, for two systems A and B, in computing the proportion of utterances on which A achieves a better score than B. Their work is a significant advance but remains limited to pairwise comparisons.
Contribution 2. Our second contribution consists in going one step further than [73] by applying our ranking procedure an arbitrarily large of NLG systems with respect group of fixed criterion. Our evaluation methodology can be seen as a natural extension of [73] since it coincides with the latter in the particular case of pairwise comparison. In more realistic multi-criteria scenario, combine our two contributions and develop a two-stages ranking aggregation procedure which first aggregates along utterances and then along criteria.
Experiments. Our two contributions rely on our aggregation procedure which is proved to be effective through several experiments.
1. We explain on a simple synthetic example the superiority of our approach compared to the mean-aggres aggrregagno procedure and the mairwise-aggregae procedure, both in terms of consistency and robustness.
2. We use our ranking procedure on 10 multi-tasks / multi-criteria benchmarks and observe leads to different conclusions than mean- and pairvise-aggreaaiion procedures.
3. We argue our procedure is more robust by investigating its stability with respect to the addition of criteria and with respect to the addition of systems.
Our code and the collected data will be released to accelerate the adoption of what we think is reliable evaluation method for multi-tasks and multi-criteria benchmarks.
2
Problem Formulation and limitations of existing methods
2.1 General Considerations
When comparing systems performances, two settings can be distinguished depending on the infor- mation granularity at our disposal and on the way one wishes to use this information. In general, each system is scored on each instance of test set with respect to a given metric. The final (single) score of the system with respect to this metric is obtained through aggregation procedure

---

tek
Figure 1: Illustration of the two considered frameworks relying on different information granularity: task-level information (below) or instance-level information(above) From the latter, one can derive the former relying what call an instance-leve aggregation. task- level aggregation then be per- formed to synthesize a system per- formance.
sptic
Lodk
astunnn-evvs Ragerapttion
Task-ievel Information
egprogation
will call instance-level aggregation, and which has be chosen by the practitioner (usually, the mean of the instances-scores), Then, the final benchmark score of the system is obtained through an aggregation we call task-level aggregation of the scores of this system for each metric of the considered benchmark. See Fig. for an illustration.
Notations.
given N systems evaluated on T 2 tasks, each task t E (1...., , being associated with a metric and a test set made K; 2 instances. For every n E {1...., N). t (1....,7) T and & E {I,-.., K,), we denote by 8n,t, E R the score of system of task
Suppose
the instance
Instance-level aggregation. The performance of system n on task is an aggregation of its scores (#n,,,)) on each instances. This aggregation is chosen by the practitioner and usually the mean-aggregation defined by 8 n.,
1 Eigkgk, 8n,1k- mucan == K.
only has access the aggregated scores of each system Task-level aggregation. Sometimes, on each task, that is, for every n E {1....,N) and E {1...., .) to a score Sn, E R which corresponds
instance-level aggregation for system n the instances of task t. From these quantities, one can then compute, for each system, a synthetic score reflecting the overall performance of this system across every considered task. Again, the usually-taken synthetic score of system is the mean- arean-aggregation: sn ;= + EiSEST Sn,t.
2.2 Problem Formulation
Ranking objective. When benchmarking systems, the goal is to output a ranking of each systems according to some objective criterion. Formally, we need to introduce the symmetric group on N elements, denoted by GN which elements are the N! permutations this notation, our goal is to output permutation
NN. Equipped with
= lo. R E GN.
corresponding the rankings of the systems. For instance, one reads "system i the a,-th best system". Depending on the granularity of the information at our disposal, we distinguish two problems.
Ranking Systems from Task Level Information. Given set of scores (sn,ti 1 n N, 15 t & T) of N systems on T tasks, find a proper aggregation procedure.
Ranking Systems from Instance Level Information. Given set of scores (Sn,t,k S n S N. < t T, 1 S k < K2) of N systems on the different instances of T tasks, find a proper aggregation procedure.
2.3 Limitation of Existing Methods
Mean-aggregation procedure. The aggreggggon procedure consists in taking the permutation of mean that would rank the aggregated means s 1 < N. This procedure suffers from several flaws. First, it is not well-suited when the metrics associated with the considered tasks are not on the same scale. Consider,
instance, the situation where one of the tasks (say task (o) is associated with metric that is at significantly larger scale than the others. In that case, the ranking obtained through mean-aggre agggegagggnnt would probably correspond to the ranking induced by task lo- One could argue that remedy would first normalize each metric so that everything is the same scale. However, the

---

Task
Task2
Task3
Task4
Task5
Task6
SUM
0.3 (3)
5 (3)
(1)
0.02 (2)
1.0 (1)
0.4 (3)
16.72 (13)
0.1 (2)
4 (2)
13 (2)
0.01 (1)
2.2 (3)
0.3 (2)
19.61 (12)
C
0.0
3
15 (3)
0.03 (3)
2.0 (2)
0.2
20.23 (11)
Table 1: Example of where pairwise rankings can be paradoxical. Mean aggregation outputs
A B C while pairwise ranking considered in [73] fails to rank the systems and produce
E> B,A= C. Our method does not have this flaw and outputs C>>B> A.
resulting aggregation would still fail to capture each task's intrinsic difficulty. Worse, this procedure is impractical in cases where some metrics are unbounded for instance, this is the case of the BARTSsoree [102]. Finally, another weakness of the mean- -aan-gggregttoon ranking procedure is that the score of a system is computed irrespective of its relative performance with respect to the others. This simple observation has been pointed out by [73] who advises, in the special case of two systems and one metric, to compute the number of times system is better than the other on the instances.
Pairwise ranking.To To be bit more formal, the pairwise ranking aggregation proposed by [73] to rank two systems A and B, which scores on given task are given by ......, $4 and , consists in computing
A Σ 12 and AB K XA-
Then, A is better than if and only if XA Åg. As explained by the authors, this method is more relevant than the mean-agent aggregation in the context of NLG evaluation. However, it is limited to the evaluation of two systems and does not apply for a general number N 3 of systems. solution would be to give rank for each pair of systems and then aggregate these pairwise rankings.
However, this would lead to prohibitive (2) computational factor for the complexity. Moreover, the conclusion of these pairwise rankings can be paradoxical. Tab. below provides a toy example
where three systems A, B, and C are evaluated on 6 tasks, and where the pairwise comparisons give the paradoxical conclusion B A, C > B and
Ranking via Kemeny consensus
We turn the description of methodology rank arbitrary number of systems multi-tasks / multi-criteria benchmarks.
3.1 Kemeny Consensus
Let consider the problem of ranking N systems on T tasks based on the information of the scores Sn,e of each system n on each task t. We believe a robust approach to this problem consist in relying the relative performance between systems each task. More precisely, for each task t {I,...,T) we consider
a' (oi --.....,000 o'yl GN.
where of corresponds to the rank of system task t, in decreasing order. Then, would like to find an appropriate procedure that aggregates the T rankings a. o'....,,"" o",, More formally, we would like to define function
f: GN x ... ....XX GN
GN-
T times
This function, from a set of T permutations corresponding to rankings, should return a final per- mutation that summarizes them. One difficulty is that the mean procedure makes no sense on the symmetric group, which is not vector space. It turns out that a very natural choice consists in taking the so-called Kemeny consensus [52] aggregation procedure, which somehow corresponds to compute a barycenter.
Kemeny consensus. Let d be the Kendall distance on the symmetric group, defined for every n. GN by d(n,,):= == E>E ((,=n))(s57))00 Kemeny consensus o² of ,...,," is solution of the following minimization problem mintge Eisisrdla's
Why is Kemeny consensus natural? proved by Young and Levenglick [100], the Kemeny consensus aggregation procedure is the only rule that satisfies three natural properties: neutrality,

---

meaning that it does not depend on the order of the tasks; consistency, meaning that if the tasks are split in two subsets and that the aggregation in the subsets rank system i above system j then oj >> of and the Condorcet criterion [27]. meaning that an item wining all its pairwise comparison ranked fist. Moreover, the Kemeny consensus is also the maximum likelihood of the widely-used Mallows statistical on the symmetric group [99].
3.2 Borda's 's count approximation
If the Kemeny consensus is the ideal objective one would like obtain, its computation is, in general, an NP-hard problem [6, 40] although some regularity assumptions, rarely satisfied in practice, can speed up the computation, see for instance [8] and [11]. Fortunately, there exist many ways to get satisfying approximations of the latter: see for example [1] for comprehensive empirical study, For our experiments, we choose the so-called Borda's count procedure, defined hereafter for the instance-level and/or task-level aggregation.
Borda's count. The Borda's 's count consists, from a set of permutations r ......."? .p E at correspond- ing the ranking of N systems across L tasks instances, to the ranks of each system and then to rank the obtained sums. Formally, it
1 1. Compute sum, == Σ the for every 1 ssns N,
2. Output 1)
Borda(1) .7)..r))) (argsort (argsort(sum), -....,num)))))))
GN that ranks the sums,
sum,
There least four explanations for choosing Borda's count procedure. First, it coincides with the pairwise ranking procedure in the case of two systems, making it a natural generalization. Second, there exists theoretical result assessing it is 5-approximation of the Kemeny consensus [30] with respect to the Kendall distance [42]. Third, it is an unbiased estimator of the Kemeny consensus with low sample complexity for data distributed according to standard rankings models such as Mallows [13, 44J. Fourth, from a practical perspective, [1] observe it is efficient, accurate, and actually 10 times faster than the other approximation algorithms. Fourth, We are now position to give our answers to the initial ranking problems from Task Level Information and from Instance
Level Information.
3.3 Our Ranking Procedures
How to rank Systems from Task Level Information. Let of E G be permutation that ranks the scores $1,45-.-1 &N Our aggregation procedure (a") output is o² = Borda o*).
How to rank Systems from Instance Level Information. We actually give two different procedures. For every task {1....,T} and every instance E (1...., ".....,,K] Kt} of that task. let gi.k be the permulation that ranks the scores $1,t,ki.... SN See Figure 2 for an illustration.
Two-level aggregation 2. ). This procedure
1. Compute o¹ Borda(o L.k S
K2) for
for each task / E (1....,Ty
Two kod apprrpriixe
VESIST Bedadath,
2. Output " Borda 1).
Figure 2: Illustration of our two aggregation pro- cedures to rank systems from instance-level in- formation.
One-level aggregation (o'). This procedure out- puts o' = BBrda( 1 t < T, < K,).
3.4 How to compare rankings
The rest of the paper is dedicated to synthetic and empirical experiments, on which we demonstrate the soundness of our approach. order to obtain quantitative result, one needs to be able to compare different rankings quantitatively. Two measures can be used for that purpose: (1) the Kendall
5

---

distance and (2) the Kendall correlation (7) [53, 54]. The Kendall distance computes the number of inversions between two permutations and therefore adapted for our purpose of ranking systems. The values of r range from [-1, 1] where the value of 1 corresponds to strong agreement, and close to indicates strong disagreement.
4 Synthetic Experiments
In this section, we validate on simulated data the performance of our method on two criteria: robustness to manipulation and robustness to scaling.
4.1 Data Generation
The toy experiment analysis is carried out on synthetic scores over N 20 systems, T 20 tasks and K = 20 instances. For each n E f1..... [T). we model the performance of system n by Gumbel r.v. Gn centered at Φ at scale B 1, where ó E [0, 1] is dispersion parameter. The scores of system n, (8n,,,))),R. are i.i.d. samples of Gn centered at á n with scale 8 = 1, where Φ [0, 1] is dispersion parameter. Moreover, the scores of different systems
sampled independently. Since Gn+1 G, follows logistic distribution with mean & at scale 1, this imply that P[ G2 > 0) > 0.5, the probability that system n + performs better than system is at least 0.5. Therefore, for all k, t, the rankings of systems is a realization of the ground-truth ranking [V], with a noise term controlled by the 'dispersion' parameter c.
Extreme scenarii correspond to the choices & = 0 and Φ = 1. More precisely, ø = 0 implies that all scores Sn,tkk have the same distribution, whereas ø 1 induces a strong consensus, i.e., clear system ranking emerges.
Remark 1. Sampling Sn,tk according to the described procedure is equivalent to sampling the ranking of the systems from the well-know know Plackett-Laee distribution [39, 75] with weights w; o+n. Interestingly, this distribution over ranking can be seen both from the utilitarian perspective in which the scores 8n,,k are real numbers and from ranking-modee perspective in which the ranking of systems have known distribution.
4.2 Robustness to manipulation
Setting. To test the robustness of our ranking procedure, we analyze its stability with respect to perturbations of the scores. More precisely, way to corrupt scores of a given task consists in sampling (sn, f, n),, as i.i.d. samples following Gumbel distribution centered at -n. This implies that, for that task t, the underlying ranking is [N]...]] namely the exact opposite of the ground truth [,. The robustness to manipulation analysis shows how the eтгоr on the final ranking of systems increases the scores of some tasks are 'corrupted'. Here, the error is computed relying on the normalized Kendall distance between the ground-truth ranking [1,...N and the ranking of systems obtained relying on the corrupted scores.
Results: For each of the considered methods, "mean of and 2 we report in Fig. 3 the results of the robustness analysis when and the number of corrupted task varies. The results of the robustness analysis show that " outperforms o' which at the same time consistently outperforms mean Overall, for the same number corrupted tasks and dispersion, the error of /² is always the smallest. Moreover, the score-based method o meen gets an error larger than .75 when just 2, 3, and 5 out of the total of T 20 tasks have been corrupted, while for o' the same error is achieved with 5, 7 and 10 corrupted tasks. The most robust method is the two-level o for which 10, 11 and 11 out T 20 tasks have to be corrupted to get the same error 0.75.
Takeaways: We conclude that the ranking-basee methods are more robust than o mean, In particular, the 2-level aggregation ²! is the most robust aggregation procedure.
4.3 Robustness to scaling
To further compare the ranking, we corrupt the scores of given task by re-scaling them by factor of x > O. Whereas it does not affect our ranking procedure (every ranking induced by task-instance pair remains the same), it increasingly perturbs the mean aggregation procedure as x increases. Re-scaling the scores by factor of 2 produces error larger than 90%. For larger ø. re-scaling the

---

1.0
100
033
G17
0.25
0.60
ni0 c)
02
rir
00
-1.0
Number of corapied Lasts
Top Systems
Top Systemm
Synthetic Scores
(a) Agreement
(b) Correlation Analysis
Analysis
Figure 3: Robustness on syn- thetic scores.
Figure 4: Global Analysis of Instance Level Raning
scores by a factor of produces the same enror (see Fig. 7 for detailed results). Takeaways: Re-scaling one task's 's score with arbitrarily large number will always produce an arbitrarily large error for mean aggregation while not affecting ranking based aggregation.
5 Empirical Experiments
In this section, we present our results on real evaluation scores, Our large scale experiments relies on real evaluation scores (over 270k scores) which are described in Ssec. 5.1. In Ssec. 5.2 gather experimental results for Ranking Systems from Task Level Information, while Ssec. 5.3 is dedicated the problem of Ranking Systems from Instance Level Information.
5.1 Data Collection
Datasets with Task Level Information We collect the results of GLUE [91], SGLUE [92] and
XTREME [50]. For GLUE the dataset is composed of N 105 systems that are evaluated on
different tasks: CoLA [93], SST-2 [87], MRPC [38], STS-B [14], QQP, MNLI [96], QNLI [80], RTE
31, 48] and WNLI [57]. For SGLUE, the final dataset gathers scores from N = 24 systems that
are evaluated on 10 different tasks: BoolQ [17]. CB [33], COPA [82], MultiRC [55], ReCoRD [104].
RTE, WiC [74]. WSC and its derivatives AX-b AX-g [57]. XTREM benchmark is composed of
N = 15 systems and include tasks such as sentence classification (using XNLI [29, 95] and PAXS-X
[98, 106]), structured prediction (relying on Universal Dependencies v2.5 [68] and Wikiann 79]),
sentence retrieval (with BUCC [110, 111] and Tatoeba [3]) and question answering (via XQuAD
[4, 80), MLQA [58), TyDiQA-GoldP [18]. For all benchmarks, various types of metrics with various scales are reported (i.e accuracy, fl, correlation).
Datasets with Instance-level information In this setting we focus on NLG evaluation as these scores are among the easiest to be collected. We focus on five different tasks: summary evaluation. image description. dialogue and translation. For summary evaluation, we use TAC08 [32]. TAC10, 10, TAC11 [69]. RSUM [9] and SEVAL [41]. For sentence-based image description we rely on FLICKR [101] and for dialogue we use PersonaChat (PC) and TopicalChat (TC) [64]. Finally for machine translation, we rely on the multilingual quality estimation (MLQE) introduced Ranasinghe et al.
[81]. For all datasets except MLQE, we consider automatic metric based on S3 (both variant pyr/resp)
[72], ROUGE [59] (including 5 of its variants [67]). [1-2] [60], Chrfpp [77], BLEU, BERTScore
[105]. MoverScore [107]. For MLQE we solely consider several version of BERTScore, MoverScore and ContrastScore. We also add human evaluation which is specific to each dataset. All details corresponding to these dataset can be found in Appendix A.
5.2 Task-level Aggregation Experiments
In this section, we address the aggregation problem when task-level information is available, We first study the final ranking obtained by different methods on GLUE, SGLUE, and XTREM. Then, assess the robustness of o" when removing tasks.
Comparison with mean-aggregation To compare the rankings o mean and a², we compute (i) the
Results can be found https:: https://super. httpgs//supeergguuemmmmmee.. gluebenchmark.co com/
7

---

Dalasel
Top
Top 3
Top 5
Top 10
GLUE
TTamm
Team
XT.
0.66
0.8
0.9
Alex
0:00::
0,003
ULR
GLUE
1
0.8
0.8
(320))
DEBURIA
2
2,40
1:000
(52.0)
(99.0)
SGLUE
0.9
3um
Dataset
L
Last 5
Last 10
T
5ume:
EXT.
0.8
0.9
0.82
U1RR=>
GLUE
0.8
0.7
0.92
DERL
10
7
HICTI
SGLUE
0.91
ELECTRA
10
Table 2: Agreement count between Top a/Last N systems on the Ranking when Task Level Information is available. T is computed on the total ranking.
Table 3: Qualitative analysis between rank- ing obtained with o" or a Results in parenthesis report the score of the considered aggregation procedure.
1.0
28
1.
26
10
2-
6.0
37
6-
1.3
Agsteg
fggrss
0.65
Agg kg
36
35
0.50
3.5
0.35 130
038
6:30 Nettetion
835
170
9813
0.25
5.50
675
1.0
3.0
0.25
0.88
6:75
0.66
0.25
330
6.5
200
Teals (4)
milerton
sriletion
% critetion
(a) GLUE
(b) Persona Chat
(c) Topic Chat
(d) FLICKR
(e) MLQE
Aggreg
3.0
05
3.0
14
0.80
0.86
24
3.
35
34
98
45
km
05
Aggre
255
2.4
6.65
1.30
0.8
625
6.50
875
1.30
0.0
6.5
35g
675
1
310
6.5
0.0
6.5
180
6.0
6.35
350
0.7
1.6
(f) EXTREM
(g) SUM Eval
(h) TAC08
TAC09
(j) TACI0
Figure 5: Impact of adding/removing metrics/tasks. The first column refers to ranking obtained with task-level information, while others columns refer to ranking obtained with instance-level information.
agreement rate (in %) which is the proportion of common top-ranked systems between o mean and o², and (ii) the Kendall Tau correlation (r) between the rankings.
Results. In Tab. 2, we compare the rankings of aforementioned methods for Top K systems (strongest systems) and Last K systems (weakest systems). For the three benchmarks, we observe high correlation between the final rankings (i.e. correlation values are in the range [0.80, 1)). To a finer degree, we also observe that methods tend agree on which are the best/worst systems. Although menn and a² agree on the best/worst systems, they do not rank them in the same order (see Tab. 3). For instance, on XTREM, the third-best system according o mean (rank 2) is actually the sixth-best system according to o". Takeaways. When changing the aggregation function, the response to our initial question "what the best systems?" varies.
How does the addition/removal of new tasks/metrics affect the ranking? When building a bench- mark, practitioners can always add new tasks refine the model performance assessment (it boils down to adding a new column in Tah. 1). In this experiment, we analyze how adding and removing tasks affect the rankings of the aggregation procedures.
Setting. We compare the rankings obtained when considering subset of the tasks and the one obtained using all the tasks. Formally, for a given number of tasks S T, we randomly sample tasks, compute the rankings obtain by our procedure and by the mean procedure, af and mean, on these t tasks, and finally compute the Kendall correlation between g. (resp. o (h and the "ground truth" a* (resp. o mean) ). We repeat this random sampling 100 times to obtain mean/variance plot of the correlations in the function of the number of sub-tasks.
Results. We report in Fig. 5(a.f) the obtained results for varying size of subsets. Interestingly
8

---

observe that correlation between the ~r and o" is consistently higher than the one between o o and Tmean This difference is particularly visible in the range (0.25% 0.75% We observe similar behavior when considering SGLUE (see Fig. 8)
Takeaways. The ranking from o² is more robust task addition/drop than the one from meen.
5.3 Instance-level Aggregation Experiments
For instance level aggregation, we conduct experiments the 9 aforementioned data-sets. We study both the final ranking obtained for each aggregation (i.e. o², 2 and o mean as well as the effect of task addition/removal.
Global analysis When conducting our experi- ments, observe that the three different aggre- gation procedures lead to three different state- of-the-art in 8 datasets out of 9. Furthermore, they never agree on the top systems. In what follows, compare o², 0²l and meenn
PC
TC
FLI.
MLQE
-0.08
-0.01
-0.03
Tomme 22)
0.32
0.27
0.29
0.01
Tarmaan of)
-0.10
-0.15
-0.04
0.00
RSUM
SEVAL
TAC08
TAC09
TACI1
0.04
0.14
0.28
0.06
-0.06
Setting. We compare the obtained ranking by (1) comparing the Kendall correlation (see Fig. 6). (2) comparing the number of agreements between the top N systems, (3) computing the Kendall correlation between them.
0.07
0.52
0.32
0.37
0.37
0
0.10
0.23
0.19
0.07
Figure on global instance-level rankingg..
Results. When considering the agreement analysis of Fig. 4(a), we observe that の and meen select a high number of common top systems. However, the correlation the rankings induced by o² and meen these top-systemn is low (see Fig. 4(b)). This is also the case for the correlation of the entire rankings (see Fig. 4). In short 2 and o mean select similar systems but rank them differently. Similar analysis shows that af disagrees from 2 and mean both top systems and their orders.
Takeaways. 2¹ exhibits more similar behavior than of with respect to mean
What is the impact removing/adding tasks? NLGG, different metrics (i.e. task) assess the quality of generated sentence along different axis. As adding ranking, we investigate the impact of task addition/removal on the final system ordering.
metric may affect the final
Setting. Similarly to ??, we study the evolution between the correlation between the ranking com- puted on a subset of tasks and the ground truth ranking (computed on all tasks) for each of the three procedures.
Results. We observe that for datasets both 2 of obtain higher correlation and lower variance compared to meen when adding/removing tasks. Results for RSUM reports a similar trends (see Fig. 9).
Takcaways. The ranking obtained with either of or 2 are more robust to task addition/drop than the one from mean
Conclusion and Future Works
In this paper, we introduced new aggregation procedures to rank systems when either task level scores instance level scores are available. Our methods, which are theoretically grounded and rely on Kemeny ranking consensus, address fundamental flaws of the widely used arithmetic mean.
We conducted extensive numerical experiments, which show that our methods are both more reliable and more robust than the mean aggregation while leading to different conclusions on which are the best systems. Overall, when task-level (resp. instance-level) information is available, we would recommend using the aggregation procedure o" (resp. o 2!) rather than the mean (resp. o mean and r').
Although we focused on NLP benchmarks, our methodology could be applied in other modalities (e.g. Computer Vision, Audio). Another interesting avenue would be to consider a weighted version of the Kemeney consensus where weight could reflect task-specific criteria (e.g fairnsss, robustness, complexity).

---

7 Acknowledgements and Fundings
We thank Maxime Peyrard, Saibo Geng, and Wei Zhao for sharing their collected dataset. A warm thanks to Matthieu Labeau for his sharp eyes and excellent comments when reading our paper manuseript. This work was granted access to the HPC resources of IDRIS under the allocation 2021-101838 and 2021-101913 made by GENCI. Nathan is funded by the projet ANR LIMPID.
10

---

References
[1] Alnur Ali and Marina Meilä. Experiments with kemeny ranking: What works when? Mathe- matical Social Sciences, 64(1):28 40, 2012.
[2] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv:21 /.1.: 10952, 2021.
[3] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Lingwistics, 7:597-610, 2019.
[4] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. arXiv preprint arXiv: 1856, 2019.
[5] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[6] John J Bartholdi, Craig A Tovey, and Michael A Trick. The computational difficulty of manipulating an election. Social Choice and Welfare. 6(3):227-241, 1989.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[8] Nadja Betzler, Michael R Fellows, Jiong Guo, Rolf Niedermeier, and Frances A Rosamond. How similarity helps to efficiently compute kemeny rankings. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume Volume 1, pages 657-664, 2009.
[9] Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re- evaluating evaluation in text summarization. arXiv preprint arXiv:2010.077000 100, 2020.
[10] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(344):324-355, 1952.
[11] Felix Brandt, Markus Brill, Edith Hemaspaandra, and Lane A Hemaspaandra. Bypassing combinatorial protections: Polynomial-time algorithms for single-peaked electorates. In Twenty-Fourth AAAI Conference Artificial Intelligence, 2010.
[12] Róbert Busa-Fekete, Dimitris Fotakis, and Emmanouil Zampetakis. Private and non non-private uniformity testing for ranking data. Advances in Neural Information Processing Systems, 34, 2021.
[13] Ioannis Caragiannis, Ariel D Procaccia, and Nisarg Shah. When Do Noisy Votes Reveal the Truth? In Proceedings of the Fourteenth ACM Conference on Electronic Commerce, EC 13, pages 143-160. ACM, 2013. ISBN 978- -777----503-192--1 doi: 10.114574425540248257777 URL http:// /ttp:://do...ee acm. acm. org/ / 10. /11111//////444411 2482570.
[14] Daniel Cer, Mona Diab, Eneko Agirre, Inigo task 1: Semantic textual preprint arXiv:1708.0000 2017.
and Lucia Specia. Semeval-2017 -uuaatty-mmtiiliguull and cross-lingual focused evaluation. arXiv
[15] Emile Chapuis, Pierre Colombo, Matteo Manica, Matthicu Labeau, and Chloe Clavel. Hierar- chical pre-training for sequence labelling in spoken dialog. arXiv preprint 111.522009.11152. 2020.
[16] Emile Chapuis, Pierre Colombo, Matthicu Labeau, and Chloe Clavel. Code-switched inspired losses for generic spoken dialog representations. arXiv preprint arXiv:2108. 12465,, 2021.
[17] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.1004.. 2019.

---

[18] Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qu: A benchmark for information-seet seeking question answering in typologically diverse languages. Transactions of the Association for Computa- tional Linguistics, 47454 2020.
[19] Pierre Colombo. Learning to represent and generate text using information measures. PhD thesis, Institut polytechnique de Paris, 2021.
[20] Pierre Colombo, Wojciech Witon, Ashutosh Modi, James Kennedy, and Mubbasir Kapadia. Affect-driven dialog generation. arXiv preprini arXiv: 1904.02793, 2019.
[21] Pierre Colombo, Emile Chapuis, Matteo Manica, Emmanuel Vignon, Giovanna Varni, and Chloe Clavel. Guiding attention in sequence-to to-sequence models for dialogue act prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7594-7601,
2020.
[22] Pierre Colombo, Emile Chapuis, Matthieu Labeau, and Chloe Clavel. Improving multimodal fusion via mutual dependency maximisation. arXiv preprint arXiv:2109.0099 2021.
[23] Pierre Colombo, Chloe Clave, and Pablo Piantanida. Infolm: A metric to evaluate 2021.
summarization & data2text generation. arXiv preprint
[24] Pierre Colombo, Chloe Clavel, and Pablo Piantanida. A novel estimator of mutual information for learning to disentangle textual representations, arXiv preprint arXiv:2105.02685, 2021.
[25] Pierre Colombo, Guillaume Staerman, Chloe Clavel, and Pablo Piantanida. Automatic text evaluation through the lens of wasserstein barycenters. arXiv preprint arXiv:2108. 12463, 2021.
[26] Pierre Colombo, Chouchang Yang, Giovanna Varni, and Chloé Clavel. Beam search with bidirectional strategies for neural response generation. arXiv preprint arXiv:2110.03389 2021.
[27] N. Condorcet. Essai sur I'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix. de l'Imprimerie Royale, 1785.
[28] Alexis Conneau, German Kruszewski, Guillaume Lample. Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv: arXivv/805.001 1805.01070, 2018.
[29] Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.0500 2018.
[30] Don Coppersmith, Lisa Fleischer, and Atri Rudra. Ordering by weighted number of wins gives good ranking for weighted tournaments. In Proceedings of the seventeenth annual ACM-SIAM symposium Discrete algorithm, pages 776-782, 2006.
[31] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177-190. Springer, 2005.
[32] Hoa Trang Dang, Karolina Owczarzak, et al. Overview of the tac 2008 update summarization task. TAC, 2008
[33] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124, 2019.
[34] Mostafa Debghani. Yi Tay, Alexcy A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint 2Xiv:2107.07002. 2021.
[35] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evalu- ation for any target language. Proceedings of the ninth workshop on statistical machine translation, pages 376-380, 2014.
12

---

[36] Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, et al. NI- augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02221,200 2021.
[37] Tanvi Dinkar, Pierre Colombo, Matthieu Labeau, and Chloé Clavel. The importance of fillers for text representations of speech transcripts. arXiv preprint arXiv:2009.. 1340, 2020.
[38] William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (IWP2005), 2005.
[39] Luce R Duncan. Individual choice behavior: A theoretical analysis, 1959.
[40] Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference World Wide Web, pages 613-622, 2001.
[41] Alexander Fabbri, Wojciech Kryšciñski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409, 2021.
[42] Ronald Fagin, Ravi Kumar, and Dakshinamurthi Sivakumar. Comparing top lists. SIAM Journal on discrete mathematics, 17(1):134-160, 2003.
[43] Michael A Fligner and Joseph S Verducci. Distance based ranking models. Journal of the Royal Statistical Society, 48(3):359-366 1986.
[44] Michael A Fligner and Joseph S Verducci. Multistage Ranking Models. Journal of the American Statistical Association, 83(403):892 1988. ISSN 01621459, doi: 10.2307/ 2289322. URL http::tt://w//t
[45] Alexandre Garcia, Pierre Colombo, Slim Essid, Florence d' A'chéé Buc, and Chloé Clavel. From the token to the review: A hierarchical multimodal approach to opinion mining, arXiv preprint arXiv:1908.1 1216, 2019.
[46] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Berez-Belltacchini. Creating training corpora for nig micro-planning. In 55th annual meeting of the Association for Computational Linguistics (ACL), 2017.
[47] Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondíej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major Simon Mille, Emiel van Miltenburg. Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96-120, Online, August 2021. Association for Computational Linguisties. doi: 10. 10.18853/v1/2221 http://///221mmmmmm-e -ee--emmmmm 1.10. URL attps://aclantaae
org. (trg/2021.0 gem- 10.
[48] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9, 2007.
13

---

[49] Yinuo Guo and Junfeng Hu. Meteor++ 2.0: Adopt syntactic level paraphrase knowledge into machine translation evaluation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Pupers, Day 1), pages 501-506, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.186535/1/11955337.. URL https: //www. //wpw.aclll aclweb.w............ org/ hergg/nntolggy//////ee aanthology/W19955557 W31--557.
[50] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411-4421. PMLR, 2020.
[51] Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.
[52] John G Kemeny. Mathematics without numbers. Daedalus, 88(4):577-591, 1959.
[53] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938.
[54] Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-251,
1945.
[55] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pages 252-262, 2018
[56] Anna Korba, Stephan Clémençon, and Eric Sibony. A learning theory of ranking aggregation. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, AISTATS 2017, 2017.
[57] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference the Principles Knowledge Representation and Reasoning, 2012.
[58] Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MIqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv: 191007755 2019.
[59] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pages 74-81, 2004.
[60] Chin- Lin, Guihong Cao, Jianfeng Gao, and Jian- Yun Nie. An information-thcorrtic approach to automatic evaluation of summaries. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 463-470, 2006.
[61] Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaicheng Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Zi-Yi Dou, and Graham Neubig. Explainaboard: An explainable leaderboard for nip. arXiv preprint Stt::2204.06387 2021.
[62] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural lan- guage decathlon: Multitask learning as question answering. arXiv preprint arXiv: 08730, 2018.
[63] Angelina McMillan- Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammana- manchi, Sebastian Gehrmann, and Yacine Jermite. Reusable templates and guides for docu- menting datasets and models for natural language processing and generation: A case study of the HuggingFace and GEM data and model cards. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), I), pages 121-135, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.186553w1/2201..em URL https:t//////. aclanthology.
httphooooogyyoggg/2/// 002/2021. / gem-1. 1111.
[64] Shikib Mehri and Maxine Eskenazi. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.00455. 2020.
14

---

Jean-Yves [65] Volodymyr Mnih, Csaba Szepesvári, and Jean- Yves Audibert, Empirical bernstein stopping. In Proceedings of the 25th international conference on Machine learning, pages 672-679, 2008.
[66] Roger B Myerson. Fundamentals of social choice theory. Technical report, Discussion Paper, 1996.
[67] Jun-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for rouge. arXiv preprint 15xiv:150., 06034, 2015.
[68] Joakim Nivre, Mitchell Abrams, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, et al. Universal dependencies 2.2 (2018). URL http://ddl handle. net/1 1234/1-198xxxx LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University, Prague, http://hdl. handle. net/1 1191xx..
[69] Karolina Owezarzak and Hoa Trang Dang. Overview of the tac 2011 summarization track: Guided task and aesop task. In Proceedings of the Text Analysis Conference (TAC 201 1), Gaithersburg, Maryland, USA, November, 2011.
[70] Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, 2017.
[71] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 1-118, 2002.
[72] Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, 2017.
[73] Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. Better than average: Paired evaluation of nlp systems. arXiv preprint arXiv:2110.10746, 2021.
[74] Mohammad Taher Pilehvar and Jose Camaaho-Cooladoo Wic: the word-in con-int dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808 09121, 2018.
[75] Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 193-222, 1975.
[76] Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, 2015.
[77] Maja Popović. chrf++: words helping character n-grams, In Proceedings of the second conference on machine translation, pages 612-618, 2017.
[78] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings the Thind Conference Machine Translation: Research Papers, pages 186-191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https: //wps::///www../ aclweb. org/ anthology/ W18-6319.
[79] Afshin Rahimi, Yuan Li, and Trevor Cohn. Massively multilingual transfer for ner. arXiv preprint 4rXiv:1902.00//9; 2019.
[80] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 100.::6606.05250000 2016.
[81] Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. An exploratory analysis of multilingual wevellevel quality estimation with cross-lingual transformers. arXiv preprint arXiv:2106.00144 2021.
[82] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 201 AAAI Spring Symposium Series, 2011.
15

---

[83] Sebastian Ruder. Challenges and Opportunities in NLP Benchmarking, http://ruder.. 1o/ hepp-enchmarknng 2021.
[84] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 1 2021.
[85] Aditya Siddhant, Junjie Hu, Melvin Johnson, Orhan Firat, and Sebastian Ruder. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. 2020.
[86] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam,, Mare Lanetot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):
484 489, 2016.
[87] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[88] Guillaume Staerman, Pavlo Mozharovskyi, Stéphan Clémençon, and Florence d' Alché Buc. Depth-based pseudo-metrics between probability distributions. arXiv e-prints, pages arXiv- 2103, 2021.
[89] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 1. 1,0006, 2020.
[90] Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, and Christopher Pal. Would you rather? a new benchmark for learning machine alignment with cultural values and social preferences. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pages 5369-5373, 2020.
[91] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1eXiv:8804.077 2018.
[92] Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for ceneral-purposs language understanding systems. arXiv preprint arXiv: arXiv:1905.000 09001095500077 2019.
[93] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-644 641, 2019.
[94] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv: 200.:::2009015222 2021.
[95] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.054400 2017.
[96] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challeng corpus for sentence understanding through inference. arXiv preprint arXiv: 2017.
[97] Wojeicch Witon, Pierre Colombo, Ashutosh Modi, and Mubbasir Kapadia. Disney iest 2018: Predicting emotions using an ensemble. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 248-253, 2018
[98] Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv preprint arXiv:19088 / 2019.
[99] H Peyton Young. Condorcet's theory of voting. American Political science review, 82(4): 1231-1244, 1988.
16

---

[100] H Peyton Young and Arthur Levenglick. A consistent extension of condorcet's 's election principle. SIAM Journal on applied Mathematics, 35(2):285-300, 1978.
[101] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions.
Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[102] Weizhe Yuan, Graham Neubig. and Pengfei Liu. Bartscore: Evaluating generated text text generation. arXiv preprint arXiv:2106. 1520, 2021.
[103] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30 (9):8805-2824 2019.
[104] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.1200 12885, 2018.
[105] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint 19Xv::900.006775 2019.
[106] Yuan Zhang. Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXivv1904.000 130, 2019.
[107] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv: /909.00000000 02622, 2019.
[108] Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, and Zhilin Yang. Fewnlu: Benchmarking state-of-the-art methods for few-shot natural language understanding. arXiv preprint 1xxv:2109.127/44 2021.
[109] Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language models uniformly better? comparing performance at the instance level. arXiv preprint M.Kiv:2005.00020 2021.
[110] Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the second buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60-67, 2017.
[111] Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 1 11th Workshop on Building and Using Comparable Corpora, pages 39-42, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to code and datasets? [N/A]
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers.Nott that the Checklist section does not count towards the page limit.In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
17

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our works aim at developing safe ai systems.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Both in the supplementary and main paper
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the supplementary.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] In the supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A].
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A].
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
18


Reviews:
[{"review_id": "ZDoIXhvUxlK", "reviewer": "Reviewer_1ozF", "summary": "The paper proposes to address system benchmarking across several tasks as a ranking optimization problem and proposes a solution essentially based on aggregating preference rankings using Borda count. This is validated on synthetic data and using a large amount of real benchmarking on multiple NLP tasks, illustrating the benefits of the proposed method over simple averaging.\n", "questions": "\n1. Why is notation changing in Sec. 3.2., e.g. L tasks instead of T? Is this trying to convey something?\n2. Please clarify the two argsorts on l.175?\n3. Can you clarify how Borda count applies to KxT ranks over T tasks in one-level aggregation (l.188)? \n4. Could you address the issue of ties mentioned in the previous section? How do you handle those?\n\nMisc typos: \n\nl.79: way, one -> way one\n\nl.121: impracticable -> impractical\n\nl.159: to -> two\n\nl.162: Mallows statistical -> statistic?\n\nl.172: it -- if in-line, verb agreement on l. 173-174 should be 3rd person, otherwise remove 'it' \n\nl.205: where φ ∈ [0, 1] is a dispersion parameter [redundant with l.203]\n\nFig. 4 is oddly positioned on p.7 when it is referenced on p.9. Typo in caption: \"Raning\"\n\nl.295: I think /sigma^{mean} should be /sigma^{mean,t}\n\nl.326-334: Shouldn't Fig. 5 (b-e) and (g-j) be referred to somewhere here?\n\nl.329: ?? [ref missing]", "limitations": "I don't see any potential negative societal impact of this work, beyond the potential negative societal impact the entire field of NLP may have. In fact, by addressing the sensitive issue of properly assessing progress and improvements in NLP system, this work may have positive impact and help discard spurious improvements that may actually harm society.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\nThe paper takes a different view on multiple benchmark aggregation, which is welcome especially in NLP, given that evaluation in the field is usually poorly performed and simplistic. The motivation is clear, the proposed solution (Borda count) is simple and effective. Its benefits are well argued and tested in an experimental section that uses results from a large amount of different tasks and systems.\n\n*Weaknesses:*\n\nReadability: Some figures (Figs. 2,3,4) are hard to read properly due to small font, even with on-screen magnification (Tab. 3 also). In addition, Figs. 3-5 are not self-contained, and not properly explained in the text either. As a consequence, the information the reader gets from these plots is sadly limited.\n\nAlso, The flip side of the strength above is that there is an awful lot of experimental results packed into the last two pages or so. So much in fact that it is difficult to get a clear picture of what is going on without referring to the supplementary material.\n\nLimitation: One issue that is not addressed at all as far as I can tell is the handling of ties, an issue on which various variants of Borda counts differ. This is particularly relevant in this context for at least two reasons: First, proper benchmarking should consider significance (statistical or practical) of differences, and systems that are not significantly different would typically be considered as \"tied\". Second, many NLP tasks (e.g. categorization) produce discrete outputs and discrete scores, which would therefore yield many similar instance-level scores. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OJ6tuwTJ7vu", "reviewer": "Reviewer_BuYF", "summary": "This paper proposes a ranking method for comparing NLP systems that does not simply take the average of target scores. The authors propose to use a method called Kemeny consensus in which Kendall distance or correlation is used for choosing the best permutation (as a ranking candidate). The authors apply their method to simulated data, showing their ranking is robust to manipulation and scalable. In the experiments using real-world task sets such as GLUE and a set of NLG tasks, the authors show that their proposed ranking and mean-aggregation ranking correlate to some extent (but have some difference) and that their ranking is robust to adding and removing tasks.", "questions": "There are some grammatical and stylistic errors in the paper.\n\nMinor:\n- what is EXT in Table 2?", "limitations": "The authors only use NLG tasks for their instance-level analysis, but other types of tasks, such as machine translation and question answering, should be considered.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper demonstrates that there might be a way of ranking systems in benchmarking that is different from simply taking the average of instance or task performance and comparing those scores. Such a ranking method may yield different outcomes in comparing systems, so it is highlighted that we need to be aware of the possibility of a different interpretation when doing the mean-aggregation comparison.\n\nHowever, I do not understand the assumptions the authors have in this study. I have the following concerns:\n\n- I think the paper fails to emphasize the necessity of ranking systems by aggregating benchmarking scores in some way. In other words, I do not think there should be the best ranking when benchmarking because the best model has to be determined depending on a pragmatic purpose.\n- I am not sure if the proposed ranking method captures the intrinsic difficulty of tasks, which seems one of the main goals in this paper. When we interpret benchmarking results, the task difficulty needs to somehow be associated with human evaluation or external measures. However, in this study, the proposed ranking method only considers the relative difficulty regarding the evaluated systems.\n- I do not necessarily think that ranking has to be robust to adding or removing tasks. Different systems may have different abilities that do not allow any sequential ranking of systems. Therefore, adding and removing tasks may change an ideal ranking.\n\nI think these concerns weaken the substance of their experiments and contributions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ghun1FEUEg6", "reviewer": "Reviewer_qERs", "summary": "This paper proposes a method for ranking a set of systems based on their performance across multiple tasks. The authors argue that the naive method of averaging task scores can be biased if the scores have different ranges. They advocate ranking in order of summed per-task ranks instead (Borda's count procedure), citing theoretical support for this approach. A variant \"two-level\" algorithm, applicable when instance-level scores are available, first applies the Borda procedure to combine per-instance system rankings within each task, then applies the procedure again to combine the resulting per-task rankings.\n\nThese methods are compared to ranking using mean scores on both synthetic and real data. On synthetic data, the two-level approach is found to be most resistant to scoring noise. On a large number of NLP tasks, both proposed methods (standard and two-level) are more robust to adding or dropping tasks. They assign global rankings to systems that are similar to the baseline - putting the same systems near the top or bottom - but make fine-grained adjustments, sometimes resulting in different overall winners.\n", "questions": "43: This implies that your method does capture task intrinsic difficulty, but I can’t see that this is the case.\n\n133: I don’t really buy the complexity argument. Even if there are O(100) systems, there are still only O(10^4) pairs, which seems completely manageable.\n\n149: If you are willing to relax the interpretation of ranks as permutations, and think of them as scores, then taking the mean rank actually seems quite intuitive. It’s basically the same as the Borda approximation, except that means contain more information than ranks, e.g. systems whose average ranks are 1.1 and 1.2 are closer than would be implied by the Borda ranks of 1, 2.\n\nFigure 2: typo in the “task t” column: K_2 -> K_t, K_1 -> K_t\n\n3.3 This description is a bit confusing. I imagine both schemes produce a per-task ranking that gets combined using Borda, with the difference being that in one-level the per-task ranking comes from mean scores across instances, whereas in two-level the per-task ranking comes from applying Borda over instances. If that’s the case, consider adding a bit of text like the above, or making the notation and diagram more expressive.\n\nTable 3: What are the scores in the sigma* column? If these are the Borda rank sums, I would expect the top-ranked systems to have smaller values.\n\n329: Missing reference.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Addresses an important problem, and proposes a simple, well-motivated solution that performs well. The idea isn’t novel, but it isn’t well known in NLP, and I think it has the potential to be widely adopted.\n\n- Does a good job motivating the proposal and citing extensive relevant literature.\n\n- Clearly organized and well presented, with a few exceptions detailed below.\n\n- Carries out experiments on a broad range of task settings.\n\nWeaknesses:\n\n- A bit thin on substance. A lot of the paper is taken up with a formal presentation of the method, which is quite easy to grasp intuitively. Some more space might have been devoted to analysis of the results, to provide more evidence that the new rankings are better (a difficult problem, however).\n\n- An obvious objection to summarizing scores with ranks is that it throws away information about how close scores are to each other. One way to deal with this is to allow ties, for instance in cases where there isn’t statistical support for assigning different ranks. The whole aspect of confidence intervals on the scores is completely missing from the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept