Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to address system benchmarking across several tasks as a ranking optimization problem and proposes a solution essentially based on aggregating preference rankings using Borda count. This is validated on synthetic data and using a large amount of real benchmarking on multiple NLP tasks, illustrating the benefits of the proposed method over simple averaging.

strengths_and_weaknesses: *Strengths:*

The paper takes a different view on multiple benchmark aggregation, which is welcome especially in NLP, given that evaluation in the field is usually poorly performed and simplistic. The motivation is clear, the proposed solution (Borda count) is simple and effective. Its benefits are well argued and tested in an experimental section that uses results from a large amount of different tasks and systems.

*Weaknesses:*

Readability: Some figures (Figs. 2,3,4) are hard to read properly due to small font, even with on-screen magnification (Tab. 3 also). In addition, Figs. 3-5 are not self-contained, and not properly explained in the text either. As a consequence, the information the reader gets from these plots is sadly limited.

Also, The flip side of the strength above is that there is an awful lot of experimental results packed into the last two pages or so. So much in fact that it is difficult to get a clear picture of what is going on without referring to the supplementary material.

Limitation: One issue that is not addressed at all as far as I can tell is the handling of ties, an issue on which various variants of Borda counts differ. This is particularly relevant in this context for at least two reasons: First, proper benchmarking should consider significance (statistical or practical) of differences, and systems that are not significantly different would typically be considered as "tied". Second, many NLP tasks (e.g. categorization) produce discrete outputs and discrete scores, which would therefore yield many similar instance-level scores. 
questions: 
1. Why is notation changing in Sec. 3.2., e.g. L tasks instead of T? Is this trying to convey something?
2. Please clarify the two argsorts on l.175?
3. Can you clarify how Borda count applies to KxT ranks over T tasks in one-level aggregation (l.188)? 
4. Could you address the issue of ties mentioned in the previous section? How do you handle those?

Misc typos: 

l.79: way, one -> way one

l.121: impracticable -> impractical

l.159: to -> two

l.162: Mallows statistical -> statistic?

l.172: it -- if in-line, verb agreement on l. 173-174 should be 3rd person, otherwise remove 'it' 

l.205: where φ ∈ [0, 1] is a dispersion parameter [redundant with l.203]

Fig. 4 is oddly positioned on p.7 when it is referenced on p.9. Typo in caption: "Raning"

l.295: I think /sigma^{mean} should be /sigma^{mean,t}

l.326-334: Shouldn't Fig. 5 (b-e) and (g-j) be referred to somewhere here?

l.329: ?? [ref missing]
limitations: I don't see any potential negative societal impact of this work, beyond the potential negative societal impact the entire field of NLP may have. In fact, by addressing the sensitive issue of properly assessing progress and improvements in NLP system, this work may have positive impact and help discard spurious improvements that may actually harm society.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a ranking method for comparing NLP systems that does not simply take the average of target scores. The authors propose to use a method called Kemeny consensus in which Kendall distance or correlation is used for choosing the best permutation (as a ranking candidate). The authors apply their method to simulated data, showing their ranking is robust to manipulation and scalable. In the experiments using real-world task sets such as GLUE and a set of NLG tasks, the authors show that their proposed ranking and mean-aggregation ranking correlate to some extent (but have some difference) and that their ranking is robust to adding and removing tasks.
strengths_and_weaknesses: This paper demonstrates that there might be a way of ranking systems in benchmarking that is different from simply taking the average of instance or task performance and comparing those scores. Such a ranking method may yield different outcomes in comparing systems, so it is highlighted that we need to be aware of the possibility of a different interpretation when doing the mean-aggregation comparison.

However, I do not understand the assumptions the authors have in this study. I have the following concerns:

- I think the paper fails to emphasize the necessity of ranking systems by aggregating benchmarking scores in some way. In other words, I do not think there should be the best ranking when benchmarking because the best model has to be determined depending on a pragmatic purpose.
- I am not sure if the proposed ranking method captures the intrinsic difficulty of tasks, which seems one of the main goals in this paper. When we interpret benchmarking results, the task difficulty needs to somehow be associated with human evaluation or external measures. However, in this study, the proposed ranking method only considers the relative difficulty regarding the evaluated systems.
- I do not necessarily think that ranking has to be robust to adding or removing tasks. Different systems may have different abilities that do not allow any sequential ranking of systems. Therefore, adding and removing tasks may change an ideal ranking.

I think these concerns weaken the substance of their experiments and contributions.

questions: There are some grammatical and stylistic errors in the paper.

Minor:
- what is EXT in Table 2?
limitations: The authors only use NLG tasks for their instance-level analysis, but other types of tasks, such as machine translation and question answering, should be considered.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for ranking a set of systems based on their performance across multiple tasks. The authors argue that the naive method of averaging task scores can be biased if the scores have different ranges. They advocate ranking in order of summed per-task ranks instead (Borda's count procedure), citing theoretical support for this approach. A variant "two-level" algorithm, applicable when instance-level scores are available, first applies the Borda procedure to combine per-instance system rankings within each task, then applies the procedure again to combine the resulting per-task rankings.

These methods are compared to ranking using mean scores on both synthetic and real data. On synthetic data, the two-level approach is found to be most resistant to scoring noise. On a large number of NLP tasks, both proposed methods (standard and two-level) are more robust to adding or dropping tasks. They assign global rankings to systems that are similar to the baseline - putting the same systems near the top or bottom - but make fine-grained adjustments, sometimes resulting in different overall winners.

strengths_and_weaknesses: Strengths:

- Addresses an important problem, and proposes a simple, well-motivated solution that performs well. The idea isn’t novel, but it isn’t well known in NLP, and I think it has the potential to be widely adopted.

- Does a good job motivating the proposal and citing extensive relevant literature.

- Clearly organized and well presented, with a few exceptions detailed below.

- Carries out experiments on a broad range of task settings.

Weaknesses:

- A bit thin on substance. A lot of the paper is taken up with a formal presentation of the method, which is quite easy to grasp intuitively. Some more space might have been devoted to analysis of the results, to provide more evidence that the new rankings are better (a difficult problem, however).

- An obvious objection to summarizing scores with ranks is that it throws away information about how close scores are to each other. One way to deal with this is to allow ties, for instance in cases where there isn’t statistical support for assigning different ranks. The whole aspect of confidence intervals on the scores is completely missing from the paper.
questions: 43: This implies that your method does capture task intrinsic difficulty, but I can’t see that this is the case.

133: I don’t really buy the complexity argument. Even if there are O(100) systems, there are still only O(10^4) pairs, which seems completely manageable.

149: If you are willing to relax the interpretation of ranks as permutations, and think of them as scores, then taking the mean rank actually seems quite intuitive. It’s basically the same as the Borda approximation, except that means contain more information than ranks, e.g. systems whose average ranks are 1.1 and 1.2 are closer than would be implied by the Borda ranks of 1, 2.

Figure 2: typo in the “task t” column: K_2 -> K_t, K_1 -> K_t

3.3 This description is a bit confusing. I imagine both schemes produce a per-task ranking that gets combined using Borda, with the difference being that in one-level the per-task ranking comes from mean scores across instances, whereas in two-level the per-task ranking comes from applying Borda over instances. If that’s the case, consider adding a bit of text like the above, or making the notation and diagram more expressive.

Table 3: What are the scores in the sigma* column? If these are the Borda rank sums, I would expect the top-ranked systems to have smaller values.

329: Missing reference.
limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes to address system benchmarking across several tasks as a ranking optimization problem and proposes a solution essentially based on aggregating preference rankings using Borda count. This is validated on synthetic data and using a large amount of real benchmarking on multiple NLP tasks, illustrating the benefits of the proposed method over simple averaging. Reviewers questions regarding how ties are resolved, taking task difficulty, and use-case were addressed convincingly by the authors. I hope they revise their manuscript so that it is easier to follow, the results section is packed with information, too many graphs to the point where finding information is difficult. I would strongly urge the authors to remove some analyses in the appendix. It would be good to also tell us how the various systems are being evaluated. 
confidence: Certain
Decision: Accept