Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a method to select reference samples from a MT dataset that are useful to discriminate among the output of MT systems, aiming to reserve sentences for which MT systems produce different/diverse outputs.

The idea of the paper is that given X metrics that are used to evaluate the models, discriminative reference samples must be those that cause the systems to score very differently among them. This essentially translates into very easy and difficult sentences being little discriminative, as all the system will obtain very good or bad scores, respectively.

The authors show that their variance-aware subsets obtain higher correlations with respect to human judgements than the original datasets (WMT16 to WMT20).

strengths: 1.  The premise and motivation of the work is valid, IMO. If the community designs a dataset, for MT or any other NLP task, where all the models perform very close, then these sentences are not really useful to score and discriminate among systems. We should rather focus on the sentences that actually can discriminate among systems. According to the authors (lines 32-33), previous attempts are based on human and automatic paraphrasing and are human or computational expensive. On the contrary, the authors method seems intuitive and it’s inspired by simple statistics, arguing that output showing high score variance (those showing more variance than x% of the test samples) are the ones that serve to discriminate the systems.
weaknesses: 1. The authors show that their technique improves correlations to human judgements. However, I feel the authors also would need to show whether the reference-based metrics and system ranking change or not on that subset, to check whether it would be actually crucial to consider their technique in MT standard evaluations.

2. In the contributions (lines 49-58), the authors mention that their approach reduces high computational costs, since it reduces the size of the test set for RL and NMT systems. Yet, I expect the bottleneck of the computational cost is on the training set, as the test set should be executed (ideally) just once. In this context, I wonder how the approach truly contributes to ‘green AI’ as all sentences are used during training. Note that the test set is just a representation of the real world, where the model will be deployed and translate huge amounts of sentences, including very easy and very difficult ones. Would it be possible to expand on this?

3. I see how the methods can be used to present variants of MT datasets and benchmarks, but the paper itself does not present one. Although I don’t see this as a major weakness, I also feel the fit with this track might not be perfect.

correctness: Yes, the main claim of the paper is correct.
clarity: The paper is overall well-written. I have some minor questions to the authors that I stated in the additional feedback.
relation to prior work: Yes. The authors argue that the main difference is that previous efforts focused on increasing the reference diversity through human and automatic paraphrasing, but that these methods are expensive.
additional feedback: 1. Is ‘lightweight’ (line 1) used here as a synonym of ‘small’?
2. Could you specify what you exactly mean by BERT-family models in line 167?
3. What are the MT models used to compute the outputs? I might have missed it, but I only can see a mention of ‘system hypotheses’ in line 150.
4. The goal of the paper is to obtain the samples so that the translations for those samples show a higher correlation with respect to the human judgements in comparison to the whole test set, which the authors achieve. However, what if we simply remove references where the scores by the systems are very low, wouldn’t we potentially obtain a higher correlation? Although easy sentences might be useless to discriminate among systems, I presume they would help to obtain even a higher correlation which is what this paper shows, or I might be missing something? I’m not saying it’s something it should be done, but could it be something that happens?
5. Could Figure 4 specify what is considered a short, medium or long sentence?
6. The human paraphrasing paragraph (lines 245-259) was not clear to me, if the paper is accepted, would it be possible to detail it a bit more?
review_title: A method to preserve only discriminative sentences in MT datasets

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces automatically generated, variance-aware test sets for machine translation, including 70 test sets covering 35 translation directions based on the datasets from WMT16 to WMT20. 

The authors first describe the problems of machine translation evaluation and highlight the shortcomings of the currently available approaches. They then describe the concept of variance-awareness and explain the novel filtering methodology to generate the test sets. 

The paper makes a significant contribution to the field of machine translation evaluation by not only providing the datasets but providing a automated methodology for generating variance-aware tests sets from different datasets and providing detailed analysis of discriminative and indiscriminative test instance. 
strengths: The paper is well written with a clear structure. The paper provides a novel method of automatically generating high-quality test sets for machine translation evaluation and publishes the generated test sets to promote reproducibility and to allow researchers to use the generated test sets for evaluating machine translation models. The authors also promise to release the code for automatic test set generation which will be a highly valuable asset for future research.
weaknesses: In general there is not many weaknesses in the paper. The only section that I found a bit lacking was the 3.1 Motivation, which describes the motivation behind variance and variance-aware test sets. I am not sure the example from entrance examinations is the best one. I would propose rewriting the section focusing on why variance-awareness is important in machine translation in particular.
correctness: Based on the information provided in the paper and without seeing the actual code, the datasets seem to be constructed in a sound way. Review of the source code would be required to make sure the implementation is correct.
clarity: The paper is clear and well written for the most part. As stated above, I would suggest rewriting the section 3.1 Motivation. 
relation to prior work: The discussion of related work is detailed with good references to previous work. 
additional feedback: Good work! I look forward to seeing this paper published.
review_title: A great addition to machine translation evaluation toolkit

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents a technique to filter machine translation test sets in order to keep a set of instances that allow to better evaluate and compare systems, as well as a set of test sets (for 70 language pairs) generated using this technique. The main idea is straightforward: to keep those test instances where the performance of different systems (as measured by one of a set of possible metrics) has a large variance, and filter away those where the performance is very similar (be it because they are very easy, and all the systems get them right, or very fall such that they all fail). The resulting test sets can be evaluated and compared to the original test sets in terms of correlation to human judgments. The authors perform such an evaluation using a range of different MT evaluation metrics, variance thresholds and correlation coefficients, concluding that correlation to human evaluation typically increases with respect to the original test sets (i.e., the smaller filtered test sets provide a better evaluation of systems, at least measured against the human standard). They also include some analysis of the kept vs. filtered instances in terms of characteristics like sentence length, PoS tags or translation difficulty, which provides some insight on what the technique is doing.
strengths: While the technique is simple, I think it is an interesting approach, and correlation with human evaluations shows that the resulting test sets meet their intended purpose of providing a better evaluation while being smaller. Thus, I think this contribution will have a meaningful impact and will be used. 

The presentation and analysis is rather thorough: several MT evaluation metrics are considered (which also serves, indirectly, as a comparison between metrics), as well as several correlation coefficients, apart from the hyperparameter search for the optimal variance percentile threshold for each metric.

The paper not only provides the technique and its evaluation, but also some insights on the kind of instances that the system is keeping and removing.
weaknesses: It would have been nice to have some language-specific analysis in Sections 4.2 and 4.3 (e.g. does the technique work better for some language pairs than others?) apart from 4.4, although I understand it can be difficult for space reasons.
correctness: The experimental design is thorough and well explained, experiments are extensive and the claims are sound as far as I can see, with one small example: I am not sure I totally understand the conclusions of the "Human paraphrasing" analysis in Page 9. If I understand correctly, what Table 4 says is that the method tends to reserve the sentences from the original test set that are more similar to the paraphrases. But where does the conclusion that "deeper reason of correlation improvement brought by VAT may be similar to the effect of human paraphrasing" come from? Can't this table be explained, for example, by the fact that for more discriminative examples, humans generate closer paraphrases (be it due to their length, the kind/frequency of words they contain, etc.) In my view the claim of that conclusion is not supported, although it might be that I'm missing some point of the analysis and it can be fixed by some clarification or rephrasing.

The rest of the analyses all look sound and clear to me.
clarity: The paper is well written and clear, except for the issues with the "Human paraphrasing" part mentioned above. 

Several references are made to "the entrance examination" which I suppose refers to the Chinese entrance examination (gaokao). Perhaps this should be specifically mentioned for the sake of readers from other cultures which may not find the reference obvious (or another option is to just make the analogy with exams in general, as what is said is arguably true of any exam).

The paper would probably benefit from some proof-reading - I found a bunch of typos and grammar issues (listed in the "Additional Feedback" below) but I may have missed more, although there's nothing that affects the understandability of the content.
relation to prior work: The paper is adequately contextualized with respect to prior work.
additional feedback: I confirm that I have read the author rebuttal. It has reaffirmed my positive view of the paper.

Why does BLEURT behave so differently with respect to Spearman's rho (in Figure 3c) with respect to the other two correlation coefficients? It achieves good results with Pearson and Kendall but rather poor ones in terms of Spearman's rho. I think this merits a comment.

Figure 7 would be easier to see if the bars were non-overlapping (using side-by-side bars of different colors).

l. 19: metrics that exploits -> metrics that exploit

l. 73: do not consider semantic -> do not consider semantics

l. 76: making evaluate the semantic overlap possible -> making it possible to evaluate the semantic overlap

l. 104: paraphrased references have been proved that it can significantly improve -> paraphrased references have proved that they can significantly improve

l. 120: a discriminative test instance must make the evaluation exhibits -> a discriminative test instance must make the evaluation exhibit

l. 144: since it is the well-recognized benchmarks -> since they are the well-recognized benchmarks

l. 181: the metrics driving by the pre-trained models -> the metrics driven by the pre-trained models

l. 219: are like to be -> are likely to be
review_title: A simple, nice way of creating compact and useful evaluation datasets

Meta-review: The paper presents a new evaluation dataset for machine translation, focusing on sampling test examples that can effectively rank different systems. The evaluation set contains a wide range of language pairs (35 language pairs) and shows a higher correlation with human judgments than original, unfiltered datasets. 

All reviewers agreed that the paper points out a new perspective on the MT task, and the same idea of filtering evaluation datasets to improve discrimination ability can be applied for other tasks. The paper also includes in-depth analysis, such as what kind of sentences are getting filtered, and how it connects to human paraphrasing. My initial concern about using same models to compute variance and then to evaluate output was addressed during the rebuttal period. 
Decision: Accept