Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a dataset, a trained evaluation metric and a leaderboard for evaluating discourse phenomena for machine translation. They test this on a range of discourse level translation models and develop metrics which evaluate the models according to their performance on four discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. 

Strengths:
This paper delivers multiple contributions which could have significant impact on the field of discourse level machine translation. They release data for three language pairs and using their method one could extend it relatively easily into others. I like the thoughtful way that the authors find examples of hard discourse phenomena and each phenomena requires distinct handling. 

Weaknesses:
They rely on previous work to create the Anaphora test set and evaluation model (Jwalapuram et al. (2019)). They should have explained at a high level how the Jwalapuram evaluation model works and they should have given a general idea of the rules used to filter the anaphora test set: how many rules, an example, would this be possible to do for other languages or does it only work well for English? 
I would have liked more discussion about the extensibility of their approach into languages other than English. 
They should have used sacrebleu or at least specified if the BLEU scores were tokenised and true cased. 

I think this paper should be accepted because of combined strength of the dataset/metric/leaderboard. I think fine grained evaluation of hard phenomenon is the way forward for improving already very good MT models. 


review_title: Resources for discourse in translation

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors propose specific test sets for document-level NMT. They target anaphora, coherence/readability, lexical consistency and discourse connectives, and are available for multiple language pairs. The first two challenge sets rely on model-based evaluation.

Strengths:

The test sets, which target various discourse phenomena, directly evaluate the output of the models (contrarily to some existing multiple-choice challenge sets).

The types of mistakes made by NMT models are manually examined.

The authors validate the quality of their metrics by comparing against human judgements (although the number of samples is arguably small).

Weaknesses:

The evaluated NMT models all date from 2018 or earlier.

The anaphora challenge sets are only a minor update over previous work.

All language pairs use English as the target language.

Other remarks and questions:

For the anaphora and coherence/readability test sets, future work may "cheat" by using the evaluation models as part of the NMT systems.

Is normalizing the scores actually useful? The reference scores should be the same across all systems, so it only shifts all results by a constant and doesn't affect relative performance.

Whats steps would be needed to construct similar test sets for En->X language pairs?
review_title: Progress towards better evaluation of context-aware NMT

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a benchmark for discourse phenomena in machine translation. Its main novelty lies in the relatively large scale, spanning three translation directions, four discourse phenomena, and 150-5000 data points per language and phenomenon. A relatively large number of systems from previous work is benchmarked on each test set, and agreement with human judgments is measured.

positives:

- clearly, a lot of thought and effort has gone into the creation of the benchmarks, and this is the most diverse set of benchmarks for discourse phenomena in MT so far. 

- extensive experiments with models from previous work, along with human analysis.

negatives:

- two of the four benchmarks, anaphora and coherence, are evaluated by neural models trained on WMT outputs, so the interpretation of scores is opaque, and their validity is unclear. Specifically, Jwalapuram et al. (2019) train a neural network to distinguish references from MT output based on the ELMo representations of pronouns, but in principle, this model can use signals other than the correctness of pronouns translation to make this distinction. Similarly, the model by Moon et al. (2019) was originally trained to distinguish real documents from randomly shuffled ones, and I can see how their complex neural network would then learn to rely on coherence features. However, this submission uses reference translations as positive, MT output as negative examples, so it again may learn to use features other than coherence for its decisions.

- also, I'm not fully convinced about the validity of the automatic discourse connective evaluation. According to the manual analysis, there is a large proportion of false negatives (synonymous translations flagged as errors), and rankings would change if synonyms were counted as correct. I was also not satisfied with the evidence that the omission of connectives is generally an error. The human study was a bit simplistic in that it just deleted connectives (although more changes might be needed to ensure grammaticality) or used noisy MT output rather than alternative human references. It also seems to have been monolingual. If the test set consists of examples with ambiguous or implicit discourse relations in the source, then it may actually be the right translation strategy to omit them. I'm worried that a benchmark that rewards explicitation and punishes leaving discourse relations implicit may set the wrong incentives.

- I was surprised by the low results (already in terms of BLEU) of some of the tested variants. Authors describe in great care their efforts to fairly reproduce previous work, including the use of original code and hyperparameters where possible, but I can't help but think that the models are suboptimally trained, and that statements about whether context-aware models consistently improve discourse phenomena are tainted by this. 

recommendation:

I'm leaning negative on the current version of the paper and benchmark. I think the test sets have been carefully assembled, and along with the various types of models evaluated on them, this work has value. But before I'd recommend that the benchmarks actually be used in the field, authors would need to improve upon the evaluation scores used for anaphora, coherence, and discourse connectives and make sure they really are targeted towards the phenomena they claim to measure, and do not have large blind spots.

further questions and minor problems:

- did you perform early stopping? What was your stopping criterion?

- do you have an explanation why anaphora scores differ wildly between systems for ZH-EN (table 9)? 

- the discourse connectives test sets are based on examples where the reference contains a connective, but MT output does not. What MT system was used? Do the respective source segments generally contain explicit discourse connectives, or are the respective discourse relations generally implicit in the source?

review_title: commendable, large-scale test sets; are metrics targeted and trustworthy?

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents four methods for creating benchmarking datasets, each focusing on a particular discourse phenomenon which is difficult/hard to solve with existing context-aware neural machine translation (NMT) models.  It also evaluates several existing NMT models using the created data and makes a remark about their current status, such as the superiority of one method in one task.  This work has the originality and I am happy to know that some researchers devote their efforts to address these issues.  Moreover, I am sure that the field of NMT definitely benefits from this work, once the created data are made publicly available.

However, I am very skeptical that the contribution of this paper fits for ICLR, since the methodology presented in this paper itself mainly consists of a pile of human efforts.  In other words, if I understand the contents properly, only the technical advancement presented in this paper is TgtCon, a variation of the anaphora-centric method proposed by Voita et al. (2018b).  However, it cannot be a substantial merit to the community, since it does not necessarily perform better than existing methods and the reason of its deterioration is not analyzed.

Considering the main focus of the paper, i.e., creating benchmarking datasets, the paper should rather fit for a journal article in the field of natural language processing, where the authors can give more details in creating the datasets, such as procedure, tool, and attributes of annotators, not in appendices, and more careful analyses of the results, including WHY each method does (not) perform well on a particular discourse phenomenon.

Indeed, when I was reading this paper, I suffered from the fact that the main part of the paper is not self-contained.  For instance, the proposed TgtCon is not explained in Section 2 but the readers are advised to see an appendix.  Other information that is indispensable in data creation but missing in this paper is the detail of human judgment, such as the proficiency of the evaluators, protocol, and judgment criteria.  Each section reports on agreement ratio, but not Cohen's kappa.  With these reasons, I am not completely convinced of the quality of the resulted datasets and the portability of the proposed methods.

Below shows some questions.

Q1. Is there a reason to choose these four particular discourse phenomena?  Are they exclusive to each other?

Q2. Test sets for coherence/readability and discourse connectives are significantly smaller than those for the other two phenomena.  I am not sure that a test set with 200 or less examples is enough.

Q3. In Section 3.2.1, the authors regard the inconsistency of named entities.  However, in many error classification schemes, terminology errors are, irrespective of incorrect translations and inconsistency, distinguished from coherence errors.  For instance, the Multidimensional Quality Metrics (*1) locate them in different branches.

(*1) http://www.qt21.eu/mqm-definition/

Q4. What does "Random translations" in Section 3.3.1 mean?
review_title: Valuable investigations into discourse phenomena but does not fit for ICLR

Meta-review: This paper introduces a dataset and a trained evaluation metric for evaluating discourse phenomena for MT. Several context-aware MT models are compared against a sentence level baseline. The paper develops metrics which evaluate the models according to their performance on four discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. Data is released for three language pairs (all using English as the target language). 

First, I’d like to point out that creating datasets and benchmarks for analyzing/evaluating discourse-level errors in machine translation is an extremely valuable contribution. This paper is addressing a very relevant problem and even though there is no new model/method/algorithm being proposed, this work *fits* this conference - it is my opinion that the community should welcome and value more than it currently does the efforts spent in creating high quality datasets that can help make progress in the field.

There was substantial discussion among reviewers about this paper. 

The main weaknesses raised by the reviewers were:
- Limited information about the process to create the anaphora test, which was a contribution of prior work (Jwalapuram et al. (2019) - this was addressed in the updated version; but the anaphora challenge sets seem to be only a minor update over previous work.
- All language pairs use English as the target language, and it is not simple to extend this approach beyond English target languages.
- Lack of detail on how BLEU scores were computed (tokenised? true cased? My recommendation is to use sacrebleu) - this was clarified in the rebuttal.
- The evaluated NMT models all date from 2018 or earlier.
- Two of the 4 benchmarks (anaphora and coherence) are evaluated by neural models trained on WMT outputs, which makes the interpretation of scores is opaque, and their validity is unclear.

While the creation of a benchmark for discourse evaluation of MT is a laudable effort as mentioned above, it is my opinion that due to some of the weaknesses above the current version of this work is not yet ready for publication. However, I strongly encourage the authors to improve upon these points and resubmit their work to another venue. I list some suggestions below to improve this paper.

My biggest concern with the current version is the last weakness above. As pointed out by a reviewer, the framework of Jwalapuram et al. (2019) provides empirical support for the model's sensitivity (if there is a pronoun error, does the metric pick it up?). But they don’t necessarily capture model *specificity* (if the metric ranks one output higher, can we be confident that this is because of a pronoun translation error?). For the coherence metric, authors make an argument that their metric is sensitive to coherence issue, but concerns remain about whether it is sufficiently specific to these issues. In the rebuttal, authors argue that BLEURT is sentence-level, but they could easily aggregate sentence-level judgments and report correlation between BLEURT and human coherence  judgments to show that their metric correlates better with human coherence judgments than BLEURT or even just BLEU. Besides BLEURT, I would add there are other recently proposed metrics that may capture discourse phenomena (neural metrics trained against MQM annotations or sentence-level human assessments with document context) and should be compared against: check COMET [1] or PRISM [2] (the latter is sentence-based but could be adapted for paragraphs or documents).

There is also prior work comparing various context-aware machine translation approaches against a sentence-level baseline, some with negative findings [3,4,5]. I suggest the authors look at this related work in future iterations of their paper.

[1] https://arxiv.org/pdf/2009.09025.pdf

[2] https://arxiv.org/pdf/2004.14564.pdf 

[3] https://www.aclweb.org/anthology/2020.eamt-1.24.pdf

[4] https://arxiv.org/pdf/1910.00294.pdf 

[5] https://www.aclweb.org/anthology/2020.emnlp-main.81.pdf 

Decision: Reject