Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ### Summary
This paper proposes a multi-pass generation process for NMT. It introduces an evaluator model that learns to score reference translations higher than model outputs, and serves as a policy to determine how many refinements to make during inference, and which samples to prioritize during training. Samples that have a low quality score are prioritized, such that the translation (rewriter) module is trained with a focus on low-quality inputs where rewriting can still yield improvements.
The performance of this rewriter-evaluator is evaluated on Chinese-to-English and English-to-German benchmark tasks with RNNs and Transformer architectures, and shows superior quality than previous works. Training time is reported to be comparable to a standard encoder-decoder, while inference is roughly 4 times slower. In an ablation analysis, the influence of individual components on output quality (such as a copy mechanism and hyperparameters, e.g. the maximum number of iterations) is compared. In addition, the relevance of the evaluator during inference is demonstrated in a comparison with an oracle sentence selection.

To summarize, I'd like to see this paper accepted if it addresses the questions below and makes an effort to increase reproducibility. The ideas are novel and the combination of training algorithm and architectural modifications seem effective. In seq2seq NLP applications where inference time is not a concern, the proposed methodology could improve quality in practice.

### Strengths
- The prioritized training with model outputs is novel. The idea is neat, and solves the problem of having to pick model outputs of different quality for training.
- The proposed training and models show great improvement over previous benchmarks (disregarding inference effort) on two language pairs and with two different architectures.
- The ablation shows the individual contribution of the proposed set of modifications and the performance under different hyperparameter settings.
- Training and inference time requirements are analyzed.
- The paper is written clearly and therefore easy to follow.

### Weaknesses
- Reproducibility: The code is not published, and half of the experiments are on non-freely available datasets (NIST LDC). The experiments on Zh-En could have been performed on a public benchmark like WMT 2017 as well.
- Comparison with previous works: It is unclear whether these are Transformer-based or RNN-based, and how they relate to the implementation of this paper (e.g. if the baseline MLE scores are the same), and if hyperparameter were tuned equally well. In addition, inference effort in terms of rewriting iterations is not considered in the comparison.
- Analysis: There are a few open questions (see below) that would strengthen the paper and highlight the effectiveness and the workings of the novel elements.

### Details
- Geng et al. 2018 do not report any issues with variance in their experiments, so I'd refrain from using generic RL statements to argue that their work is inferior.
- Related work in Active Learning (see below), Curriculum Learning (e.g. Zhou et al. 2020 (https://www.aclweb.org/anthology/2020.acl-main.620.pdf), and Quality Estimation (e.g. Zho et al. 2020 (https://arxiv.org/abs/2005.03519)) often use model scores directly for measuring model competence/quality/confidence, so it would be insightful to replace q with model scores to analyze if the evaluator is learning anything beyond what the rewriter already captures. See also Lee et al. 2020 (https://arxiv.org/abs/2002.07233) for relating model scores to translation quality.
- Eq 1 introduces only 2 arguments for z, but Eq. 6 feeds 3.
- The parameters are confused for the losses in Eq. 6 and 7.
- Algorithm 1: Why not directly pushing quadruples into A? There is no filter between l13 and l16.
- For the analysis, it would be great to compare how many passes each training samples receives on average during training, and how it relates to how many passes are done in inference. This would help to understand the importance of seeing model outputs during training.
- Does the baseline MT also use a pointer generator? If not, it would be good to add this for a more fair comparison.
- With longer inputs, the rewriter also performs more computation than the normal MT Transformer. Therefore, I'd like to see a comparison to a simple MT baseline where the source is concatenated twice (as described in Sec. 4) as input to the encoder. If there is no benefit from doing that, the advantage must come from inserting previous outputs.
- How are hyperparameters delta and K determined? And how large is the queue? Limiting the training to 3 epochs seems quite low.
- The comparisons to previous work are inconsistent: It is not clear which scores come from reimplementations and which ones are taken from previous papers. The reimplementation should also directly be compared to the original paper to validate the quality.
- Table 1 hides the number of iterations/rewrites. It would be great to add, to make the advantages of the proposed method shine better.
- The ablation should be done on the dev set rather than the test set.


### Related Work
- The prioritized queue is related to works in Active Learning and Curriculum Learning. Relating to these works might inspire an improvement of the algorithm or some shared insights, e.g. Liu et al. 2018 (https://www.aclweb.org/anthology/K18-1033.pdf), Platanios et al. 2019 (https://www.aclweb.org/anthology/N19-1119.pdf). 
review_title: Effective and novel multi-pass decoding for NMT

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors complement iterative refinement for neural machine translation with an evaluator model that controls the termination of the translation process. Their approach is an alternative to the policy network used in [1].

The translation process is terminated when the predicted quality of the current translation is inferior to the previous one by a margin of $\delta$ or more, or after a fixed number of iterations otherwise. To train the models, the authors propose a sampling mechanism that prioritizes low-quality translations.

Strengths:

The proposed approach is fairly intuitive and improves upon two baselines over multiple datasets.

The paper is mostly clear and the authors give enough details to reproduce experiments, at least for En-De.

Weaknesses:

The Zh-En training data is not publicly available.

Comparison to previous work is not that convincing because most use baselines that are significantly worse than Transformer. The oracle experiment is only conducted with RNNSearch.

Questions:

In Table 3, I assume you display training time per epoch. What is the training time to convergence? Given that the sampling process is auto-regressive (i.e. conditioned on previous tokens), how can your approach have a speed comparable to standard training?

Did you train models without the priority queue, simply training over a fixed number of iterations for each example? If so, what are the results?

Why do further iterations sometimes decrease translation quality?

[1] Geng et al. Adaptive multi-pass decoder for neural machine translation. EMNLP, 2018.
review_title: Mostly solid work, but results are not fully convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a rewriter-evaluator framework for multi-pass decoding, where translation hypothesis at last iteration is refined by the rewriter and further evaluated by the evaluator to decide whether to terminate the iterative decoding process. Compared to previous studies, the evaluator offers this framework the capability of flexibly controlling the termination. The authors also propose a prioritized gradient decent algorithm that biases the training process to those low-quality translation samples. Experiments on NIST Zh-En and WMT15 En-De demonstrate that the proposed model significantly outperforms several strong baselines.

Pros:

= The idea behind the rewriter-evaluator framework is easy to follow.

= The proposed method achieves significant performance improvement against several multi-pass baselines on both Zh-En and En-De translation tasks.

= The authors demonstrate that the proposed training algorithm has similar training time to the vanilla baseline, i.e. no training time loss (Table 3).

Cons:

= Some model details are missing, and the NIST Zh-En training data is not publicly available so it’s hard to exactly replicate the experiments.

= Although the framework enables flexible termination, the evaluator requires a threshold that has a large impact on translation quality and must be carefully tuned (Table 2).

= Baselines and model optimization should be further improved to fully convincing readers.

My detailed comments are as follows:

1.	The authors claim that their model can better handle the termination. One important experiment is to testify how many iterations the model uses for translating one sentence, and what factors could affect the iteration number, such as sentence length (would long inputs require more passes?). Particularly, it would be great to have an experiment to compare the termination difference between the proposed model and the Adaptive Multi-pass Decoder (Geng et al., 2018), and show evidence how the proposed one outperforms Geng et al., 2018 (i.e. RL-based model).

2.	The authors didn’t provide full model details about how to combine source encoder outputs (x) and target encoder outputs (z) in the decoder part, i.e. more details about Ea. (4) are required. 

3.	Compared to other baselines, the authors adopt the copy mechanism. Could you please provide an ablation study to justify its impact, such as retraining a model using the rewriter-evaluator framework without copying?

4.	For Transformer, the authors concatenate the source input and target translation but disables cross-attention over them. Could you please give some explanation behind this practice? What if we allow the cross-attention here?

5.	Using non-public dataset, like NIST Zh-En, is not suggested in my opinion. Other researchers might not be able to replicate this experiment at all. Running experiments with (C)WMT Zh-En would be a better alternative. Besides, using tokenized BLEU with ‘multi-bleu.perl’ is also not suggested nowadays. Use sacrebleu and report the signature, instead.

6.	The results, in Table 1, for WMT15 En-De on newstest2015 are not convincing. Based on my own experience, a standard Transformer-base model can already achieve a tokenized BLEU score of ~29. I think this weak baseline comes from the fact that the authors train Transformer with batch size of 80 and optimize model parameters with RMSProp, as shown in experimental settings, Section 5.1. The authors should update their experimental training.

7.	If I understand correctly, Algorithm 1 requires online decoding process, that is, performing greedy or beam search decoding during training to get real-time estimation for q and r. In my experience, the decoding is very time-consuming. However, results in Table 3 show that there is almost no training time difference! Could you please show some theoretical explanation about this? How did you perform decoding, or get z^k from z^k-1 in practice?

If the authors can address all my concerns, I’d like to update my scores. 

review_title: interesting idea but many questions not addressed

final_score: rating: 4: Ok but not good enough - rejection
review_content: ** Summary **
In this work, the authors proposed a rewriter-evaluator framework for neural machine translation (NMT). The translation is achieved by an iterative way: at the $k$-th iteration, a rewriter generates a translation sequence $z^{(k)}$ based on the source input $x$ and the output of previous iteration $z^{(k-1)}$. The evaluator is used to provide a score for $z^{(k)}$, used to indicate when to stop refinement. 
The authors used priority queue to organize the training samples, where the sequences with low translation quality will be processed first. 
The authors conduct experiments on NIST Zh->En and WMT’15 En->De.

** Pros **
(1) Proposed a new method for improving the NMT in an iterative way.

** Cons **
(1) Experimental results not convincing.
(2) The method lacks a strong explanation why it works.

** Clarify **
I think the clarify of the current version should be greatly improved.
1. What is the configuration of your Transformer and what does ``we use default settings in fairseq (Ott et al., 2019)’’ mean? Please give the specific number of layers, hidden dimensions, dropout, batchsize here. By the way, according to my knowledge, people always use Adam for optimization in NMT. Why do you choose Rmsprop?
2. Where is Figure 5.2?

** Significance **
1. About the evaluator: In Section 2, I do not get how $q^{(k)}$ is obtained in detail. It is not enough for me to get Eqn.(4) only. If you mean that the detailed definition is in Eqn.(12), I do not think it is a good measure for quality estimation, since it only measures the similarity between source input the previous translation in some hidden space. That is, the evaluator is not well-defined.
2. Experimental results not convincing. Why don’t you evaluate on WMT’14, where the baselines are provided in (Ott et al., 2019)?
3. The sacreBLEU score should also be reported (at least, in the appendix), so that we can compare your results with more algorithms in a fair way.
4. It is better to give several cases on how your method gradually improves the translation quality.


** About the baseline **
I am not satisfied with results of WMT’15 En-De as you reported in Table 1. 
There are some pre-trained checkpoints on English->German, which is available at 
https://github.com/pytorch/fairseq/tree/master/examples/translation.
I download the model `transformer.wmt16.en-de`, whose md5sum is (evaluated by the command line md5sum)
``68f2ea37f8eb5431f1c6b92c44c6a7bf  model.pt``

The data is preprocessed by 
```
BINHOME=`the place to store mosesdecoder and subword-nmt`
SCRIPTS=$BINHOME/mosesdecoder/scripts
TOKENIZER=$SCRIPTS/tokenizer/tokenizer.perl
NORM_PUNC=$SCRIPTS/tokenizer/normalize-punctuation.perl
REM_NON_PRINT_CHAR=$SCRIPTS/tokenizer/remove-non-printing-char.perl

cat $fname | \
perl $NORM_PUNC $lang | \
perl $REM_NON_PRINT_CHAR | \
perl $TOKENIZER  -l $lang | \
python $BINHOME/subword-nmt/subword_nmt/apply_bpe.py -c $BPE_CODE  > ${fname}.bpe
```
And the translation is obtained by 
```
cat wmt15.en.bpe | \
python3 $FAIRSEQ/interactive.py /tmp/wmt16.en-de.joined-dict.transformer \
--source-lang en --target-lang de \
--path /tmp/wmt16.en-de.joined-dict.transformer/model.pt \
--buffer-size 2000 --batch-size 128 --beam 5 --remove-bpe | \
grep ^H | cut -f3- > out
```
The multi-bleu/sacreBLEU for WMT’14 are 29.14/29.0, which are almost consistent with the results in (Ott et al., 2019) (i.e., 29.3 vs 28.6)
However, the multi-bleu/sacreBLEU for WMT’15 are 31.65/31.8, which are much higher than your baseline and algorithm. Therefore, I think you should re-implement your method following the settings in (Ott et al., 2019) to get more convincing results. 

review_title: An interesting idea but the results are not solid. 

Meta-review: This paper builds upon recent iterative refinement approaches NMT with an evaluator model that controls the termination of the translation process, yielding a “rewriter-evaluator framework” for multi-pass decoding. Their approach is an alternative to the policy network used in Geng et al (EMNLP 2018). The main delta wrt previous studies is that the evaluator offers this framework the capability of flexibly controlling the termination. While the idea behind the rewriter-evaluator framework is sensible and well described, and the proposed method achieves significant performance improvement against reported baselines, reviewers pointed out some concerns with the baselines and model optimization details. More analysis of the termination procedure against the RL-based model of Geng et al. 2018 could shed some light on why the proposed approach is better. Some analysis testifying how many iterations the model uses for translating one sentence, and what factors could affect the iteration number, such as sentence length, would greatly improve the paper. A second weakness pointed out by reviewers is related to the results of WMT’15 En-De reported in Table 1, where the reported baseline numbers seem to be weaker than expected. As pointed out by one the reviewers, pre-trained checkpoints on English->German (available at https://github.com/pytorch/fairseq/tree/master/examples/translation) exist which achieve much higher sacre BLEU than the reported baseline. I found the authors’ answer not very convincing regarding this point. Therefore, I recommend rejection. I suggest the authors, in future iterations of their work, address some of the issues pointed out by the reviewers and re-implement their method following the settings in (Ott et al., 2019) to get more convincing results. 
Decision: Reject