Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper presents a strategy to integrate prior word alignments into NMT models. It is not clear the motivation for this in the NMT context, especially why the prior alignments are crucial information that is necessary to be given a-priori to the Transformer. Besides this, the description of the method and the discussion of related work is given, SMT methods are briefly mentioned but the usage of the idea in previous work, also SMT literature is necessary.
The applicability of the method is discussed very generally for NMT, however most languages do not have a one-to-one mapping in words and even in extremely low-resource cases it is difficult to see how the alignment information can be useful except for translating for instance foreign words and named entities (in languages where transliteration is not necessary).
The description of the method seems sound and the experiments are performed for two languages with improvements in BLEU scores. Have the authors analyzed what is exactly being improved in the translations?
The paper is mostly clear but has too many grammatical errors and can benefit from revisions and more editing.

review_title: An interesting idea but needs more progress

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes to incorporate word alignment information as a word substitution model. Basic idea is to jointly train a separate encoder using a cross entropy loss which predicts a source input sequence with words substituted by aligned target words. The learned representation is combined with a Transformer either by simple summation, gating or joint attention mechanism. Experimental results on Romanian/English and Korean/English tasks show very marginal gains over the baseline Transformer.

# Pros

* It is a very interesting work on combining word alignment information in a model by predicting aligned target words by following alignment links.

* Small gains are observed on Romanian/English and Korean/English.

* The attention weight visualization especially for word substituted encoder presents sharper distribution, which might indicate that useful information might be learned in the attention mechanism.

# Cons

* The gains are very small and thus, I suspect that the difference might be negligible when running statistical significance tests.

* Analysis is a bit weak in that it does not compare  the quality of alignment against the translation qualities.

# Details

This work presents an interesting idea, though, I have a couple of concerns to this submission. Thus, I have some hesitation for full acceptance.

* I'd like to see any tradeoffs of word alignment qualities and translation qualities by randomly distorting word alignment links. This work simply tweak sampling rate to vary the ratio of using word alignment links, not distorting links, and thus, it is not a direct measure to verify the quality tradeoff.

  * Thank you very much for adding the comparison of GIZA and fastalign. However, simply swapping a model for alilgnment does not given us details about the tradeoff of alignment quality and the end-to-end results, since alignment models have different characteristic to capture the correspondence in two langauges, e.g., assuming linearlity and/or fertility. I'd rather like to see a much simpler approach of distroting alignment to avoid influence of the models employed for word alignment.

* Given that non-aligned words are also predicted in the word substituted model in Equation 8, I suspect the model is simply learning to copy from input to output for non-aligned words. I think the loss in Equation 8 should be applied only to those words substituted by alignment links.
  
  * Thanks for the explanation. I'd like to see ablation studies regarding the loss.

* Please verify the experimental results by statistical significance tests.

  * Thank you for adding the tests. 

review_title: Interesting idea of incorporating word alignment in a model

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to integrate word alignment obtained from SMT into an NMT system. This is an exciting topic not only because it can help interpretability, but also because the same mechanism could be used e.g. for imposing a specific terminology in translations, something that was relatively easy to do with SMT but is much harder to achieve with NMT. The proposed method involves computing word-word alignment using existing SMT models (GIZA and FastAlign in the experiments) and integrating that information in the decoder of a transformer-based NMT model. Experiments on English-Romanian and English Korean show small improvement over a standard baseline.

Modeling is described at a high level and would require significant guesswork to re-implement. The core of the method is in the word substitution model (Sec 3.2) and how it is integrated in the decoder (Sec 3.3). Unfortunately neither is described with much detail. For example, there are a dozen operators involved in describing the model in Eqs. 1-17, some of them defined, some of them relatively easily guessable, other entirely unclear.
It would greatly help to clearly define the input and outputs of the model (e.g. vectors of context, matrices of probabilities or activation, appropriate dimensions, etc.)

Regarding evaluation, it is a positive to have results on two language pairs, in both directions. In the English-Romanian pair, GIZA seems to produce slightly better results, although the difference is unlikely to be significant. Still, it is odd that only  FastAlign was used for Korean. Although gains seem fairly consistent, they are also very limited and unlikely to be significant (no significance test seems to have been performed). Also the variability of performance (e.g. due to sampling) is not assessed. Finally, the only comparison is to a straight transformer baseline. None of the methods mentioned in the prior art have been tested.

The analysis of the impact of the alignment on the attention weights in Sec 5.2 is interesting. What is the motivation for using different sentences and epoch #, rather than show how alignment is modeled as training progresses (as claimed in the text) on the same sentence?

Misc:
[Sec 5.1]: why is the sampling rate suddenly called the « compression ratio »?

** UPDATE**
Read the author answers, thanks. There is a significant amount of new material added in response to the reviews, including some interesting new findings (e.g. Tab. 3, 4). Unfortunately that did not help the clarity issue. I did not see a new example in Section 3.2.
Basically, the stronger points of the paper (the results) are stronger, but the weaknesses (clarity, motivation) are not really addressed. The new version uploaded by the authors is an improvement but I still lean towards rejection.
review_title: Interesting problem but the modeling is confusing

Meta-review: Two reviewers suggested to reject and the other reviewer also thought it below the threshold.
Decision: Reject