Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper broadens the understanding of pseudo-label training (PLT) in neural machine translation (NMT). It finds that PLT can not only improve model performance but also enhance model stability. 
strength_and_weaknesses: Strength:   
Overall, PLT is a simple and effective method for improving the model performance of NMT. The understanding of PLT is an important topic. This paper gives some insights that might be useful.

Weaknesses:
1. The main weakness is that the analysis of the model inertia just focuses on the evaluation of model results but ignores the analysis of the pseudo-labels themselves. Which kind of characteristics of the pseudo-labels affect model inertia? For example, previous works have shown that NMT models are more likely to make frequent predictions, monotonic translations, and so on. In other words, why simplifying the training data can bring such benefits? The core questions are not well answered.
2. After carefully reading the paper, I still cannot find how can the paper findings guild future research. Could we design better PLT methods based on the findings, and how? Or could we better train NMT models with PLT?
3. I strongly suggest the authors include a comparison of non-autoregressive models, which might attract more attention from the community.
4. In 4.1, COMET or BERTScore could be a better choice for evaluating the robustness or consistency.

---
Thank you for the response that has alleviated my part concerns. I would like to increase my score a bit. However, similar to Reviewers ypio and zsxu, how can the paper's findings guild future research is still unclear which limits me from giving a higher rating.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
The paper is mostly clear. It can be improved if it can give a more detailed formulation of each setting in Section 5 since this part has many repetitive and similar settings (e.g., PLT (XX), PLT-delta(XX)). 

**Quality**
This is an OK paper and indeed provides some insights for the line of research. However, it can be a better paper if the authors can further investigate the core factors that lead to model inertia.

**Novelty**
Both PLT and the metrics to analyze model inertia are not novel. But combining these has some novelty.

**Reproducibility**
The experimental settings are clear for reproducing.
summary_of_the_review: This is an interesting paper that investigates the relationship between PLT and model inertia. However, it does not provide enough insights to guide future research. Therefore, I recommend rejection at this moment.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Pseudo-labels are widely used in practical NMT, either in a classical (sequence-level) KD scenario or for forward translation. The paper demonstrates that both forms of pseudo-labels improve the sanity of NMT by showing that models are both more robust against input perturbations and less prone to variations between training runs due to random initialization. Experiments on 6 language pairs using existing robustness measures and new model stability metrics as well as human evals support their claims convincingly.
strength_and_weaknesses: The paper makes a pretty straightforward point: Pseudo-labels improve the robustness of NMT inference and the stability of NMT training. This point may not be overly surprising to MT practioners. But that doesn't mean that it is a useful contribution to demonstrate this as thoroughly and clearly as this paper.

I don't see any major flaws, but here are some minor points:
- Where applicable, significance tests in Sec. 5.1 would be good.
- I'd encourage the authors to add some citations on faithfulness / robustness in NMT. Things like https://arxiv.org/pdf/2005.12398.pdf also come to mind, as well as the renewed popularity of the lottery ticket hypothesis.
- "NMT lacks a reliable automatic segment-level quality metric" There is evidence that neural MT metrics (BLEURT, COMET, ...) are more reliable on the sentence level than BLEU.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written, formulas look clean and consistent, although using an indicator function in Eq. 1 seems odd (but not wrong) to me. Experimental settings are described sufficiently, and the reported numbers are effective to support the claims. Baseline BLEU scores in Table 2 seem to be in the right ballpark.
summary_of_the_review: A solid and sober paper, but not the most exciting one.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors do research on pseudo-label training that is also known as knowledge distillation or teacher-student training, which transfers task knowledge learned from a larger model to a smaller model by letting the smaller model mimic the teacher's outputs rather the gold labels during the training. They study "model inertia effects" of the pseudo-label training and discuss how to address the robustness training in NMT models. The NMT models are trained on six different language directions and assessed in a variety of automation metrics to capture robustness and consistency, in addition to the n-gram match accuracy like BLEU scores.
strength_and_weaknesses: Strengths
- Extensive experimental results to assess the NMT models and pseudo-label training in different scores
- These analytic report would be useful to understand the impact of pseudo-label training

Weaknesses
- Lots of useful observation from the extensive experiments, however, there is few proposed approaches. Or, theoretical proof would be appreciated. 
clarity,_quality,_novelty_and_reproducibility: The experiments are carefully designed, e.g. they employed the test set with the source-original ones, which have the results be more reliable. 
When discussing the model stability or robustness, the authors mainly report automation evaluation metrics and I was wondering how the training curve or convergency has changed in the pseudo-label training. Do you have the loss curve plots for each model and see any trends? 
The paper looks a good analytic report but does not come up with a novel approach or proposal. It is okay in this format, though I would like to learn how to further improve robustness in NMT models based on these findings.
summary_of_the_review: The paper provides substantial experimental results and discuss inertia effect in the different MT task settings. The paper came up with a lots of observation and suggestions in a carefully designed experiments, however, it does not end up with any novel approaches or concrete conclusions for further improvement. We probably would like to have more theoretical proof or explanations behind the observations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper essentially reframes NMT models’ stability to model updates and input perturbations as *model inertia* and conduct various analysis experiments on it in the  pseudo-label training setups, aka forward-translation distillation. The paper shows that quality gains in different methods are due to the use of pseudo-labeled data in training.
strength_and_weaknesses: Strength

* The paper seems quite clear and easy to understand.
* The experiment results are decent

Weakness

* Lack various comparisons with related methods that try to do the same, such as robustness training against adversarial attack.
* The paper doesn't conclude or provide any suggestions or implication from the results so that it is hard to find the key takeaways from the paper, and it's difficult to imagine what we can do about the knowledge from the paper. For example, the paper should show experiments that key learning from analysis of model inertia helps us develop better model that perform better. I do not see any of such indication in the paper. And thus, I find it difficult to find this useful.
* Missing citations of some pseudo-label training forward translation methods, such as data diversification (Nguyen et al., 2020, NeuIPS)

clarity,_quality,_novelty_and_reproducibility: 
* The quality is not on par, as I am expecting better explanation on the implication and relevant suggestions to improve NMT models from these analysis
* Novelty is fine.
* Reproducibility: there is no code available.
summary_of_the_review: Overall, I think the paper is written with clear presentation. However, I think the analysis of model inertia is largely expected, and does not provide key learnings as there is no suggestions or conclusions on how to improve NMT model with this knowledge.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the impact of training on pseudo labels (PL) in neural machine translation (NMT). In addition to improving model accuracy, pseudo labels are shown to also enhance training stability and robustness to input perturbations.

Strengths 
- extensive experiments, clear analysis of PL training in NMT.

Weaknesses:
- the paper does not delineate the implications of its conclusions and how future research could benefit from it.
- the paper relies on BLEU and BLEURT for evaluation but does not evaluate if PL trained models produce different translation, e.g. simpler, more mechanical translation, see ixEF review.

justification_for_why_not_higher_score: - the paper is clear and shows a good experimental investigation with a strong analysis of the results.
justification_for_why_not_lower_score: - the paper could communicate recommendations for future research and would benefit from analyzing if PL-training induces a change in the style of translations.
Decision: Accept: poster