Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method for seq2seq tasks which incorporates the target sequence itself to learn a latent variable that informs the target prediction. This is done by using an EM-network that conditions on the target, with cross-attention to the regular seq2seq encoder representations, yielding the latent representations. EM-network is trained to predict the source-target alignment using a CTC loss for speech recognition. For machine translation, the masked target is given as input to the EM-network and task is to reconstruct the actual target. The latent representation informed soft predictions are then distilled back into the seq2seq model. While the speech recognition setting seems more novel, the MT setting seems very similar to data2vec.

strength_and_weaknesses: Strengths
- An interesting approach to incorporate the target sequence to inform the latent variable for seq2seq tasks.
- Empirical results show improvement in state-of-the-art on a speech recognition dataset and two machine translation datasets.

Weaknesses
- Since the EM network uses additional parameters over the other seq2seq baselines, it is not clear if the improvements are due to more parameters or learning a better latent variable. It will be helpful to clarify the number of parameters in the EM-network and the seq2seq model, and contrast it with the number of parameters in the baselines.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear and easy to follow. It seems like an extension of data2vec that is at least novel for speech recognition tasks.
The source is code is provided, it should help with reproducibility.
summary_of_the_review: I find the approach interesting overall, though it feels a small improvement over data2vec. I am recommending a weak accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Tasks: speech recognition, translation

Models: CTC for speech, attention-based encoder-decoder (AED) for translation

The paper proposes a new training method to improve the quality of the latent variable and thus the overall quality of the model. Or in the case of AED, it acts as a sort of regularization.

The method is inspired from expectation-maximization (EM). Conventional EM computes p(z|x,y) given the model p_seq2seq(z|x). In the proposed method, the quantity p(z|x,y) is not calculated from the original model, but there is a separate model for it, p_separate(z|x,y). Then it uses distillation (KL or mean-squared-error) to train p_seq2seq based on this. This separate model for p(z|x,y) shares the same encoder (param sharing) but then has some additional cross-attention to y on top.

Everything is jointly trained with the loss: L_org + L_em + alpha * L_kd

L_org is the original loss: CTC loss or framewise CE for AED.

L_kd is the distillation loss (KL or mean-squared-error). Mean-squared-error is taken when L_org is CTC.

L_em is for training the separate posterior model for p(z|x,y). It's defined as - log p(y|x,y) = - log sum_{z:y} p(z|x,y).

For the AED model, where there is no latent variable, this is changed further to use a masked training loss instead for L_em.

strength_and_weaknesses: Strengths:

- Interesting idea to improve the latent variable, or in general for regularization.
- There is the intent to release the source code.

Weaknesses:

- Some important aspects are unclear. See below.
- The whole mathematical formulation has several issues. I think this already starts with framing it as EM-like. I cannot really pinpoint this to one specific aspect, but there are many things which are not quite right, and this also shows that in the end it needs many changes deviating from the original motivation, e.g. using mean-squared-error for the distillation instead of KL, and then also the AED model does not fit into the picture at all, and also requires another masked-based loss for the separate posterior model. This does not mean that the proposed method by itself is bad, but just the mathematical derivation or motivation seems wrong.
- It uses pretrained models and brings in the aspect of SSL, although this is totally irrelevant for the proposed method, and actually distracts from it. This also means it cannot properly be compared to other results from the literature. It rather should follow the original exact task, and only use the right training data, and not use SSL.
- It would be interesting to further study the behavior of the latent variable, specifically the alignments. There is some figure and some discussion on it but this could be extended much more.
- The source code is just a dump of Fairseq, probably including their modifications. But there is no explanation of what code parts are actually changed, and also is not properly forked from the Fairseq repo, so it's really difficult to see the actual differences and changes to Fairseq.

clarity,_quality,_novelty_and_reproducibility: Saying that the latent variable in conventional models does not depend on the target y was a bit confusing to me. Because when you actually compute the CTC loss, you actually compute the posterior p(z|x,y) for the loss. When carrying out the forward-backward computation for the error signal, this is actually exactly this posterior which you compute there.

So this is just a new way to compute this posterior. Instead of computing it exactly, it is now approximated by a separate model.
But the paper does not really explain this well and is rather explaining it in a somewhat obscure and confusing way.

For example: "leveraging the target sequence as the model’s additional training input". This is wrong. In training of CTC or so, surely you leverage the target sequence. You do that when computing p(z|x,y) for the CTC loss.

Or: "their shortcoming is that the latent variable captures only the source-side information". I don't really understand this. Surely it must be this way.

Further, I was all the time confused about how this actually applies to the AED model, and only somewhere in the middle realized that for the AED model, it does not really apply at all, but just uses a similar motivated loss and training method including the separate model.
What is actually meant by "conventional sequence model" or "seq2seq learning"? This should really be clarified further. CTC is give as one example. So is it only about CTC? Or what about the other common models like hybrid NN-HMM, Transducer, attention-based encoder-decoder (AED), or segmental models? Later through reading, I think it's about CTC for speech, and AED for translation, so only those two model types, and nothing else. But this should be totally clear from the beginning on in the paper, probably even in the abstract.

What is meant by CE-based objective function? CTC is a CE-based objective function as well. So this statement or categorization does not make sense to me. This is also not really defined anywhere, so this is a critical flaw. From other parts of the paper, I assume that "CE-based objective" is a very strange way of referring to AED based models?

It seems that the CE-based EM-network does not really fit well into the whole mathematical motivation of the paper.
Distillation loss, eq 4: Are these the latent variables? The latent variables are discrete, so the equation does not make sense then. Or are these actually the log probs for the latent vars? Or what exactly are those \hat{z}?

I think it distracts from the actual work to use SSL pretrained models. The SSL part is in no way relevant for the work, and potentially hides some aspects.

"Latent CTC Alignment. As visualized in Figure 3, we contrasted the latent alignments" - what does this mean? The latent variables are discrete. The figure does not seem to show any discrete values. So what exactly do we see in the figure?

summary_of_the_review: The method by itself is interesting and probably could be a useful building block for better regularization.

I think many aspects need more clarification, and maybe even the whole mathematical formulation and derivation should be improved, or reformulated.

Further, I think the baselines are badly chosen, and distract from studying the actual method.

The analysis could also be extended.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on learning a better latent variable for sequence-to-sequence models. To achieve this goal, the authors design an EM-Network, which first equips an auxiliary network with the sequence model to learn a meaningful representation based on both source and target input, and then distils its knowledge to improve the prediction of the sequence model. This paper connects EM-Network and CTC/CE and provides a theoretical relationship with the standard EM algorithm. Experimental results on ASR/MT tasks show the effectiveness of the proposed method.

strength_and_weaknesses: Strengths:
- This paper designs a novel EM-Network to improve the performance of original sequence-to-sequence models.
- The proposed method achieves new state-of-the-art translation performance on the WMT14 English-German benchmark.

Weaknesses:
- The proposed method is very similar to deep mutual learning. Some strong and similar baselines are missing, such as R-drop, and knowledge distillation from a larger model. 
- Actually, the theoretical analysis of the proposed method is the formula derivation of ELBO, which implicitly optimizes the conditional probability of sequence-to-sequence models. Compared with directly optimizing the conditional probabilities of sequence-to-sequence models, the performance gain in this way is unclear to me. I double that the performance gain comes from deep mutual learning.
- The latent variable in this paper is very tricky, i.e., the prediction of the sequence model. It is not sure whether the definition of such latent variables is suitable. There are no detailed discussions with previous latent variable models.


Detailed Comments:

This paper investigates learning a better latent variable for sequence-to-sequence models and achieves new state-of-the-art translation performance on the WMT14 English-German benchmark. Overall, the presentation of this paper should be refined. It is hard for me to understand the motivation of the proposed method for improving sequence learning. 

My main concern is the experiment settings. I believe that the proposed knowledge distillation brings performance improvements, but performance gain comes from deep mutual learning. Thus, some strong and similar baselines should be considered, such as R-drop, and knowledge distillation from a larger model, and current experimental results are not convincing enough. In addition, the theoretical analysis in section 3 is the formula derivation of ELBO, which implicitly optimizes the conditional probability of sequence-to-sequence models. Compared with directly optimizing the conditional probabilities of sequence-to-sequence models, the motivation for optimizing ELBO is unclear to me. In my view, the designed latent variable is very tricky, i.e., the prediction of the sequence model. It is not sure whether the definition of such latent variables is suitable. There are no detailed discussions with previous latent variable models.

Questions for the Author(s):
- Please explain the motivation for optimizing ELBO. 
- Have you tried other KD methods (e.g., distilling large model) or deep mutual learning (e.g., R-drop or bidirectional agreement on sequence learning)?
- Is it necessary to share parameters between sequence and EM-Network models?
- Have you tried other latent variables beyond the prediction of the sequence model?

Missing References:
- Previous papers on latent variable models, such as latent variable models for NMT.
- Deep mutual learning and R-drop.

clarity,_quality,_novelty_and_reproducibility: Please refer to Detailed Comments.
summary_of_the_review: This paper investigates learning a better latent variable for sequence-to-sequence models and achieves new state-of-the-art translation performance on the WMT14 English-German benchmark. However, the proposed method is very similar to deep mutual learning, but correspondent comparisons are missing. Experimental results are not convincing enough. In addition, the motivation for optimizing ELBO is unclear to me and there are no detailed discussions with previous latent variable models.

-----------
Thanks for the authors' response and clarification. I appreciate these experiments in Appendix D. However, the improvements over KD/R-drop methods seem marginal. It is better to add significant tests. I tend to keep my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper describes a distillation framework for sequence-to-sequence models that is intended to improve the learned latent representation. The idea is to provide the teacher model with the target label sequence as an _input_ which is accessed via a cross-attention mechanism. Many model parameters are shared between the teacher and student models, and depending on the application, some care may be needed to ensure that the teacher cannot find a trivial solution when given access to the target label sequence. Experiments on speech recognition (Librispeech) and WMT'14 and IWSLT'14 En-De and De-En (machine translation) tasks are presented to show the advantages of the proposed method.

# Strengths
- The basic idea of the paper, trying to improve sequence-to-sequence models via distillation from a model that has access to the ground truth as an input, is clever and well motivated.
- The machine translation results are reasonably strong, though one reviewer would have like bootstrap-based significance tests to verify that they are solid.
- The comparison of training L_org to L_em for the ASR task (Figure 4) makes a convincing case for the distillation idea.
    
# Weaknesses
- The reviewers and AC did not find the attempt to frame the proposed algorithm in terms of the EM algorithm to be particularly convincing or helpful. Portions of the derivations were confusing, some of the notation was sloppy (though it was improved over the course of the discussion), and there were enough differences in the details for the ASR and MT tasks, such as the need to use MSE loss instead of KL divergence in the CTC model and the need to condition the teacher on masked label sequences instead of the correct sequences, that the overall analogy to EM didn't really hold together.
- The consensus among the reviewers and AC is that the proposed algorithm is better presented as a form of distillation or regularization. The AC suggests that one way to think about this algorithm is as a form of distillation with privileged information (Lopez-Paz et al., "Unifying distillation and privileged information," in Proc. ICLR, 2016, https://arxiv.org/pdf/1511.03643.pdf) in which the privileged information used by the teacher is the target label sequence.
- It would be useful to try to quantify the degree to which this regularization leads to better alignments. For the speech recognition experiments, some of the analyses of alignments performed in Raissi et al., "HMM vs. CTC for automatic speech recognition: Comparison based on full-sum training from scratch", 2022 (https://arxiv.org/pdf/2210.09951.pdf) might be a useful tool.
- The ASR experiments without a self-supervised model using extra data would be more convincing if the results were closer to state-of-the-art.

justification_for_why_not_higher_score: This paper enjoyed a lengthy and thorough discussion between the authors and reviewers, and the AC thanks the reviewers for the extra effort they put into this paper.

The primary reason not to score this paper higher is that the reviewers and AC do not find the analogy between the proposed distillation framework and the EM algorithm to be convincing or helpful, and they recommend that if this work is submitted to another venue, the authors remove that discussion and instead discuss the method as a form of distillation, possibly with access to privileged information. A secondary reason for not accepting the paper is that the empirical results on the Librispeech task should be stronger.
Decision: Reject