EM-Network: Learning Better Latent Variable for Sequence-to-Sequence Models
Authors: Ji Won Yoon
SungHwan Ahn
Hyeonseung Lee
Minchan Kim
SeokMin Kim
Nam Soo Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Connectionist temporal classification
Speech recognition
Machine translation

Abstract: In a sequence-to-sequence (seq2seq) framework, the use of an unobserved latent variable, such as latent alignment and representation, is important to address the mismatch problem between the source input and target output sequences. Existing seq2seq literature typically learns the latent space by only consuming the source input, which might produce a sub-optimal latent variable for predicting the target. Extending an expectation-maximization (EM)-like algorithm, we introduce EM-Network that can yield the promising latent variable by leveraging the target sequence as the model's additional training input. The target input is used as guidance to provide the target-side context and reduce the candidates of the latent variable. The proposed framework is trained in a new self-distillation setup, allowing the original sequence model to benefit from the latent variable of the EM-Network. Specifically, the EM-Network's prediction serves as a soft label for training the inner sequence model, which only takes the source as input. We theoretically show that our training objective can be a lower bound for the log-likelihood of the sequence model and is justified from the EM perspective. We conduct comprehensive experiments on two sequence learning tasks: speech recognition and machine translation. Experimental results demonstrate that the EM-Network significantly advances the current state-of-the-art self-supervised learning approaches. It improves over the best prior work on speech recognition and establishes state-of-the-art performance on WMT'14 and IWSLT'14 datasets. Moreover, the proposed method even achieves considerable performance improvement for fully supervised learning.

Under review as a conference paper at ICLR 2023
EM-NETWORK: LEARNING BETTER LATENT VARI- ABLE FOR SEQUENCE-TO-SEQUEE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In a sequence-to-sequene (seq2seq) framework, the use of an unobserved latent variable, such as latent alignment and representation, is important to address the mismatch problem between the source input and target output sequences. Existing seq2seq literature typically learns the latent space by only consuming the source input, which might produce sub-optimal latent variable for predicting the tar- get. this paper, we introduce EM-Network that can yield the promising latent variable by leveraging the target sequence the model's additional training in- put. The target input is used as guidance to provide the target-side context and reduce the candidates of the latent variable. The proposed framework is trained in a new self-distillation setup, allowing the original sequence model to benefit from the latent variable of the E.M-Netorr. Specifically, the EM-Network's prediction serves as a soft label for training the inner sequence model, which only takes the source input. We conduct comprehensive experiments two types of seq2seq models: connectionist temporal classification (CTC) for speech recognition and attention- encoder-decoder (AED) for machine translation. Experimental results demonstrate that the EM-Network significantly advances the current state- of-the-art approaches. improves over the best prior work on speech recognition and establishes state-of-the-ane performance on WMT' 14 and IWSLT' 14 datasets.
INTRODUCTION
Throughout the literature on deep learning, sequence-fo-sequence (seq2seq) learning has achieved great success a wide range of applications, especially speech and natural language processing. Given a source-target pair (r.y). task of the seq2seq learning is to learn a function for mapping a source sequence x to target sequence y, which generally suffers from source-target mismatch problems, e.g., unequal length, different domain, and modality mismatch. To deal with this issue, learning the latent variable z and how to improve its quality are deemed critically important in sequence modeling. For example, in automatic speech recognition (ASR). the connectionist tem- poral classification (CTC) (Graves et al., 2006) model defines the latent alignment 2 to learn the mapping between the speech feature and the word sequence y. shown in Figure 1 For the counterpart speech synthesis task, some studies (Kim et al.) 2020 2021) have proposed learning an internal alignment between the text æ and the corresponding speech y. In natural language processing (NLP), BERT family models (Devlin et al., [019 Liu et al. 2019 Lan et al. 2020) em- ploy masked language modeling (MLM) learn the contextualized representations 2, where the randomly masked tokens are predicted given the context of other tokens. From the perspective of 2, the sule-super learning (SSL)-based pre- pre-training enables the model to obtain the robust latent representation from the input x, which offers the desired performance for predicting the target on the downstream task.
However, it is difficult to learn the optimal latent variable for the learning task. For example, in the case of ASR, CTC models often converge to sub-optimal alignment distributions and produce over-confident predictions (Liu et al.) 2018 Yu et al. 2021 Miao et al.) 2015).. Since there is an exponential number of possible alignment paths, and the alignment information between source and target sequences is rarely available during training, settling the optimal alignment is quite chal- lenging. From the feature perspective, powerful representation important to achieve the desired performance. For the recent NLP studies, including machine translation (MT), large pre-trained

---

Under review as a conference paper at ICLR 2023
Goal: Learning a better Intent
EM-Netwock CTC
EM-Netwock
for AED
Source
Lanent Variable
Target
Prodattr
Sharal Pancen
HERT
Sparee
Concoradionl
Dovarmannn
Spond
Meckal
JCTC ASD
ICTC er ABDo
A: CTC Aligrement
Contestualiedd Represenaaiin
Tage
Sease
* mecie
Conventional
*3 GrADD
Sequence Model
EM-Network
Proposed Self-distillation
Figure I: (Left) Conventional sequence model converts the source x into the target y through the latent variable 2. (Middel) EM-Network can estimate the promising latent variable by using the additional training input y. For the AED model, the masked version of the target ý is used instead of the whole target y. (Right) EM-Network is trained with a self-distillation setup, where EM- Network's s predcition serves as soft labels for training the original sequence model.
LM is highly required to obtain the contextualized representation (Zhu et al. 2020 Wu et al. 2021 Xu et al. 202T). In this paper, we propose novel framework termed EM- NewNee that can effec- tively improve the quality of the latent variable z and thus the overall quality of the seq2seq model. In particular, EM-Network encapsulates two key components. First, the proposed framework lever- ages the target sequence as the model's additional training input, where the target input y is used as guidance to capture the target-side context and reduce the candidates of 2. Second, based on the us- age of the target input, we present a new self-distillation strategy as collaborative training, where the original sequence model can benefit from the E.M-Network's 's knowledge in a one-stage manner. The prediction of the EM-Network serves consumes only the source input. The proposed self-distillation acts as a sort of regularization for the seq2seq model, and its performance gain comes from a deep mutual learning scheme (Zhang al. 2018). where the students learn collaboratively and teach each other. However, the main difference from the previous mutual learning approaches (Zhang et al. 2018 Liang et al. 202T is that the pro- posed method utilizes the target input when trainig the EM- Network instead of merely considering the ground truth as the sole target in training. Since the target input is used as guidance to provide the target-side information, the prediction of the EM-Network (teacher mode) is more accurate than that of the inner sequence model (student mode). Therefore, the sequence model can effectively benefit from the soft labels of the EM-Network, which will be additionally discussed in Section 6 In addition, we attempt to apply the proposed self-distillation to the CTC framework, an unexplored area in mutual learning research.
soft label for training the inner sequence model, which
Modeling the conditional probability of the EM-Network (teacher mode) is determined by whether the latent variable is explicitly defined, shown in Figure [ The CTC computation adopts the alignment z, and it is difficult to settle on the optimal alignment with the conventional framework. The proposed EM-Network computes the posterior P(=|ar,) y) for the loss, which aims predicting a better CTC aligmment by leveraging the source and target inputs. Therefore, the CTC model distilled from the EM-Network does not have consider the exponential number of possible CTC alignments. We theoretically show that the proposed objective function can serves as the proposed Q-function and is justified from the EM-like algorithm perspective.
For the attention-bel braention-based encoder-decoder (AED), where there is no explicit latent alignment, it is challenging directly apply the same training scheme as the EM-Network for CTC. Simply taking the target y the additional input may cause an obvious but trivial solution, where the model converges with the conditional probability P(y Jz, y) = 5(y). Inspired by the MLM, we present an alternative that employs the masked version of target ÿ the additional input instead of using the whole target y. The EM-Network for AED computes the posterior P( z y) for loss and provides more robust contextualized representations that can benefit the learning task.
We conduct comprehensive experiments on multiple benchmarks, including ASR and MT tasks. The CTC and AED models considered for ASR and MT tasks, respectively. Experimental results demonstrate that the EM-Network improves over the best prior work on ASR and establishes SOTA performance on WMT' 14 and IWSLT" 14 datasets.
2

---

Under review as a conference paper at ICLR 2023
Seguence Model
Tareet lapat
Probublitty foc FM-Narwork
Leeu (p)
CTC
P(a
Incer Sutimass
Linear Sollmax
Key valte
AED
P(ylu,
Emsoatication
Shanef
Selfstierzio
Prametux
Sell-ateeiinn
LiM-NNrwork
Soquesee Model
Sequince
Embeiding
Auniliary Netwerk
Sequenee Model
Target Input
Source Irput
Source Input
Figure 2: Overview of the EM-Network. Note that the EM-Network includes the parameters of the sequence model. Green components ((), orange components (p), blue components (0) represent the parameters of the EM-Network, the auxiliary network, and the sequence model, respectively. When the sequence model based on the AED, teacher forcing applied the inner sequence model during the training.
METHODOL.OGY
In this section, we introduce how to design the EM-Network for learning better latent variable (Section 2.1 and Section 2.2). and its applications with CTC (Section 2.3 and AED (Section 2.4). We further provide a theoretical interpretation of the proposed framework from an EM-like algo- rithm perspective in Section
2.1 THE ARCHITECTURE OF EM-NETWORK
EM-Network. As shown in Figure 2 EM-Network mainly consists of two parts: (1) a sequence model that performs the original learning task given the source input x and (2) auxiliary network to learn a meaningful representation of both source v and target y. We add the auxiliary network on top of the conventional sequence model. For the convenience of notation, we let Φ, 0, and p denote the parameters of the EM-Network, the sequence model, and the auxiliary network, respectively. Note that the EM-Network includes the parameters of the sequence model, where 0 C ó.
Sequence Model. Since the sequence model in the EM-Network performs the same task as the conventional seq2seq models, we can flexibly select the architecture of the sequence model de- pending on the task demands. For example, in our ASR experiments, the conventional CTC model consisting of 12 transformer (Vaswani al.] (017) blocks, applied as the sequence model. For the MT task, the architecture of the sequence model follows that of the BiBERT' (Xu et al. 202T).
Auxiliary Network. The auxiliary network performs a fusion of the source and target y repre- sentations when generating the prediction, and its architecture is based on the self and cross-attention modules. Specifically, the cross-ates attention takes the representation of as the query and the repre- sentation of y as the key-value pairs.
2.2 TRAINING AND INFERENCE PROCEDURES
Self-distillation. EM-Network first yields the predictions, which serve as soft labels in distillation, by encoding both the source and target y inputs (teacher mode). Then, the sequence model learns the parameter to predict the soft labels given the source input x (student mode).
Training. In the proposed framework, there three kinds of losses during the training phase: (1) the original sequence learning loss (or, such as CTC loss and frame-wise cross entropy (CE), with the source input train the sequence model, (2) the proposed sequence learning loss Lem(d) given both source and target inputs for training the EM-Network, and (3) self deff-distillation loss Lxald, 8) that trains the sequence model predict the soft labels generated from the EM- Network. We optimize the following loss to train the proposed framework:
Lirain (6, 0) = Lorg (0) + Lem (Φ) + aL 0)
(1)
where a is a tunable parameter. In our self dittiiistilatiinn manner, the soft labels from the EM-Network updated in each iteration since ø is affected by the two losses Lam (Ø) and Lza( 0).
3

---

Under review as a conference paper at ICLR 2023
Inference. During the inference, only the sequence model is used to generate the prediction. Since the auxiliary network removed during the inference, the additional computational load of p is only required for the training procedure. In our SSL-based ASR experiments, about 10 M parameters are used as the auxiliary parameters to train the EM-Network, where the number of parameters about 93 M. For the fully-supervised ASR setting, the number of parameters of 0 is about 13 M, and only 1 M parameters are required for the auxiliary network.
is
As aforementioned in Section EM-Network can cover two popular loss functions for sequence learning: CTC and CE objective. We begin in the CTC objective setting (Secion 2.3 and then turn to the CE-based EM-Network (Section 2.4
2.3 EM-NETWORK FOR CTC
Standard CTC Objective. Before describing the proposed method in the CTC framework (Graves) al. 2006), it might be beneficial briefly discuss the CTC objective function. Given an input sequence z, the core idea of CTC is leveraging intermediate alignment 2 by allowing label repetitions possibly interleaved with a special blank token (c). CTC trains the sequence model 0 to minimize
the following loss function: Lorg (0) log p(y]2; 0) log Σ P))|2;0)
y=B(=)
where B is a many-to-one mapping in the CTC algorithm with y B(z). By) is the set of possible alignments z compatible to y. The mapping is done by merging repeated labels from the paths and then removing all blank tokens (f), e.g., B B((eaa.......)) {aab) }) where } denotes y and the sequence fc..a.,.c..bbb)) represents 2.
Modeling Conditional Probability with Target Input. As mentioned earlier, the crucial chal- lenge how to use the target input while preventing the trivial solution. Since the CTC algorithm explicitly adopts the latent alignment 2, we can easily apply the target as the EM-Network input. The relationship between and y is many-to-one, which can be formulated as B(z) = y. Intuitively, it is challenging to predict (many) given the target y (one). Therefore, the training objective func- tion with the target input can be given as
Lem-ete = - log
Σ
P(z|æ, 6) ((())
(3)
The target input is used as guidance to provide the target-side information and reduce the candidates of the latent variable.
Distillation Loss. EM-Network is trained in a self-distillation setup where the prediction given source and target inputs serves as a soft label for training the original sequence model. The distilla- tion loss in the proposed method is as follows:
Lid-cte = 2e - Eorgli
(4)
where Sem and Eorg are softmax outputs of the EM-Network and the sequence model, respectively.
Note that the argmax value of is the predicted CTC alignment. Since the conventional distil- lation loss using CE or Kullback-Leibler (KL)-divergence generally fails to converge in the CTC
framework, as reported in previous studies (Senior al. 2015 Takashima al., 2018 2019 Yoon
et al. 20211), we follow the distillation loss of Yoon et al. (2021b) that adopts l2 loss function for transferring the latent alignment.
2.4 EM-NETWORK FOR AED
Standard CE Objective. Different from the CTC, the AED model does not explicitly define the latent alignment 2 and the many-to-one mapping 13. The CE loss for AED can be formulated as
Lorg-ce (0) = log P(y|æ;0),
(5)
Since simply taking the target y as the AED model's 's input may cause obvious but trivial solution where the model converges with the conditional probability P(y|æ, y; 0) = 8(y), difficult to learn meaningful representation of the target input y-
4

---

Under review as a conference paper at ICLR 2023
Modeling Conditional Probability with Target Input. To address this issue, employ a mask- ing strategy in the MLM task. The masked version of the target ÿ is applied as the additional input instead of directly using the target y. Given the target sequence y, we use masking func- tion M. which randomly masks the tokens of with the probability of X. Then, the masked target My, X) = ÿ is fed to the EM-Network as the auxiliary input. The training objective function with the masked target y/ can be calculated as follows:
Cem-ce(@) (0) -lo P(yla, Ø) where MMy,
(6)
Distillation Loss. In the case of the CE objective function, we utilize the conventional distillation loss, where the KI. divergence is computed using softmax outputs between the student and the teacher. The distillation loss for AED-based EM-Network is given by
Lid-ce (0, 8) = Dx (Jem lldorg)
(7)
where Dem- Yorg denote the softmax outputs of the EM-Network and the sequence model. DRL represents the KL-divergence.
3 CONNECTION TO EM ALGORITHM
In this section, we explore the connection with the EM algorithm. The detailed derivations are provided in Appendix
Standard EM Algorithm. The traditional EM algorithm finds maximum likelihood parameters of the model that depends on unobserved latent variables z. iteratively performs an expectation (E) step and a Maximization step (M step). The E step defines Q-function ((11(()) as the expected value of the log-likelihood of 0, which can be formulated as follows:
Q(0)0(!)) 0222,000) (log P(x, 2;
Then, the M step computes the parameters that maximize the Q-function found on the E step.
EM-like Perspective for CTC- EM-Network The proposed framework is motivated by the EM algorithm. In the case of the EM-Network for CTC, the Q-function is calculated as follows:
100)0(9) ))) = Eel ple) [log 21,,;|a;00 0)]
DDx (P y; g(t) (()) 0))
(8)
R - Lxdd-oc
(9)
where P(z|æ, y; (l() (()) represents the conditional probability of the CTC-based EM-Network in Eq. (3) and log P=z|=; 0) denotes the CTC-based sequence model in Eq. 2 Here, we ignore the constant factor - H( (P(2llz, y, g(*), (())) in the fourth equality and the first approximation. As shown in Eq. (8). negative value of KL-divergence between the EM-Netwont and the sequence Section 2.3 CTC model often fails to converge with the distillation loss using the KI--divereence due to its alignment-free property (Senior et al., 2015] Takashima et al.) 2018,) 2019) Yoon et al.] 20215). To sidestep this convergence problem, we use le loss instead of the KL-divergence, corresponding to the distillation loss Ltd-ote- Eq. (9) shows that Lkd- can be considered as the approximation of the EM-Network's 's Q-function. Note that B(2) = y and
model serves as the O-function of the EM-Network. However, as aforementioned
+ p in the CTC-based EM-Network.
Based on the above perspective, we can actually show our proposed training objective in Eq. (4) is a lower bound for the ligeeiihldood of the original sequence model. Our goal is maximize the log-likeldhood below,
P(2|2;
log P(ylz;0)
Σ P(z|z, y: 80) ,1))) log P(z|z, y: ((t),p(t)) p(())
y=B(=)
- (Dxi(P(2lz, (:) (()))((()));
- --dd (b)
(10)
5

---

Under review as a conference paper at ICLR 2023
can confirm that negative value of Ckd-ete serves as the lower bound for the log- ligelkklihood of the original CTC-based sequence model log (xy 0). Therefore, both Eq. (9) and Eq. (10) indicate that maximizing the EM-Network's s Q-function is equivalent to maximizing the lower bound of the sequence model's likelihood. The tight lower bound will be additionally discussed in Appendix F
where the first inequality follows from Jensen's inequality. From Eq. 10
Maximizing the lower bound (minimizing Lid-ete) is also closely related to the upper bound for the sequence model's log- likelihoodd. The conventional KD generally assumes that the teacher's perfor- mance determines the upper bound of the student (Zhang et al. 2020 Mishra & Marr] 2018 Clark al., 2019). Under the assumption. the log-likeling likelihood of the EM-Network (teacher mode) can be regarded as the upper bound for the log-likelihood of the sequence model. Considering that the EM- Network uses the ground truth as the additional input to perform the learning task, this assumption seems more reasonable. Different from the previous offline KD) methods where the teacher model is generally fixed, we update the EM-Network and sequence model simultaneously. The training loss curves are given in Figure 4 The proposed distillation loss Lxd-ete regularizes the EM-Network (teacher mode) to learn the knowledge of the sequence model (student mode), providing the tight upper bound for the sequence model. For the counterpart, minimizing Lid-ete also corresponds to making the sequence model mimic the behavior of the EM-Network, indicating that the log- likelihood of the sequence model is close its upper bound. Therefore, we can derive the tight upper bound by maximizing the lower bound. On top of that, the upper bound can be maximized by minimizing the loss Lem-ete in Eq. B indicating that the proposed objective function partially maximizes the likelihood of the sequence model.
EM-like Perspective for AED-based EM-Network When considering the AED, the Q-function can be computed with similar perspective in Eq. (8) which is given by
@0))0)) (()) = By a. 0,0,000 ples [log P(ū, y (2; 0)]
DKz(P(ul ūn 0(t)) ( (l))) Lid on (0))) 0).
(II)
Then, we start from the log-likelihood of the AED model, which is formulated in Eq. 5
P(yla;; 0)
log P(yl=;0) 2
Σ
Plule, ū;
0(0) (0)))) P(ulæ, Di 001), p(t))
)=M(nAA
D-DK2(P(ull) ú: 0(1,
( (P)
-Lad (0(0)
(12)
Eq. (2) indicates that the KL-divergence between the EM- EM-Network (P (yyzz z. ÿ: phi)) and the AED model (P(y) [:) 0))) accords with the distillation loss in Eq. 7 By Eq. [1.. we know that negative value of the Kl.-divergence between the EM-Nettork and the sequence model serves as the Q- function of the EM-Network. Therefore, maximizing the AED-based EM-Network's 'ss Q-function is equivalent to maximizing the lower bound of the sequence model's likelihood. As mentioned above, the likeliiol of the EM-Network serves the tight upper bound for the sequence model.
4 EXPERIMENTAL SETUP
Speech Recognition. Our experiments were conducted using the fairseq (Ott et al. 2019) toolkit.
We utilized widely used LibriSpeech (Panayotov al.) 2013) dataset. In the proposed framework, we first pre-trained the sequence model (6) on the full 960 hours of LibriSpeech (LS-960) by using the objective of the data2vec since it is the current SOTA model in the SSL literature. Then, the whole EM--Nttork (Φ) was fine-tuned with LS-960, where the dev-other subset was used as the validation. The architecture of the sequence model followed the Base setup of the conventional SSL-based model, consisting of 12 transformer blocks. The tunable parameter was experimentally set to 2. When applying the language model (LM), we used the official 4-gram KenLM (Heafield. 2011 The detailed implementation is additionally presented in Appendix C
Machine Translation. We evaluated the EM-Network on IWSLT' 14 and WMT' 14 datasets for Anglish-to-Germanl (En-De) and Gergan-to-FEnglish (De-En) translation tasks. The implementation

---

Under review as a conference paper at ICLR 2023
Table I: (Speech recognition) Word error rate (%) on LibriSpeech test set. Bold represents supe- rior results. The results ILS-SSL and HuBERT were implemented using the public fairseq (Out et al. 2019) toolkit. For data2vec and wav2vec 2.0 models, we evaluated the performance using the checkpoints provided by the fairseq.
No LM
w/ 4-gram LM
Method
clean
other
clean
other
HuBERT Base
3.79
9.04
2.68
6.21
wav2vec 2.0 Base
3,40
8.42
2.59
6.19
ILS-SSL Base
3.44
7.79
2.65
5,84
data2vec Base
2.78
7.02
2.43
5.56
Ours, EM-Network
2.66
6.72
2.37
5.43
Table (Machine translation) Comparison our EM-Network and most recent existing methods on IWSLT' 14 and WMT' 14 datasets. Bold represents superior results. We reimplemented the results of BiBERT using the public code (Xu et al. 2021), which are shown inside the parentheses.
(a) BLEU score IWSLT' 14 test
(b) BLEU WMT 14 newstest2014 test
Method
En-De
De-En
Method
En-De
De-En
Adversarial MLE
35.18
Large Batch Training
29.3
DynamicCony
35.20
Evolved Transformer
29.8
Macaron Net
35.40
BERT Init. (12 layers)
30.6
33.6
BERT-Fuse
30.45
36.11
BERT-Fuse
30.75
MAT
36.22
31.26
34.94
BIBERT
Mixed Representations
29.93
36.41
(30.80)
(34.53)
UniDrop
29.99
36.88
Ours, EM-Network
31.30
35.40
30.45
38.61
BiBERT
(30.48)
(38.66)
Ours, EM-Network
31.80
39.49
of the EM-Network was based on official source code provided by previous work (Xu et al. 2021). Firstly, we pre-trained the sequence model with SSL, and then the whole EM-Network was fine-tuned on the MT task. In the case the pre- pre-trainii training, the sequence model was trained on the same English and German texts BiBERT (Xu ct al. 20 derived from the OSCAR (Suárez et al. 2020) corpus. The architecture sequence model was based on the BiBERT, recent SOTA method on the MT' task, and followed its pre-training scheme. During the fine-tuning, the parameter and the masking ratio À were set to and 50 %, respectively.
MAIN EMPIRICAL RESULTS
Since the current SOTA models were typically based the SSL, we used the pre-trained SSL model in our experiments. Due to the space limit, the experimental results with the fully super- vised learning baased models were described in Appendix (D) From the results, observed that the proposed method achieved considerable performance improvement for fully supervised learning.
Speech Recognition. We compared the EM-Network with previous SOTA works from the liter- ature, including wav2vec 2.0 (Baevski et al. 2020), HuBERT (Hsu et al. 2021), ILS-SSL (Wang et al. 2022), and data2vec (Baevski et al. 2022). As shown in Table L EM-Nerwond outperformed the recent approaches on the LibriSpeech. is important to note that our approach followed the pre- training scheme of the data2vec, and we applied the proposed scheme to the data2vec's 's fine-tuning stage. On the LibriSpeech, we improved upon the best data2vec by 0.12 % on the clean/other test datasets with greedy decoding, yielding relative error rate reduction (RERR) of 4.32 %/4.27 % compared to the data2vec baseline. Also, the proposed approach consistently achieved supe- rior WER results when applying beam-search decoding with the LMM. Specifically, compared to the beam- seamchearch decoding case, the performance improvement of the EM-Network was much more sig- nificant with the greedy decoding. Since the EM-Network considered linguistic (text) information by using the additional target input, the sequence model could produce strong performance even without the LMM

---

Under review as a conference paper at ICLR 2023
Model
Prediction
Data2vec
NOT ON SO ON BOARD THE JOHN WAS HART
Sequence Model (0) in EM-Nec Network
NOT A SOUL ON BOARD JOHN WAS HART
MM-Network (e) w/ Target Input
NOT A SOUL ON BOARD THE JOHN WAS HURT
Frame
Frame
Label
Label
OU
90
WAS
SOUL
WAS HART
Conventional Data2vee
Sequence Model (0) EM-Network
Figure 3: Total frame-wis softmax output examples in test-other dataset, where the target reference is "NOT A SOUL ON BOARD THE JOHN WAS HURT". The x-axis refers to acoustic frames, and the y-axis refers to the character labels. The first label index denotes the "blank" label.
Machine Translation. Table 2a shows a comparison of the proposed method with the recent ap-
proaches on IWSLT' 14 En- ne-De and De-En translations. We chose the eight best- pest-performed MT
algorithms date: Adversarial MLE (Wang al., 2019). 2) DynamicConv (Wu al., 2019) 3) Macaron Net [Lu et al.) 2020), 4) BERT-Fuse (Zhu et al. 2020). 5) multi-branch encoders (MAT) (Fan al. 2020p. 6) mixed representations from different tokenizers (Mixed Representation) (Wu al., 2020). 7) uniting different dropout techniques (UniDrop) (Wu et al. 2021), and 8) BiBERT (Xu et al. 2021). From the results, is verified that EM-Network outperformed all of them and yielded around 1.4 (En-De) and 0.9 (De-En) BLEU point gains over the previous SOTA results. Recall that EM-Network used the same settings as BiBERT - the main difference was applying the auxiliary network with the additional input ÿ and the proposed self-distillation framework during the fine-tuning stage - yet the EM-Network yielded a significant improvement over BiBERT.
For the high-resource scenario, conducted experiments WMT" 14 En-De and De-En transla- tions and compared the EM-Network with prior existing works that achieved high BLEU scores on WMT' 14 dataset, including large batch training (Out et al.) 2018), Evolved Transformer (So al.] 2019), initializing the BERT by leveraging pre-trained checkpoints (Rothe et al. 2020), BERT-Fuse (Zhu et al., 2020), and BiBERT (Xu et al. 202T). From Table 2b we verified that our model also gave the best BLEU scores on the high-resource WMT' 14 En-De and De-En translations.
6 ANALYSIS
CTC Alignment. As visualized in Figure contrasted the total frame-wise softmax outputs of the best baseline data2vec and our sequence model 0 in the EM-Nerwork. The argmax value of the frame-wist wise label probability corresponds the predicted CTC alignment. In Figure 3 the conven- tional data2vec converted given speech into "NOT ON SO ON BOARD THE JOHN WAS HART" and made erroneous predictions with "ON SO" and "HART". When considering only the acoustic feature (speech voice), it is challenging to distinguish "ON SO"~A SOUL" " and "HART" However, the proposed sequence model provided a more accurate prediction than the data2vec. Different from the previous seq2seq approaches, the EM-Network performed fusion of the acous- tic (speech source input) and linguistic (text target input) representations. Since the Fe-work could consider not only the acoustic information but also the linguistic one, the sequence model learned better alignment and obtained satisfactory performance. Even though there was a single erroneous word "HART", can discover the probability regarding "U" (green circle in Figure 3) in the prediction the sequence model. Also, unlike the data2vec that included many irrelevant label probabilities (red boxes in Figure implying that the use of the target input effectively reduced the candidates of the latent CTC align- ment z. From the results, it is verified that the proposed framework could learn the promising CTC alignment by using the target input and the self-distillation setup.
sequence model had relatively fewer redundant ones,
Translation Consistency. The token repetition ratio represents the degree of the inconsistency problem in MT (Song et al. 2021: Ghazvininejad al. 2020).. We computed the token repetition ratio by dividing all consecutively repeating tokens by the total number of tokens. The teacher forcing and the greedy decoding were applied to fairly compare the predictions. From the results
8

---

Under review as a conference paper at ICLR 2023
Table 3: Token repetition ratio on IWSLT" 14 test dataset.
Model
En-De
De-En
Reference Test Set
0.23
0.18
BiBERT
4.76
5.62
Sequence Model (0) in EM-Network
4.57
4.88
EM-Network (Ø) w/ Target Input
4.42
4.70
in Table 3 we can confirm that the EM-Network effectively addressed the token repetition issue compared the prior SOTA approach. In the case of the conventional BiBERT, it only considered the past target information during the training via the teacher forcing technique. However, the EM- Network leveraged a more global target context by using the target input y. Since our approach used both past and future target information in generating the prediction, it alleviated translation inconsistency and produced better quality. In other words, the sequence model in the proposed framework could benefit from more optimal and consistent representation of the EM-Network.
Training Loss Curves. Figure 4 shows the training loss curves of the EM-Network for the ASR task. We observed that the loss fee of the whole EM-Network (d) consistently decreased, implying that soft labels from the EM-Network were updated in each iteration. In this way, unlike the conventional distillation that the knowledge is generally fixed, the EM-Network trans- ferred the knowledge more adaptively. Compared to the previous aif-distillation methods (Pei et al.) [02TT]
Cerz
Loss
10030
2602
120000
312000
Training Steps
Kong et al. 2022. Baevski et al., 2022), the proposed distillation framework did not require the additional update strategy, such as a noving-average of param- eters. It could be implemented by simply applying the additional losses Lem and Lyd- Also, since fem was lower than Lory during the training. the prediction of the EM-Network (teacher mode) was more accurate than that of the inner sequence model (student mode). Thus, the sequence model could effectively benefit from the soft labels the EM-Network.
Figure 4:
Training loss curves of EM-
Network for ASR task.
7 RELATED WORKS
Training with Target Input. There are many techniques in the seq2seq learning literature that exploit the target-related information. Autoregressive approaches /Vaswani al. 2017 Chorowski jet al. 2015) Graves 2012) have adopted a teacher forcing technique that supplies a ground truth value conditional input during the training. The teacher forcing allows the model predict the current target by utilizing the context representation of the source input and the history of labels. To make good global planning, |Feng et al.. (2021) proposed the additional seer decoder into the encoder-decoder framework, which embodies future ground truth to guide the behaviors of the con- ventional decoder. In the case of the non autorrrreeeee framework, some studies (Higuchi et al.) 2020 Chan et al.] (020) attempted to improve the CTC model's prediction by conditioning on pre- viously generated tokens. Also. Yoon et al. (202 introduced the Oracle Teacher, which uses the target information to improve the teacher model's performance in the offline distillation setup.
8 CONCLUSIONS
In this paper, we introduced EM-Network that can effectively yield a promising latent variable from both source and target inputs. Instead of merely considering the ground truth as the sole target in training, the target was utilized as the model's additional training input. The proposed self- distillation framework enabled the sequence model to benefit from the EM-Network's 's latent variable in one-stage manner. We theoretically showed that our training objective, which was motivated by the EM-like algorithm, could be a lower bound for the log- likeliool of the sequence model. Empirically. EM-Network significantly advanced the current state-of oftte-of-thharrr the-art approaches ASR and MT tasks. We hope our study will draw more attention from the community toward a richer view of the latent variable for seq2seq learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
A. Baevski, H. Zhou, A. Mohamed, and A. Michael. Wav2vec 2.0: a framework for self-supervised learning speech representations. In Proc. NIPS, 2020.
A. Baevski, W. Hsu, Q. Xu, A. Babu, Gu, and M. Aulim. Data2vec: general framework for self-supervised learning in speech, vision and language. In Proc. ICML, pp. 1398-1312 2022.
W. Chan, C. Saharia, G. Hinton, M. Norouzi, and N. Jaitly. Imputer: sequence modelling via imputation and dynamic programming. Proc. ICML, 2020.
K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based models for speech recognition. In Proc. NIPS, pp. 577-585, 2015.
K. Clark et al. Bam! born-again multi-task networks for natural language understanding. In Proc. ACI, 2019.
Devlin, M. Chang, K. Lee, and K. Toutanova. Bert: pre-training deep bidirectional transformers for language understanding. In Proc. NAACL, pp. 4171-4186, 2019.
Y. Fan, S. Xie, Y. Xia, L. Wu, T. Qin, X. Li, and T. Liu. Multi-branch attentive transformer. arXiv preprint arXiv:2006. 10270, 2020.
Y. Feng et al. Guiding teacher forcing with seer forcing for neural machine translation. In Proc. ACL-IJCNLP, 2021.
M. Ghazvininejad et al. Aligned cross entropy for non-autoregressive machine translation. In Proc. ICML, 2020.
A. Graves. Sequence transduction with recurrent neural networks. In Proc. ICML Workshop on Representation Learning, 2012.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. ICML., pp. 369- 376, 2006.
A. Gulati et al. Conformer: convolution-augmentte transformer for speech recognition. In Proc. INTERSPEECH, 2020.
K. Heafield. Kenlm: faster and smaller language model queries. In Proc. EMNLP, 2011.
Y. Higuchi, S. Watanabe, N. Chen, T. Ogawa, and T. Kobayashi. Mask ctc: non-autoregressive end-to-end asr with ctc and mask predict. In Proc. INTERSPEECH, 2020.
W. Hsu, B. Bolte. Y. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3444 2021.
Kim, S. Kim, J. Kong, and Yoon. Glow-tts: a generative flow for text-to-speech via monotonic alignment search. In Proc. NIPS, 2020.
J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end- to-end text-to-speech. In Proc. ICML, 2021.
Y. Kim and Rush. Sequence-level knowledge distillation. In Proc. EMNLP, 2016.
D. Kingma and J. Ba. Adam: a method for stochastic optimization. Proc. ICLR, 2015.
X. Kong et al. En-compactness: self-distillation embedding contrastive generation for generalized zero-shot learning. In Proc. CVPR, 2022.
O. Kuchaiev et al. Nemo: toolkit for building ai applications using neural modules. arXiv preprint a909v: 2019.
G. Kurata and K. Audhkhasi. Guiding etc posterior spike timings for improved posterior fusion and knowledge distillation. In Proc. INTERSPEECH, 2019.
10

---

Under review as a conference paper at ICLR 2023
Z. Lan al. Albert: lite bert for self-supervised learning of language representations. In Proc. ICLR, 2020.
X. Liang et al. R R-drop: regularized dropout for neural networks. In Proc. NIPS, 2021.
H. Liu et al. Connectionist temporal classification with maximum entropy regularization. In Proc. NIPS, 2018.
Y. Liu al. Roberta: arXiv: 1907.119..199 2019.
robustly optimized bert pretraining approach. arXiv preprint
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proc. ICLR, 2019.
Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T. Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Y. Miao et al. Eesen: End-to-end speech recognition using deep rm models and wfst-based decod- ing. In Proc. ASRU, 2015.
A. Mishra and D. Marr. Apprentice: using knowledge distillation techniques to improve low- precision network accuracy. In Proc. ICLR, 2018.
M. Ott, S. Edunov, D. Grangier, and M. Auli. Scaling neural machine translation. In Proc. Machine Translation: Research Papers, pp. 1-9, 2018.
M. Ott et al. fairseq: fast, extensible toolkit for sequence modeling. In Proc. NAACL-HILT: Demonstrations, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In Proc. ICASSP, pp. 5206-5210, 2015.
Y. Pei et al. Self-boosting for feature distillation. In Proc. IJCAI, 2021.
S. Rothe, S. Narayan, and A. Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264-280, 2020.
A. Senior, H. Sak, F. C. Quitry, T. Sainath, K. Rao, et al. Acoustic modelling with ed-cte-smbr Istm mns. Proc. ASRU, pp. 604-609, 2015.
D. So, Q. Le, and C. Liang. The evolved transformer. In Proc. JCML., pp. 5877-5886, 2019.
Song, S. Kim, and S. Yoon. Alignart: non-autoregressive neural machine translation by jointly learning to estimate alignment and translate. In Proc. EMNLP, 2021.
P. Suárez al. A monolingual approach to contextualized word embeddings for mid-resource lan- guages. In Proc. ACL, pp. 1703-1714, 2020.
R. Takashima, S. Li, and H. Kawai. An investigation of knowledge distillation method for ctc acoustic models. Proc. ICASSP, pp. 5809-5813, 2018.
R. Takashima, S. Li, and H. Kawai. Investigation of sequence-level knowledge distillation methods for ctc acoustic models. In Proc. ICASSP, pp. 6156-6160, 2019.
A. Vaswani et al. Attention all you need. In Proc. NIPS, pp. 5998-60088 6008, 2017.
C. Wang, Y. Wu, S. Chen, S. Liu, J. Li, Y. Qian, and Z. Yang. Improving self-supervised learning for speech recognition with intermediate layer supervision. In Proc. ICASSP, 2022.
D. Wang. C. Gong, and Q. Liu. Improving neural language modeling via adversarial training. In Proc. ICML, pp. 6555-6565, 2019.
Wu, A. Fan, A. Baevski, Y. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. ICLR, 2019.
11

---

Under review as a conference paper at ICLR 2023
L. Wu, S. Xie, Y. Xia, Y. Fan, J. Lai, T. Qin, and T. Liu. Sequence generation with mixed represen- tations. Proc. ICML, pp. 10388-1880 10388, 2020.
Z. Wu, L. Wu, Q. Meng, Y. Xia, S. Xie, T. Qin, X. Dai, and T. Liu. Unidrop: simple yet effective technique improve transformer without extra cost. In Proc. NAACL, pp. 3865-3878, 2021.
H. Xu, B. Durme, and K. Murray. Bert, mbert, or bibert? study on contextualized embeddings for neural machine translation. In Proc. EMNLP, pp. 6663-6675, 2021.
J. W. Yoon, H. Y. Kim, H. Lee. S. Ahn, and N. S. Kim. Oracle teacher: towards better knowledge distillation. arXiv preprint a1Xiv:211..3664, 2021a.
W. Yoon, H. Lee, H. Y. Kim, W. Cho, and N. S. Kim. Tutornet: towards flexible knowledge distillation for end-to-end speech recognition. IEEF/ACM Transactions Audio, Speech, and Language Processing, 29: 1126-1638,, 2021b.
J. Yu et al. Fastemit: low-latency streaming asr with sequence-level emission regularization. In Proc. ICASSP, 2021.
Y. Zhang et al. Deep mutual learning. Proc. CVPR, 2018.
Z. Zhang et al. Distilling knowledge from well-informed soft labels for neural relation extraction. In Proc. AAAI, 2020.
Zhu, Y. Xia, L. Wu, He. T. Qin, W. Zhou, H. Li, and T. Liu. Incorporating bert into neural machine translation. In Proc. ICLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
A
SOURCE CODE
Source code can be found at http::/:w...titi.oc.//ee////////
git
B
THEORETICAL ANALYSIS
Derivation of Eq.
100100 ,()))
E -00s pe0 [log Ply: =|2;
E E2j,,,,000,,0 pro [log[6(y - B(2))
Σ P(=|æ y 000), p(0)) log (()]=;0)
B==()
DKL
(P(2,,, |)))||(())) 0))
Derivation of Eq.
10
log P(yl=; 0) log Σ P(y, 2(x;0)
P(y, 2|1; 0) P(z|æ, vs (00) ,000)
log Σ
P(z|a: ys 0(1), p()))
P(zaz, y: 0(1), p(0)) log Σ 8 - B(z) P(z|z, w: O1)) pl0)
P(z|x; 0) P(=|.. y: 0(0) p(0)
log
Σ
P(z|z.y:0)
v=B(=)
P(z|r;
log
Σ
P(2|x,y;00 0(0) (0)
P(=|.r y; 800) p(t)
B(z)
P(z|=;;) 0)
2 Σ
P(= y: g())
10)) P(z|z, y: ((1), p())
B(2)
(D ,)))), P(2[z;
-Ldd (c) ,0)
13

---

Under review as a conference paper at ICLR 2023
Derivation of Eq.
12
log P(yl2; 0) log
P(u. D1;;; 0)
P(yl:: ū; 010) plo)
log
P(u. x;; 0)
Plylx, ū: 000) p(o)
Plyla az ÿ: ((t), p(1)) Ply xz 3i 010), p(o))
5(g M(y (y) 8) log Σ
Plyla, ūr g(1), p(.))
log Σ
P(ul:; 0)
P is 001) p(())
f= M(y.2)
P(y z 0)
log
Σ
P(yla, 9: 000 (0))
P(ul ū: 0(0), ,00,,p()) p()))
(=M(y.))
P(y|=z;)) 0)
Σ i=M(yA)
Plula, ū; 8(), , log
P(ylæ,, ūū 000), p(0)
DDKi(P(()la in g(0), (
Cid-ce (0)))
C IMPLEMENTATION DETAILS
Table 4: The number parameters for each model.
Task
Method
of Param.
data2vec Base
93 M
ASR
EM-Nettork
103 M (0: 93 M, p: 10 M)
BiBERT
206 M
MT (IWSLT' 14)
EM-Network BiBERT
225 M (0: 206 M. p: 19 M)
324 M
MT (WMT) 14)
EM-Network
374 M (0: 324 M. p: 50 M)
Speech Recognition. For the pre-training stage, we followed the pre-training regime of data2vec (Baevski et al. 2022). After the pre-training, pre-trained models were fine-tuned for the ASR task by applying linear projection layer. We fine-tuned the model with Adam (Kingma & Ba 2015) optimizer with an initial learning rate of le-4. For the LibriSpeech, the character set had a total of 29 labels plus word boundary token. The BASE setup contained 12 transformer blocks with model dimension 768 and 8 attention heads. Also, BASE models utilized a batch size of 3.2m samples eight Quadro RTX 8000 GPUs (each with 48GB of memory),
per GPU and were fine-tuned where training updates were set to 320k. Following the recommendation of the language model settings (Bacvski al. 2020). test performance was measured with beam 1,500 for the 4-gram LM. In the case of EM-Network, the architecture of the auxiliary network was based on the self and cross-attention modules, and we additionally employed the feed-forward layers, layer normalization layers, and residual connections for the attention layer.
Machine Translation. For the IWSLT' 14 dataset, we used four Titan V GPUs (each with 12GB of memory) with 2048 tokens per GPU and accumulated the gradient 4 times. The optimizer was Adam (Kingma & Ba 2015), with learning rate of 0.0004. We followed the same byte-pair en- coding (BPE) settings of the previous study (Xu et al. 202I). At inference time, we applied beam search with width 4 and length penalty of 0.6. In the case of the WMT' 14 dataset, newstest2012 and newstest2013 were combined as the validation set, and we used newstest2014 as the test set. Followed by the BiBERT(Xu et al. 2021), we used unified 52K vocabulary for the decoder. We
14

---

Under review as a conference paper at ICLR 2023
utilized four Quadro RTX 8000 GPUs (each with 48GB of memory) with batch size of 4096 tokens per GPU. The gradient was accumulated 32 times. The initial learning rate was set to 0.001.
Number of Parameter, Table compares the number of parameters between the best seq2seq baseline and the EM-Network. We used the best baseline as the inner sequence model of the EM- Network, and thus the parameter for 0 corresponded to that of the best baseline, such as data2vec and BiBERT. The size of the auxiliary network was mainly determined by the vocabulary size. Since the EM-Network for WMT' 14 dataset used about 52K vocabulary for the decoder, it required the relatively large parameters for p. However, during the inference, only the sequence model was used to generate the prediction. Since the auxiliary network was removed during the inference, the additional computational load of p was only required for the training procedure. Considering the performance improvements of the proposed approach, this computational load for training seems reasonable.
D FURTHER EXTENSION: EXPERIMENTAL RESULTS WITH FULLY SUPERVISED LEARNING-BASER MODEL
D.1 SETUP
For the fully supervised learning-base bbasid models, ASR models were implemented in the NeMo (Kuchaiev et al. 2019) toolkit. We trained the EM-Network with LS-960, and the sequence model (0) was based on a conformer-CTC architecture consisting of 16 conformer (Gulati et al. 2020) blocks with 176 dimensions. We used four Quadro RTX 8000 GPUs (each with 48GB of memory), and 100 epochs were spent for training the models. Adam W algorithm (Loshchilov & Hutter 2019) was employed an optimizer with an initial learning rate of 5.0. In the case of Conformer- CTC large, the current SOTA ASR model, we used the pre-trained checkpoint provided by the NeMo (Kuchaiev et 2019 toolkit. For MT task, we evalutated the models with IWSLT' datasets for En-De and Dn translations. The sequence model (0) was based Transformer (Vaswani ctt al. 2017). We used four Titan GPUs (each with 12GB memory), and 75 epochs were spent for training. BiBERT (Xu et al., 202T), which is the current SOTA MT model, was adopted as the teacher model for conducting the conventional distillation.
D.2 ASR RESULTS
Baseline Model
# of Params. 122 M 13 M
clean
other
Conformer-C Large
2.78
6.18
Conformer CTC Small
4.87
12.05
Additional # of
Method
Student
KD
clean
other
Params. for Teacher
Baseline
None
None
4.87
12.05
Guided CTC training
122 M
Conformer-CTC Small
Offline
4.63
11.56
SKD
122 M
Offline
4.53
11.33
Ours, EM-Network
M
Self
4.29
10.81
Table 5: Comparison of word error rate (%) on LibriSpeech test dataset.
In addition to the previous experiments, we checked whether the EM-Network could improve the performance of the sequence models with fully supervised learning. Table shows the results on the LibriSpeech test set. We applied Guided CTC training (Kurata & Audhkhasi 2019) and SKD (Yoon) al. 2021b) as the competing KD methods for ASR. The Conformer- CTC Large was adopted as the teacher for Guided CTC training and SKD. The Guided CTC training is the effective KD method for CTC-based ASR model. The student can guided to align with the frame-level alignment of the teacher by using the guided mask. The SKD is the recent KD method in the ASR task, and we used its distillation loss for transferring the soft labels of the EM-Network (Eq. 4b). The main difference between the SKD and the EM-Network self-distillation setup with the additional target input, while the conventional SKD was based on
that the EM-Network
trained with the
offline knowledge distillation. Even though the teacher's knowledge in Guided CTC training and
15

---

Under review as a conference paper at ICLR 2023
SKD fixed during the student model training, the soft label of the EM-Network was updated in each iteration so that our approach could transfer the knowledge more adaptively. As shown in Table [ both Guided CTC training and SKD required the additional teacher model (122 M pa- rameters) to perform the distillation. However, only M parameters, which corresponded to the auxiliary network in the E.M-etwork, were required to distill the knowledge in the proposed frame- work. From the results, it is verified that the EM-Network performed well with the fully-supervised learning-based model. Compared to the conventional KD methods using the SOTA ASR model as the teacher, EM-Network achieved better performance on the LibriSpeech test dataset. We con- firmed that the proposed method can significantly improve the performance of the sequence model with the fully supervised learning-based model, yielding 4.29 % (RERR: 11.91 %) and 10.81 % (RERR: 10.29 %) on test-clean and test-other, respectively.
D.3
MT RESULTS
Baseline Model
# of Params. 206 M 120 M
En-De 30.48 27.87
De-En
BiBERT
38.66
Transformer
34.18
Additional # of
Method
Student
KD
En-De
De-En
Params. for Teacher
Bascline
None
None
27.87
34.18
Sequence-level KD
206 M
Offline
29.33
35.97
Transformer
R-Drop
0 M
Self
29.43
35.99
Ours, EM-Network
19 M
Self
29.84
36.47
Table 6: Comparison of BLEU score on IWSLT' 14 test set.
For MT task, we compared the EM-Network with other previous methods, including sequence-level KD (Kim & Rush 2016) and R-Drop (Liang et al. 202T), as shown in Table 6 The sequence- level KD is widely- uueedd KD method in sequence generation tasks, and the R- R-Drop is recent and effective regularization method built upon dropout, As the teacher model for the sequence-level KD, we adopted BiBERT Xu et al.) (202T), which is SOTA approach for MT task. The results in Table report that the proposed EM-Network yielded the best BLEU performance on IWSLT" 14 dataset, yielding 29.80 99 and 36.47 on En-De and De-En translation tasks, respectively.
The main difference from the R-Drop was the training loss Lem of the EM-Network. While the R-Drop only consumed the source input, the proposed method utilized the target input when train- ing the EM-Network so that the prediction of the EM-Network (teacher mode) was more accurate than that of the inner sequence model (student mode). From the results, we can verify that the tar- get input was effectively used as guidance to provide the target-side information while producing considerable performance gain. Compared to the R-Drop that did not require the additional param- eters, 19 M parameters were used as the auxiliary network p train the EM-Network. However, since the auxiliary network was removed during the inference, the additional computational load of p was only required for the training procedure. Considering the performance improvements, this additional load during the training seemed reasonable. Also, when we reduced the vocabulary size, the additional parameters would be much smaller.
E ADDITIONAL VISUALIZATION
We additionally compared the total frame-wiss softmax outputs of the best baseline data2vec, our sequence model 9 (student mode) in the EM-Network, and the EM-Network é (teacher mode), as visualized in Figure The argmax value of the frame-wise label probability corresponds the pre- dicted CTC alignment. The conventional data2vec made erroneous predictions with "ON SO" and "HART". When considering only the acoustic feature (speech voice), it challenging to distinguish "ON SO7/'A SOUL" and "HURT"/"URT". However, the EM-Network gave the correct predic- tion result with few irrelevant aligmments, indicating that the proposed method effectively leveraged the target information to find the optimal CTC alignment. By benefiting from the knowledge of the EM-Network, the proposed sequence model provided a more accurate prediction than the data2vec while it gave "A SOUL." instead of "ON SO".
16

---

Under review as a conference paper at ICLR 2023
Model
Prediction
Data2vec
NOT ON SO ON BOARD THE JOHN WAS HART
Sequence Model (0) EM-Network
NOT A SOUL ON BOARD THE JOHN WAS HART
EM Network (c)) w/ Target Input
NOT SOUL ON BOARD THE JOHN WAS HURT
Frame
blonk label
HART
Label
Prodiction: NOT ON ON BOARD THE JOHN WAS HART
Frame
blonk
SULL
MAA HART
label
Label
Prediction: NOT A SOUL ON BOARD THE JOHN WAS HART
Frame
Mlnnk
label
Label
Prediction: NOT SOUL ON BOARD THE JOHN WAS HURT
Figure 5: Frame-wise label probability examples for ufterance 1009-169440-0200 in LibriSpeech test-other dataset, where the target reference is "NOT A SOUL ON BOARD THE JOHN WAS HURT". Note that its argmax value corresponds to the predicted CTC alignment. The x-axis refers to acoustic frames, and the y-axis refers to the character labels. The first label index represents the "blank" label the CTC framework.
By Eq. 10 confirmed that minimizing Eq. maximizes lower bound of the CTC model's log- likelihood. The lower bound is tight when achieving a low distillation loss between EM-Network and the sequence model, meaning a relatively large overlap between the predictions. Due to the alignment-free property of the CTC framework, the conventional CTC models trained with the same settings, such as model architecture, training data, etc., can have different frame-level alignments, and this difference often leads to the convergence issue of the distillation (Yoon et al. 2021b). However, as shown in Figure 5 both EM-Network (teacher mode) and the sequence model (stu- dent mode) could produce similar frame-wise predictions (green boxes in Figure 5), indicating that the proposed method was indeed able to perform the frame-wise distillation effectively and give a considerable alignment overlap, making tight lower bound.
F EFFECT OF AND A
When training the CE-based EM-Network, we considered two tunable parameters or in Eq. ① and À in Eq. (6) The parameter was used to balance the distillation loss Led, and À was applied the masking probability in the masking function M. We explored the effect a and X on EM-Network evaluated the EM-Network
performance, as shown in Figure 6 Firstly. translation while varying a. a was selected from {0, 1, 2, 3, 4), and À was set to 50 %. We observed that IWSLT' 14 En-De, the performance was stably better than the prior SOTA approach, and the
IWSLT' 14 En-De
17

---

Under review as a conference paper at ICLR 2023
31:8
31.8
31.5
31.49
31.56
31.48
31.8
31.6
31.68
31.54
31.4
314
31.2
31.2
BLEU
31
31
30.8
30.50
30.8
30.5
SOTO (BIBERT)
30.6
Prior SOTA (BIBERT)
30.4
30.4
30.2
30.2
0.5 0.7
Q
Figure 6: IWSLT 14 En-De performance with varying a and A.
best performance was obtained at a 2. Interestingly, the EM-Network with 0 was slightly better than the performance of the BiBERT, but the difference was negligible. Secondly, we showed the effect of À on EM-Network performance. In this case, a was set to 2, and we selected from { (10%, 30%, 50%, 70%): ). From the results, it is verified that performance on IWLST" 14 En-De achieved the best result when À 50%.
18


Reviews:
[{"review_id": "TCbiGi39dJr", "reviewer": "Reviewer_ngMM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Tasks: speech recognition, translation\n\nModels: CTC for speech, attention-based encoder-decoder (AED) for translation\n\nThe paper proposes a new training method to improve the quality of the latent variable and thus the overall quality of the model. Or in the case of AED, it acts as a sort of regularization.\n\nThe method is inspired from expectation-maximization (EM). Conventional EM computes p(z|x,y) given the model p_seq2seq(z|x). In the proposed method, the quantity p(z|x,y) is not calculated from the original model, but there is a separate model for it, p_separate(z|x,y). Then it uses distillation (KL or mean-squared-error) to train p_seq2seq based on this. This separate model for p(z|x,y) shares the same encoder (param sharing) but then has some additional cross-attention to y on top.\n\nEverything is jointly trained with the loss: L_org + L_em + alpha * L_kd\n\nL_org is the original loss: CTC loss or framewise CE for AED.\n\nL_kd is the distillation loss (KL or mean-squared-error). Mean-squared-error is taken when L_org is CTC.\n\nL_em is for training the separate posterior model for p(z|x,y). It's defined as - log p(y|x,y) = - log sum_{z:y} p(z|x,y).\n\nFor the AED model, where there is no latent variable, this is changed further to use a masked training loss instead for L_em.\n", "summary_of_the_review": "The method by itself is interesting and probably could be a useful building block for better regularization.\n\nI think many aspects need more clarification, and maybe even the whole mathematical formulation and derivation should be improved, or reformulated.\n\nFurther, I think the baselines are badly chosen, and distract from studying the actual method.\n\nThe analysis could also be extended.", "strength_and_weaknesses": "Strengths:\n\n- Interesting idea to improve the latent variable, or in general for regularization.\n- There is the intent to release the source code.\n\nWeaknesses:\n\n- Some important aspects are unclear. See below.\n- The whole mathematical formulation has several issues. I think this already starts with framing it as EM-like. I cannot really pinpoint this to one specific aspect, but there are many things which are not quite right, and this also shows that in the end it needs many changes deviating from the original motivation, e.g. using mean-squared-error for the distillation instead of KL, and then also the AED model does not fit into the picture at all, and also requires another masked-based loss for the separate posterior model. This does not mean that the proposed method by itself is bad, but just the mathematical derivation or motivation seems wrong.\n- It uses pretrained models and brings in the aspect of SSL, although this is totally irrelevant for the proposed method, and actually distracts from it. This also means it cannot properly be compared to other results from the literature. It rather should follow the original exact task, and only use the right training data, and not use SSL.\n- It would be interesting to further study the behavior of the latent variable, specifically the alignments. There is some figure and some discussion on it but this could be extended much more.\n- The source code is just a dump of Fairseq, probably including their modifications. But there is no explanation of what code parts are actually changed, and also is not properly forked from the Fairseq repo, so it's really difficult to see the actual differences and changes to Fairseq.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ptadhk42Auq", "reviewer": "Reviewer_AXAV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on learning a better latent variable for sequence-to-sequence models. To achieve this goal, the authors design an EM-Network, which first equips an auxiliary network with the sequence model to learn a meaningful representation based on both source and target input, and then distils its knowledge to improve the prediction of the sequence model. This paper connects EM-Network and CTC/CE and provides a theoretical relationship with the standard EM algorithm. Experimental results on ASR/MT tasks show the effectiveness of the proposed method.\n", "summary_of_the_review": "This paper investigates learning a better latent variable for sequence-to-sequence models and achieves new state-of-the-art translation performance on the WMT14 English-German benchmark. However, the proposed method is very similar to deep mutual learning, but correspondent comparisons are missing. Experimental results are not convincing enough. In addition, the motivation for optimizing ELBO is unclear to me and there are no detailed discussions with previous latent variable models.\n\n-----------\nThanks for the authors' response and clarification. I appreciate these experiments in Appendix D. However, the improvements over KD/R-drop methods seem marginal. It is better to add significant tests. I tend to keep my score.", "strength_and_weaknesses": "Strengths:\n- This paper designs a novel EM-Network to improve the performance of original sequence-to-sequence models.\n- The proposed method achieves new state-of-the-art translation performance on the WMT14 English-German benchmark.\n\nWeaknesses:\n- The proposed method is very similar to deep mutual learning. Some strong and similar baselines are missing, such as R-drop, and knowledge distillation from a larger model. \n- Actually, the theoretical analysis of the proposed method is the formula derivation of ELBO, which implicitly optimizes the conditional probability of sequence-to-sequence models. Compared with directly optimizing the conditional probabilities of sequence-to-sequence models, the performance gain in this way is unclear to me. I double that the performance gain comes from deep mutual learning.\n- The latent variable in this paper is very tricky, i.e., the prediction of the sequence model. It is not sure whether the definition of such latent variables is suitable. There are no detailed discussions with previous latent variable models.\n\n\nDetailed Comments:\n\nThis paper investigates learning a better latent variable for sequence-to-sequence models and achieves new state-of-the-art translation performance on the WMT14 English-German benchmark. Overall, the presentation of this paper should be refined. It is hard for me to understand the motivation of the proposed method for improving sequence learning. \n\nMy main concern is the experiment settings. I believe that the proposed knowledge distillation brings performance improvements, but performance gain comes from deep mutual learning. Thus, some strong and similar baselines should be considered, such as R-drop, and knowledge distillation from a larger model, and current experimental results are not convincing enough. In addition, the theoretical analysis in section 3 is the formula derivation of ELBO, which implicitly optimizes the conditional probability of sequence-to-sequence models. Compared with directly optimizing the conditional probabilities of sequence-to-sequence models, the motivation for optimizing ELBO is unclear to me. In my view, the designed latent variable is very tricky, i.e., the prediction of the sequence model. It is not sure whether the definition of such latent variables is suitable. There are no detailed discussions with previous latent variable models.\n\nQuestions for the Author(s):\n- Please explain the motivation for optimizing ELBO. \n- Have you tried other KD methods (e.g., distilling large model) or deep mutual learning (e.g., R-drop or bidirectional agreement on sequence learning)?\n- Is it necessary to share parameters between sequence and EM-Network models?\n- Have you tried other latent variables beyond the prediction of the sequence model?\n\nMissing References:\n- Previous papers on latent variable models, such as latent variable models for NMT.\n- Deep mutual learning and R-drop.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HRkLLHQyaC", "reviewer": "Reviewer_57uE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method for seq2seq tasks which incorporates the target sequence itself to learn a latent variable that informs the target prediction. This is done by using an EM-network that conditions on the target, with cross-attention to the regular seq2seq encoder representations, yielding the latent representations. EM-network is trained to predict the source-target alignment using a CTC loss for speech recognition. For machine translation, the masked target is given as input to the EM-network and task is to reconstruct the actual target. The latent representation informed soft predictions are then distilled back into the seq2seq model. While the speech recognition setting seems more novel, the MT setting seems very similar to data2vec.\n", "summary_of_the_review": "I find the approach interesting overall, though it feels a small improvement over data2vec. I am recommending a weak accept.", "strength_and_weaknesses": "Strengths\n- An interesting approach to incorporate the target sequence to inform the latent variable for seq2seq tasks.\n- Empirical results show improvement in state-of-the-art on a speech recognition dataset and two machine translation datasets.\n\nWeaknesses\n- Since the EM network uses additional parameters over the other seq2seq baselines, it is not clear if the improvements are due to more parameters or learning a better latent variable. It will be helpful to clarify the number of parameters in the EM-network and the seq2seq model, and contrast it with the number of parameters in the baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject