Venue: NAACL 2021

# Multi-Task Learning with Shared Encoder for Non-Autoregressive

Machine Translation

Yongchang Hao\({}^{\dagger}\)   Shilin He\({}^{\ddagger*}\)  Wenxiang Jiao\({}^{\ddagger}\)

Zhaopeng Tu\({}^{\dagger\dagger}\)   Michael R. Lyu\({}^{\ddagger}\)   Xing Wang\({}^{\dagger\dagger}\)

\({}^{\dagger}\)School of Computer Science and Technology, Soochow University

\({}^{\ddagger}\)Department of Computer Science and Engineering, The Chinese University of Hong Kong

\({}^{\dagger\dagger}\)Tencent AI Lab

\({}^{\dagger}\)yongchanghao.w@gmai.com

\({}^{\ddagger}\){slhe,wkjiao,lyu}@cse.cuhk.edu.hk

\({}^{\dagger\dagger}\){zptu,brightxwang}@tencent.com

###### Abstract

Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common practice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt multi-task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experimental results on WMT14 English\(\Leftrightarrow\)German and WMT16 English\(\Leftrightarrow\)Romanian datasets show that the proposed multi-task NAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 English\(\Leftrightarrow\)German datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our multi-task NAT is complementary to knowledge distillation, the standard knowledge transfer method for NAT. 1

Footnote 1: Code is publicly available at [https://github.com/yonchinghao/multi-task-nat](https://github.com/yonchinghao/multi-task-nat)

_This work was conducted when Yongchang Hao, Shilin He, and Wenxiang Jiao were interning at Tencent AI Lab._

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote †: leftmargin=*]

+
Footnote†: footnotetext: [leftmargin=*]


et al., 2018; Raganato and Tiedemann, 2018) for AT and NAT models. Further, by leveraging the linguistic differences, we then adopt a multi-task learning framework with a shared encoder (i.e., MultiTaskNAT) to transfer the AT model knowledge into the NAT model. Specifically, we employ an additional AT task as the auxiliary task of which the encoder parameters are shared with the NAT task while parameters of the decoder are exclusive. Since many works Cipolla et al. (2018); Liu et al. (2019) suggest that the weights for each task are critical to the multi-task learning, in this work, the multi-task weight assigned to the AT task is dynamically annealed from \(1\) to \(0\). We name this scheme _importance annealing_. We empirically show the benefit of _importance annealing_ in both directions of the original WMT14 English\(\Leftrightarrow\)German dataset.

Further with knowledge distillation, our proposed Multi-TaskNAT achieves significant improvements on WMT14 English\(\Leftrightarrow\)German and WMT16 English\(\Leftrightarrow\)Romanian datasets. This confirms the effectiveness of our proposed model on machine translation tasks.

Our contributions are as follows:

* We propose a multi-task learning framework to boost NAT translation quality by transferring the AT knowledge to the NAT model.
* Our analyses reveal that the encoder sharing is necessary for capturing more linguistic and semantic information.
* Experiments on standard benchmark datasets demonstrate the effectiveness of the proposed Multi-TaskNAT.

## 2 Why Shared Encoder?

To verify our hypothesis that AT and NAT encoders capture different linguistic properties of source sentences and can thereby complement each other, we probe the linguistic knowledge Conneau et al. (2018) that embedded in the AT and NAT encoders on a set of tasks to investigate to what extent an encoder captures the linguistic properties. We present the detail for each probing tasks in Appendix B. Moreover, in Appendix C, we also provide a qualitative investigation to capture the difference between high-dimensional representations of AT and NAT encoders from another perspective.

The AT and NAT models referred to in the following experiments are Transformer and Mask-Predict. We train the models on the WMT14 English\(\Rightarrow\)German dataset, and the details of the experiments are introduced in the Appendix.

Probing TasksProbing tasks Conneau et al. (2018) can quantitatively measure the linguistic knowledge embedded in the model representation. We follow Wang et al. (2019a) to set model configurations. The experimental results are depicted in Table 1.

Table 1 shows the AT and NAT encoders capture different linguistic properties of source sentences. We observe that on average, the NAT model captures more surface features but less semantic features than the AT model. For example, on the sentence length prediction (SeLen) task, NAT models significantly outperform AT models since the sentence length prediction is a key component in NAT models. However, for sentence modification (SoMo) and coordinate clauses invertion (CoIn) tasks, the AT model outperforms the NAT model by a large margin. The linguistic probing results reveal that AT and NAT models capture different linguistic properties, which thereby leaves space for the encoder sharing structure.

## 3 Approach

In this section, we introduce that our shared encoder structure between AT and NAT models under the multi-task learning framework.

Multi-Task NATGiven the AT and NAT models under the standard encoder-decoder structure, we employ the hard parameter sharing method Ruder (2017) to share their encoder parameters.

Therefore, as shown in Figure 1, the proposed model Multi-TaskNAT consists of three com

\begin{table}
\begin{tabular}{l l c c} \hline \hline  & **Task** & **AT** & **NAT** \\ \hline \multirow{2}{*}{_Surface_} & SeLen & 91.7 & **93.4** \\  & WC & 76.0 & **79.1** \\ \hline \multirow{3}{*}{_Syntactic_} & TrDep & 45.8 & 46.0 \\  & ToCo & 78.3 & **79.7** \\  & BShif & **74.8** & 73.4 \\ \hline \multirow{3}{*}{_Semantic_} & Tense & 89.2 & 89.2 \\  & SubN & 86.2 & **87.5** \\ \cline{1-1}  & ObjN & 85.2 & 85.3 \\ \cline{1-1}  & SoMo & **54.0** & 53.0 \\ \cline{1-1}  & CoIn & **64.9** & 62.8 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance on the probing tasks of evaluating linguistic properties embedded in the learned representations of AT and NAT models.

 

ponents: shared encoder, AT decoder, and NAT decoder. Their parameters are jointly optimized towards minimizing the multi-task loss function, as introduced in the next section.

Multi-Task FrameworkThe loss function of the proposed Multi-Task NAT \({\mathcal{L}}\) at iteration step \(t\) is defined as the weighted sum of AT loss and NAT loss:

\[\begin{split}{\mathcal{L}=&\lambda_{t}{\mathcal{ L}}_{\mathrm{nat}}\left(X,Y;\theta_{\mathrm{enc}},\theta^{\mathrm{nat}}_{ \mathrm{dec}}\right)\\ &+(1-\lambda_{t}){\mathcal{L}}_{\mathrm{at}}\left(X,Y;\theta_{ \mathrm{enc}},\theta^{\mathrm{at}}_{\mathrm{dec}}\right)\end{split} \tag{1}\]

where \({\mathcal{L}}_{\mathrm{at}}\) and \({\mathcal{L}}_{\mathrm{nat}}\) are AT loss and NAT loss. \(\theta_{\mathrm{enc}},\theta^{\mathrm{nat}}_{\mathrm{dec}}\), and \(\theta^{\mathrm{at}}_{\mathrm{dec}}\) are parameters of the shared encoder, NAT decoder, and AT decoder respectively. \(\lambda_{t}\) is the importance factor to balance the preference between the AT and NAT models at time step \(t\) as illustrated bellow.

Importance AnnealingThe term \({\mathcal{L}}_{\mathrm{at}}\) only serves as an auxiliary and does not directly affect the inference of NAT. Therefore, we intuitively determine to lower the importance of the AT loss when the training process is close to the ending, which we named _importance annealing_. Formally, we set

\[\lambda_{t}=\frac{t}{T}\]

where \(T\) is the total steps of training. Under such a scheme, the weight for \({\mathcal{L}}_{\mathrm{at}}\) is linearly annealed from \(1.0\) to \(0.0\) along the training process, while the weight for \({\mathcal{L}}_{\mathrm{nat}}\) is increased from \(0.0\) to \(1.0\).

Training and InferenceDuring the model training with training pairs \((X,Y)\), we feed the source sentence \(X\) to the encoder and the target sentence \(Y\) to two decoders separately. The target sentence \(Y\) can be either the target sentence in the raw training data (4.1) or the generated target sentence with knowledge distillation (4.2). During the model inference, we only use the NAT decoder to generate the target tokens simultaneously while ignoring the AT decoder. Therefore, the inference overhead is the same as the NAT model before sharing.

## 4 Experiment

We conducted experiments on two widely used WMT14 English\(\Leftrightarrow\)German and WMT16 English\(\Leftrightarrow\)Romanian benchmark datasets, which consist of 4.5M and 610K sentence pairs, respectively. We applied BPE Sennrich et al. (2016) with 32K merge operations for both language pairs. The experimental results are evaluated in case-sensitive BLEU score Papineni et al. (2002).

We use Transformer Vaswani et al. (2017) as our baseline autoregressive translation model and the Mask-Predict Ghazvininejad et al. (2019) as our baseline non-autoregressive model. We integrate the Transformer decoder into the Mask-Predict to implement the proposed Multi-Task NAT model. For \(\lambda_{t}\), we use the annealing scheme described in Section 3. Since the major NAT architecture of our method is exactly the Mask-Predict model, any established decoding latency results Kasai et al. (2021) for Mask-Predict can also be applied to ours. All of the parameters are randomly initialized for a fair comparison with the Mask-Predict. More training details are introduced in Appendix A.

### Ablation Study

Table 2 shows that the performance of our Multi-Task NAT model and baseline models on WMT14

Figure 1: The architecture of our proposed model. We introduce an extra AT decoder to boost the training at the beginning, and gradually lower the importance weight of AT task by increasing \(\lambda\).

 

En\(\Leftrightarrow\)De datasets without using the knowledge distillation. The vanilla Multi-Task NAT model with the the \(\lambda\) fixed as \(0.5\) outperforms the baseline Mask-Predict model by 0.96 and 0.57 BLEU score in En\(\Rightarrow\)De and De\(\Rightarrow\)En direction respectively and even surpasses the strong baseline Transformer-Lev by 0.46 BLEU points in En\(\Rightarrow\)De translation. With the _importance annealing_, the Multi-Task NAT model achieves slight but consistent improvements over the vanilla model ("+_Importance Annealing_" in Table 2). The improvements demonstrate the effectiveness of our proposed model using multi-task learning.

### Main Result

We further evaluate the proposed Multi-Task NAT model with the standard practice of knowledge distillation. Table 3 depicts the performances of our model as well as strong baseline models. Our proposed Multi-Task NAT model achieves a significant improvement of 0.80 and 0.41 BLEU point over the strong baseline Mask-Predict model on En\(\Rightarrow\)De and De\(\Rightarrow\)En translation. On En\(\Leftrightarrow\)Ro translation, our model outperforms the baseline model by 0.77 and 0.89 BLEU scores respectively. We use the compare-mt Neubig et al. (2019)2 to determine the significance. Details for significance tests are described in Appendix A.4.

Footnote 2: [https://github.com/neulab/compare-mt](https://github.com/neulab/compare-mt)

### Analysis

We conduct probing tasks to empirically reconfirm our hypothesis in Section 2 and better understand our Multi-Task NAT in terms of linguistic properties. The results are presented in Table 4. In most of the cases, our Multi-Task NAT could learn better surface, syntactic, and semantic information than the Transformer and Mask-Predict baseline models, indicating that our multi-task learning framework can indeed take the advantages of two separate tasks and capture better linguistic properties. Notably, on the sen

\begin{table}
\begin{tabular}{c c r r r} \hline \hline  & **Task** & **Ours** & \(\Delta_{AT}\) & \(\Delta_{NAT}\) \\ \hline \multirow{2}{*}{_Surface_} & SeLen & 94.4 & 2.7 & 1.0 \\  & WC & 79.3 & 3.3 & 0.2 \\ \hline \multirow{3}{*}{_Syntactic_} & TrDep & 47.2 & 1.4 & 1.2 \\  & ToCo & 79.3 & 1.0 & -0.4 \\  & BShif & 74.7 & -0.1 & 1.3 \\ \hline \multirow{3}{*}{_Semantic_} & Tense & 88.9 & -0.3 & -0.3 \\  & SubN & 87.1 & 0.9 & -0.4 \\ \cline{1-1}  & ObjN & 85.8 & 0.6 & 0.5 \\ \cline{1-1}  & SoMo & 54.8 & 0.8 & 1.8 \\ \cline{1-1}  & CoIn & 63.3 & -1.6 & 0.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on the probing tasks of our Multi-task NAT model. \(\Delta_{AT}\) and \(\Delta_{NAT}\) denote the relative increase over the AT and NAT probing performance, respectively.

 

tence length (Selen) prediction task and tree depth (TrDep) task, the Multi-task NAT shows significantly better performance. On other tasks, our model demonstrates better or on-par performance compared to the NAT model. Regarding the coordination inversion (CoIn) task, though the Multi-Task NAT shows certainly lower performance than the Transformer, it still outperforms the Mask-Predict by \(0.5\).

### Large-scale Experiments

We conduct the larger-scale experiments on the WMT English\(\Leftrightarrow\)German. We adopt newstest2019 and newstest2020 as the test sets. The parallel data consists of about \(36.8\)M sentence pairs. We average the last 5 checkpoints as the final model. The results are listed in Table 5. The improvements suggest that our model are consistently effective on various scale of data.

## 5 Conclusion and Future Work

In this paper, we have presented a novel multi-task learning approach for NAT model with a hard parameter sharing mechanism. Experimental results confirm the significant effect of the proposed Multi-Task NAT model, which shows the complementary effects of multi-task learning to the knowledge distillation method.

Based on our Multi-Task NAT, there are many promising directions for future research. For example, 1) decoder interaction: knowledge distillation in an online fashion between AT and NAT decoders; 2) share-all framework: shared-encoder and shared-decoder with two decoding strategies, and the model can dynamically choose the optimal decoding strategy during model inference. 3) data manipulation strategies: such as data rejuvenation Jiao et al. (2020), lexical frequency discrepancy Ding et al. (2021).

## Acknowledgements

The authors sincerely thank Liang Ding for the advice of experiments settings, and the anonymous reviewers for their insightful suggestions on various aspects of this work.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. Cited by: SS4.
* C. Chen and D. Kiela (2018)SetNeval: an evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449. Cited by: SS4.
* A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni (2018)What you can cram into a single vector: probing sentence embeddings for linguistic properties. In ACL, Cited by: SS4.
* L. Ding, L. Wang, Q. Liu, D. F. Wong, D. Tao, and Z. Tu (2021)Understanding and improving lexical choice in non-autoregressive translation. In ICLR, Cited by: SS4.
* L. Ding, L. Wang, D. Wu, D. Tao, and Z. Tu (2020)Context-aware cross-attention for non-autoregressive translation. In COLING, Cited by: SS4.
* M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer (2019)Mask-predict: parallel decoding of conditional masked language models. In EMNLP-IJCNLP, Cited by: SS4.
* J. Gu, S. Bradbury, C. Xiong, V. O. Li, and R. Socher (2018)Non-autoregressive neural machine translation. In ICLR, Cited by: SS4.
* J. Guo, X. Tan, L. Xu, T. Qin, E. Chen, and T. Liu (2020)Fine-tuning by curriculum learning for non-autoregressive neural machine translation.. In AAAI, Cited by: SS4.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**WMT19**} & \multicolumn{2}{c}{**WMT20**} \\  & **En\(\Rightarrow\)De** & **De\(\Rightarrow\)En** & **En\(\Rightarrow\)De** & **De\(\Rightarrow\)En** \\ \hline Mask-Predict & 34.79 & 37.04 & 25.24 & 36.36 \\ Multi-Task NAT & **35.38\({}^{\dagger}\)** & **37.62\({}^{\dagger}\)** & **25.72\({}^{\dagger}\)** & **36.58** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation of translation performance on WMT19 En\(\Leftrightarrow\)De and WMT20 En\(\Leftrightarrow\)De test sets with knowledge distillation. The _Importance Annealing_ is adopted by default.

 

Wenxiang Jiao, Xing Wang, Shilin He, Irwin King, Michael Lyu, and Zhaopeng Tu. 2020. Data rejuvenation: Exploiting inactive training examples for neural machine translation. In _EMNLP_.
* Kasai et al. (2021) Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A Smith. 2021. Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation. _ICLR_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Li et al. (2019) Zhuohan Li, Zi Lin, Di He, Fei Tian, QIN Tao, WANG Liwei, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive machine translation. In _EMNLP-IJCNLP_.
* Liu et al. (2019) Shikun Liu, Edward Johns, and Andrew J. Davison. 2019. End-to-end multi-task learning with attention. In _CVPR_.
* Ma et al. (2020) Dabiao Ma, Zhiba Su, Wenxuan Wang, and Yu-Hao Lu. 2020. Fpets: Fully parallel end-to-end text-to-speech system. In _AAAI_.
* Morcos et al. (2018) Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In _NeurIPS_.
* Neubig et al. (2019) Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang, and John Wieting. 2019. compare-mt: A tool for holistic comparison of language generation systems. In _NAACL (Demonstrations)_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In _NAACL (Demonstrations)_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _ACL_.
* Raganato and Tiedemann (2018) Alessandro Raganato and Jorg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In _EMNLP 2018 workshop BlackboxNLP_.
* Ruder (2017) Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. _arXiv preprint arXiv:1706.05098_.
* Saphra and Lopez (2019) Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In _NAACL_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _ACL_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NeurIPS_.
* Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In _EMNLP-IJCNLP_.
* Wang et al. (2019a) Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019a. Self-attention with structural position representations. In _EMNLP-IJCNL_.
* Wang et al. (2019b) Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019b. Non-autoregressive machine translation with auxiliary regularization. In _AAAI_.
* Wei et al. (2019) Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In _ACL_.
* Zhou et al. (2020) Long Zhou, Jiajun Zhang, and Chengqing Zong. 2020. Improving autoregressive NMT with non-autoregressive model. In _Proceedings of the First Workshop on Automatic Simultaneous Translation_. ACL.

 

Implementation

We implemented the proposed Multi-Task NAT model based on Fairseq Ott et al. (2019), an open-source framework for the sequence to sequence learning. More specifically, we added the Transformer decoder (AT) to the standard Mask-Predict (NAT) model, and the encoder output is fed into the Transformer decoder with the encoder-decoder attention. We implemented a new loss function by combining the AT loss and NAT loss, and we jointly optimized all parameters. We will make all the code publicly available for future use.

### Hyperparameters

For the NAT baseline model, we followed the hyperparameter settings as described in Mask-Predict Ghazvininejad et al. (2019). More specifically, we trained all models for up to 300K steps with 128K (16000\(\times\)8) tokens per batch using Adam Kingma and Ba (2014) with \(\beta=(0.9,0.98)\) and \(\epsilon=10^{-6}\). We adopted the warm-up learning rate scheduler, which linearly increases from \(10^{-7}\) to a peak of \(5\cdot 10^{-4}\) within 10,000 steps, and then decays with the inverse square root schedule.

For the AT baseline model in our preliminary experiments, to make a fair comparison, we reused most parameter settings (e.g., training steps, initialization method, warm-up schedule) of the NAT model to train a strong AT baseline. Since half of the training tokens are randomly masked in the NAT baseline model, we set the batch size in the AT model as 64K (8000 \(\times\) 8) tokens.

For our Multi-Task NAT model, we followed the same parameter setting as the NAT baseline model. We saved checkpoints every 2,000 steps and average the last 5 checkpoints as the final model. Due to the limited training data in the En\(\Leftrightarrow\)Ro translation task, we adopted the early stopping to prevent over-fitting in both the baseline NAT model and our model.

In the inference phase, all NAT models and our model were using the iterative decoding strategy to perform non-autoregressive translation, with the max decoding steps of 10 and length beams of 5.

### Model Training

We applied the mixed precision training to all models to accelerate the training speed. The baseline model took around 15 hours to finish training on 8 Nvidia V100 GPU, while ours took around 30 hours. During the training phase, our model had an extra decoder of 25M parameters, but the parameter size (around 64M) was the same as the baseline model in inference.

### Model Inference

In our model, the auxiliary AT decoder is employed to better capture the source sentence representation during the training phase. However, since our focus is on the NAT translation performance, the auxiliary AT decoder is not necessary during the inference. We simply ignored the AT decoder in our approach to keep the same translation speed as the baseline NAT models, leaving the space for future works to leverage the ignored AT decoder and benefit the translation.

### Significance Tests

We use the compare-mt Neubig et al. (2019) with 1000 re-samples to perform statistical significance tests for the Mask-Predict model output and our model output. The results show that our model significantly outperforms the baseline model in all language directions. All these experimental results confirm the effectiveness of our proposed method even with knowledge distillation. Besides, considering the results in Table 2, as a new knowledge transfer approach, our proposed multi-task learning method Multi-Task NAT can complement the classic knowledge distillation method, which is promising for future exploration.

Figure 2: Representation similarity evolvement during the learning course, NAT (or AT) denotes the baseline of representation similarity between two baselines NAT (or AT) models with different initialization seeds. NAT denotes the representation similarity between NAT and AT encoder. Best viewed in color.

 

Probing Tasks

We conducted 10 probing tasks3 to study what linguistic properties are captured by the encoder [25]. A probing task is a classification problem that focuses on simple linguistic properties of sentences. 'SeLen' predicts the number of words in a sentence. 'WC' predicts which of the target words appear on the given sentence given its sentence embedding. 'TrDep' checks whether the encoder representation infers the hierarchical structure of sentences. In 'ToCo' task, it measures the sequence of top constituents immediately below the sentence node. 'BShif' predicts whether two consecutive tokens within the sentence have been inverted or not. 'Tense' asks for the tense of the main-clause verb. 'SubN' focuses on the number of the main clause's subject. 'ObjN' studies the number of the direct object of the main clause. In 'SoMo', some sentences are modified by replacing a random noun or verb with another one and the classifier should identify whether a sentence has been modified. 'CoIn' contains sentences made of two coordinate clauses. Half of the sentences have inverted the order of the clauses and the task is to tell whether a sentence is intact or modified.

Footnote 3: [https://github.com/facebookresearch/SentEval/tree/master/data/probing](https://github.com/facebookresearch/SentEval/tree/master/data/probing)

We first extracted the sentence representations of input from the AT and NAT encoder, which were used to carry out our probing tasks. For both the AT model and NAT model, the mean of the encoder top layer representations was used as the sentence representation. The classifier we used in this work was a Multi-Layer Perceptron (MLP) with a hidden dimension of 200. We optimized the model using the Adam optimizer with a learning rate of 0.001. The batch size was set to 64 and we trained the model for 4 epochs. The 10-fold cross-validation was also employed to get the final performance.

## Appendix C Representation Similarity

In this section, we adopt CCA [8], a widely-utilized representation comparison metric [7; 9], to calculate the encoder representation similarity throughout the training course. The CCA score ranges from 0 to 1, and a higher CCA score indicates both encoder representations contain more similar information.

In our experiments, we first extracted the sentence encoder representations for 100,000 training examples (which is sufficient enough for CCA calculation) for each model. Then, we calculated the CCA score between the encoder representations of two models, where the representation was the mean of the top-layer encoder representations. In addition, to demonstrate the evolvement of the representation similarity during the learning course, we calculated the CCA score between two models for every 10,000 training steps.

Figure 2 shows the evolvement of representation similarity during the model training. We compute the CCA similarity between two NAT (or AT) models under different initialization seeds and take the similarity as the baseline (green and blue lines in Figure 2). Comparing to the baseline similarity of NAT (or AT), a lower similarity between AT and NAT representations (red line in Figure 2) indicates that AT and NAT encoders capture different source information.