Extrapolating Multilingual Understanding Models as Multilingual Generators
Authors: Bohong Wu
Fei Yuan
hai zhao
Lei Li
Jingjing Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual Translation
Prompt Tuning
Non-autoregressive Generation

Abstract: Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these models are not capable of generating high-quality text compared with decoder-based causal language models. Can we transform a pre-trained language understanding model into an effective  language generation model? We propose a Semantic-Guided Alignment-then-Denoising (SGA) approach to adapt a multilingual encoder to a multilingual generator with a small number of additional parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-R$_{large}$. On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators. Our code is available at https://github.com/chengzhipanpan/XLMR4MT.

Extrapolating Multilingual Language Understanding Models as Multilingual Language Generators
Bohong Wul Fei Yuan', Hai Zhao 122 Lei Li', Jingjing Xu
Department of Computer Science and Engineering, Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering. Shanghai Jiao Tong University, Shanghai, China Shanghai Artificial Intelligence Laboratory Carnegie Mellon University chengzhipanpanesjtaa edu. en, yuanfei@pjlab. org. cn, Shachai@cs.sjtuu edu. cn, leiliecs, cmu.edu, j jingjingxupku, 02@gmail.
Abstract
Encading
gevenitel
Multilingual understanding models (or encoder- based), pre-trained via masked language mod- eling, have achieved promising results many language understanding tasks (e.g., mBERT). However, these models are not capable of generating high-quality text compared with decoder-based causal language models. Can we transform a pre-trained language understand- ing model into an effective language genera- tion model? We propose a Semantic- Guided trremeenntttee--Deoiiiiiin (SGA) approach to adapt multilingual encoder to a multilingual generator with a small number additional parameters. Experiments show that the pro- posed approach is an effective adaption method, outperforming widely-used initialization- based methods with gains of 9.4 BLEU machine translation, 8.1 Rouge-L L on question genera- tion, and 5.5 METEOR XLM-Riarge- On the other hand, we observe that XLM-R still inferior to mBART super- vised settings despite better results on zero-shot settings, indicating that more exploration is re- quired to make understanding models strong generators. Our code is available at https: Wgggtubb.co/kcceengggaaaann/LLLe
scmcantic holden
white
H
Semantic Guidance
M
To
T
TiMNs
Alignment
Iterative Denoising
Figure I: An overview of semantic-guided generation using pre-trained understanding models. The encoding step is responsible for mapping the source input into a shared space that supervises the following genera- tion. By taking the source imput and a blank sentence (white noise) as input, the alignment stage generates target tokens simultaneously. Then, we feed the source representations and the generated sequence into the de- noising stage for NAR denoising. The denoising step performed iteratively until the generated text keeps unchanged or reaches the maximum loop.
story generation
et al., 2020; Ouyang et al., 2021). Typically, the AR framework, where target sequence is gener- ated from left to right, succeeds in multilingual generation tasks (Chen et al., 2022; Qi et al., 20. As a comparison, encoder-based models are NAR models that are usually limited to understanding tasks (Conneau et al., 2018). Despite superior un- derstanding results over AR models, these NAR models still struggle to handle a wide range of mul- tilingual generation tasks. However, NAR models still have obvious advantages in generation effi- ciency and decoding flexibility (Gu et al., 2018; Qian et al., 2021; Huang et al., 2021; Ghazvinine- jad et al., 2019; Saharia et al., 2020), which enables generating multiple tokens at one time in arbitrary order. Considering these strengths, this paper aims to explore methods make multilingual encoder- based models better generators with a small number of new parameters.
1 Introduction
Multilingual encoder- models (e.g., mBERT (Pires et al., 2019), XLM-R (Comneau et al., 2020)), pre-trained via masked language modeling, have demonstrated strong performance on wide range of understanding tasks (Conneau et al., 2018; Liang et al., 2020; Hu et al., 2020). Existing multi- lingual pre-trained models can be classified into two settings: autoregressive models (Liu et al., 2020; Xue et al., 2021; Scao et al., 2022) and non- autoregressive models (Pires et al., 2019; Conneau
"This work was partially supported by the National Key R&D Program of China N0..2022D00160 60100)
This work was also partially supported by Joint Research Project Yangize River Delta Science and Technology Inno- vation Community (No. 2022CSJGG1400)
There is limited research focusing on empow- cring understanding models with the generation ability. Traditional methods usually use pre-trained
=Corresponding author.

---

encoders initializers for AR models in various monolingual generation tasks (Su et al., 2021). De- spite promising results, does not satisfy our tar- get that fixes pre-trained parameters to build a unified model for any language tasks. More re- cently, researchers have focused on lcarning-free approaches (Wang and Cho, 2019; Kumar et al., 2022b; Qin et al., 2022). One typical approach is iteratively choosing tokens to mask and sam- pling proposals using energy models (Mireshghal- lah et al., 2022), resulting in surprising high latency. Furthermore, these learning-fire free methods are usu- ally limited to controllable generation and are still inferior in handling complicated tasks like ma- chine translation. Unlike these monolingual stud- ies, adapting multilingual understanding models to multilingual generation has its own challenges: semantic constraints under conditional generation where the generation process should follow the semantic constraints given source texts in any lan- guage, and parameter efficiency constraints where a single model can serve text generation in any language.
pressive zero-shot cross-lingual ability in transla- tion tasks, outperforming a multilingual AR adap- tation model, mGPT MSP by a large margin. On the other hand, we also notice the gap be- tween XLM-R and AR models. Generally, XLM- R with fine-tuning is largely inferior to mBART fine-tuning, With our methods, the gap is largely reduced but still exists. In future work, we would like explore pre-training methods make multi- lingual understanding models better generators. Our contributions can be summarized as follows:
We propose an efficient adaptation method to make multilingual understanding models better generators in a parameter-cfficient way.
We present a semantic-guided denoiser, which can efficiently improve the generation quality.
Experiments show that our proposed method outperforms traditional initialization-baxt adaptation methods by a large margin.
We propose a semantic-guided approach to ad- dress these challenges, with a two-stage generation process: cegeoment-thee-denooie share the same pre-trained parameters and only add small number of new prompt parameters for adaptation. Given that masked language model- ing (MLM) also denoising objective, existing multilingual pre-trained models can be naturally adapted to good denoisers. Therefore, we intro- duce a denoising stage into our framework. The whole generation process is shown in Figure 1. The encoding part maps the source input into a shared space that supervises the following generation. By taking the source input and a blank sentence as input, the alignment stage generates target tokens simultaneously. We feed the source representa- tions and the generated sequence into the denoising module for NAR denoising. The denoising step is performed iteratively until the generated text keeps unchanged or the maximum loop is reached.
2 Related Work
In this section, we review the related studies in- cluding parameter-efficient adaptation, adapting en- coder models as generators, and non-autoregressive generation.
The two stages
Parameter efficient adaptation Pre-trained lan- guage models (PLMs) (Devlin al., 2019; Liu et al., 2019; Clark al., 2020; Conneau et al., 2020) have achieved overwhelming performance in a vari- ety of downstream tasks. Parameter-efficis tuning is a hot research direction to adapt PLMs to down- stream tasks with only training few parameters. Adapter- based methods (Bapna and Firat, 2019) are one of the popular parameter-efffcient approaches. Recent studies (Üstün al., 2021; Cooper Stick- land al., 2021) proposed to use adapters on the top of an mBART (Liu et al., 2020) model, enabling a flexible and well well-performed method for plug-and- play translation. More recently, prefix tuning and other prompt- based methods (Li and Liang, 2021; Lester et al., 2021; ; Liu et al., 2022; Tan et al., 2022) have proved to be extremely helpful, and can casily support mixed-task inference as it does not require changing the architecture of PLMs. In this work. we follow this research thread for efficient adap- tation and apply prompt-based approaches in our work.
Experiments demonstrate that our model has achieved better results on various generation tasks than traditional fine-tuning-basse approaches that directly use NAR pre-trained models as initializa- tion, with gains of 9.4 BLEU on machine trans- lation, 8.1 Rouge-L. on question generation, and 5.5 METEOR on story generation on XtagRt More promisingly, our method has achieved im-

---

Encoder promp
Alignment prompt
(1)+2); Semantic-Guided Alignment (I):3): Semantic-Guided Denoising
Semantic hidden
Dencising prompt
Encoder
Alignment
Hell world
Denoising
Hello world
Fixed NAR LM
Fixed NAR LM
Fixed NAR LM
Denathe dmowrnng
6: R
Holl
world
())
(3)
Figure An overview of the generation units. All units share the same pre-trained parameters and individual prompt parameters (shown in blue square, purple square, and brown square). The encoder maps the source input into sequence of hidden representations (yellow), which are then fed into a decoder and denoiser for target generation. The alignment unit responsible for generating piece of target text. The denoiser is responsible for refining the generated text.
Adapting encoder-based models for generation Previous works proposed to use pre-trained un- derstanding models to initialize incoder-dcooder models (Chen et al., 2021; Ma et al.. 2021) or use the contextualized embeddings produced by under- standing models as more aligned inputs for gener- ation (Xu et al.. 2021a). With the trend of scaling up models in recent years, it gradually becomes impossible fine-tune the whole language model in each language direction. In addition, there have been several learning-free methods that adopt en- coder models as energy scorers for controllable text generation (Mireshghallah et al., 2022; Kumar et al., 2022a). Although these methods do not need to fine-tune the pre-trained model, they require multi-steps of sampling and refinement, resulting in surprising inference latency.
both comparable performance and better latency with AR baselines (Lee et al., 2018; Ghazvininejar et al., 2019; Huang ct al., 2021; Saharia et al., 2020). For example. SUNDAE (Savinov et al., 2021) pro- posed step-unrolled denoising, and achieved good performance in both machine translation and text infilling. Similar iterative idea has been adopted at recent diffusion models (Li et al., 2022; Gong al., 2022). In this work, we adopt an iterative decoding idea to take advantage of the denoising abilities of encoder-based models which are pre-trained with denoising objectives.
Notation and Background
Prompt tuning For efficient adaption, we fol- low mGPT+MSP (Tan et al., 2022) and use prompt tuning to adapt an existing pre-trained NAR pre- trained multilingual understanding models to gen- erators. Formally, we denote Kt and V as the key-value pairs in the 1-th Transformer layer. The introduced prompt-tuning parameters are (K, V) pairs, which will be concatenated with current key- value pairs during training and inference. prompt tuning, we denote the forward pass of a pre-trained LM as fLM (0p. X). which accepts two inputs in- cluding prompt parameters 0,, and the source se- quence X.
Non-autoregressive generation Our work aims at adapting multilingual encoders to multilingual generators, instead of developing a NAR archi- tecture like previous NAR literature does. There- fore, there is lot of difference between our work with previous NAR studies. Despite different motivations, our implementation also uses sev- eral NAR techniques, like CTC (Graves et al., 2006; Libovický and Helel, 2018) and Mask- Predict (Ghazvininejad et al., 2019). For clarifi- cation, we also review the thread of NAR gener- ation. Single-step NAR generation is a popular research direction that generates text at one time. To mitigate the gap between single-step NAR meth- ods and AR methods, researchers have proposed dligmment-based methods (Libovický and Helcl, 2018; Ghazvininejad et al., 2020; Du et al., 2021) or glancing-bard based methods (Qian et al., 2021). As a compromise, iterative NAR methods can provide
SGA Approach
4.1 Overview
Based a fixed multilingual understanding model, Figure presents the overview of our proposed SGA, which contains three stages, including seman- tic encoding, alignment and then denoising. Sec- tion 4.2 presents the semantic encoding unit, which

---

maps sentences in all languages unified space, Section 4.3 presents the alignment unit, which gen- erates a target sentence. Section 4.3 presents the denoising unit, which refines the generated sen- tences under the guidance of semantics.
constrained generation tasks, specifically, trans- lation, we choose CTC loss objective for its ef- ficiency in speed, as it is a one-step NAR gen- eration method. For free generation tasks, CTC loss performs poorly because gree-generation tasks intensify the multi-modality problem (Gu et al., 2018), which we will discuss in Section 5.5, On the contrary, iterative methods choose the best pos- sible modality during early iterations of generation. Therefore, we use
4.2 Semantic Encoding
Suppose pre-trained multilingual language model with L layers. define prompt parameters pp.
(K! V!liA) for all layers. These parameters are then concatenated with the key-value pairs of at- tention in each layer to extract the hidden represen- tation of the source sequence X = T2,,,2 =a]. Therefore, we can get the layer-wise hidden repre- sentation 1 via
which an itera- tive NAR generation method that sacrifices speed for performance.
4.4 Semantic-guided Denoising
Due the limitation of trainable parameters and the non-autoregressive nature, the generation re- sult of the first-stage alignment is usually far from satisfying. Thanks to the denoising pre-training objective MLM, current language models can be easily adapted to denoiser. In this step, we also add prompt parameters for denoising 8yp (Kbb Vb-L)) ) to efficiently adapt the understanding model to a language-specinn denoiser. Similarly, we get semantic-guie guided denoising prompt by the (2) following equation:
h1L = SLM X)
(1)
For semantic guidance, we directly pass hid to the alignment unit and denoising unit as new prompt parameters. We use additional two pro- jection layers WK and Wv to project hg into semantic hidden states, denoted as K.'L and V.!.L.
K-LL WW VILL h W.
4.3 Semantic-guided Alignment
8pu = (cconca([[" K2-"1, concat((V, 1. vdl
(5)
This unit generates the target sequence in parallel, by taking sequence of white noise input, de- noted as Yblank (a sequence of <mask> tokens in our experiments). Similarly, we introduce align- ment prompt 0p. = (K)- V.l.L) to efficiently adapt the pre-trained model to generate a target sequence. To grab information from the source se- quence, we directly concatenate (K1:l, v1:1) and (KI-L, v!lL), where we get
We take the output sequence in alignment stage as input which is denoted as Ti- To avoid over- fitting, we add random noise including random deletion or repetition to sequence To = To +e. We can then acquire the denoised logits Ti by:
Ti = fiM (0 pay To)
(6)
Pp. = (concat( (Ik K2111) concat((V.ld D)
(3)
We repeat this step and treat Ti as new input to get Tp. The loop is running until the output sequence keeps unchanged or we reach the maximum loop number.
The alignment output is obtained via:
To = FLM (0ps Yrlank)
For denoising, we use a CTC-based denoiser after the alignment process and adopt the CTC loss (fp. (pa) given training pair (Y, Ti) where T; is the output sequence the i-th step. For translation, the outputs of the alignment stage are directly fed to the denoiser. For other generation tasks, we upsample the alignment result by a factor of 2 by duplicating each token, and then fed the duplicated sequence to the CTC-based denoiser.
Formally, we define the alignment loss as (0 pp)) given training pair (X, Y') sampling from dataset. For alignment, we use two vari- ants of non-autoregresssve loss train new pa- rameters, including Connectionist Temporal Clas- sification(CTC) (Libovický and Helcl, 2018) and Mask-Preat Predict (Ghazvininejad et al., 2019). For

---

4.5 Training Objective
For fine-tuning using pre-trained language mod- els like XLM-R and mBART, we use batch size of 4k tokens and a much smaller learning rate of 3e-5, We train the pre-trained models for a maximum of 80,000 steps. We also use Fairseq.
The final loss is a combination of the alignment loss and denoising loss by the following equation:
L Li (0p. Pp.) ) C2(0ps: 0ps)
For adaptation methods on PLMs, we directly follow the hyperparameter setting of (Tan et al.. 2022), with batch size of 32k tokens and learn- ing rale of 7c-4. We train these models for maxi- mum of 40,000 steps, and average the 5 best check- points for inference. We use THUMT (Tan et al.. 2020) for the implementation of the adaptation methods. For translation tasks, the training takes around 40 hours on 8 A100-SXM--800 -0GB GPUs in each translation direction to adapt an XLM-R targe model to generators.
5 Experiments
5.1 Settings
We run experiments on three multilingual genera- tion datasets including machine translation, ques- tion generation, and story generation. Mapping between language codes and full names of all languages used in our paper is presented in Ap- pendix A.
Dataset For experiments on both bilingual and multilingual translation, we use TED dataset (Qi et al., 2018). We focus on English- settings and choose 10 languages (Ar, De, Es, Fr. He, It, Ro. Ru, Tr, Vi) with the most training data to construct our multilingual translation task. We choose five additional languages (Kk, Be, Eu, Ms, Bs) with the least training data (less than 6k) for zero-shot cross-lingual evaluation. Details are presented in Appendix
Evaluation Metries and Hardware We calcu- late case-sensitive BLEU (Papineni et al., 2002) using the sacrebleu toolkit (Post, 2018) 8) for transla- tion evaluation We use ROUGE-L (Lin, 2004) for both question generation and story generation. We also report METEOR (Banerjee and Lavie, 2005) for story generation. For speed calculation, we average the running time on the test set with batch size set to on a single A A100-SXM-80GB GPU, and statistics are averaged by three runs.
For experiments on question generation, we use the Question Generation (QG) split of XGLUE dataset (Liang et al., 2020). Since XGLUE only provides training set in the Taglish-Englise di- rection, we use M2M-100-418M (Fan et al., 2021) translate the English training set to all other languages. We train all models in the En-+X direc- tions and evaluate them on the X-*X test sets. For simplicity, we report results on En -En, En >De and En-Fr, where results on En-tE represent the monolingual generation ability, and results on En-De, En-FFr represents zero-shot tross-lingual generation ability. For experiments story gen- eration, we use the Story Generation (SG) split of MTG dataset (Chen al., 2022). For simplicity, report monolingual generation result on En-=En, and cross-lingual generation results on De and Fr-EEn.
Pre-trained Multilingual Models We mainly use three kinds of pre-trained multilingual mod- els in our experiments, including (1) a decoder- only causal model, mGPT (Tan et al., 2022), (2) an decoder model, mBART (Liu et al., 2020), and (3) encoder-only model, XLM-R (Conncau et al., 2020).
5.2 Baselines
We mainly compare the following baselines in our experiments.
Transformer (Autoregressive, AT) (Vaswani et al., 2017). We use the transformer-big set- ting.
Transformer Non-autorcgressive, NAT) (Li- bovický and Helcl, 2018). We conduct NAT experiments Transformer with CTC loss using the transformer-big setting.
Implementations We use a batch size of 32k to train all transformer models in both AT and NAT. Following (Xu et al., 2021b), transformer-big setting with learning rate of 5e-4 and dropout rate of 0.3. We train these models for a maximum of 50 epochs, and average the best checkpoints for inference. We use Fairseq (Ott et al., 2019) for implementation.
use the
mTTrnsformer. We train multilingual AT Transformer with 12 encoder layers and 12
htw.://eesee.cccmciiieeeefefonoo 1 Balsmooth.exx! version:2.0.0

---

Group
Model
Speed
Ar-
-Eo
Fc-eEa
Ru-eEa
Tr=Eaa
Vi-ABa
Ang.
Biliggal
Tatatemer (XT)
1.0x
24.7
Mobsingua
50M
32.4
35.2
35.6
21.8
22.3
30.3
38.9
(NT)
19M
21.9
AT intidiizaiio
3MM
0.9
16.6
23.5
29.5
21.0
24.5
16.1
17.6
15.7
21.4
SOA who doneivingg
6M
6.8x
24.5
29.0
16
92.6
79.7
50.4
28.7
18.0
15.0
21.5
76.9
PLM Adqpaation
S0A
8M
3.0x
25.1
33.0
40.0
35.5
35.3
32.9
50.3
20.3
19.6
23.9
29.6
XLM Rore
AT imitalizztion
SWIM
0.6x
19.2
25.7
12.4
79.9
25.6
25.4
24.9
21.5
17.4
18.3
24.1
+ SGA who,, dansising
19k
3.7x
25.2
33.3
32.9
36.1
35.5
34.1
31.5
21.4
23,4
24.4
30.6
+ SGA
21M
1,9x
38.7
37.0
40.9
38.6
38.5
37.5
34.2
24.0
27.2
26.4
33.5
Group
Modell
Parmm.
Speed
Ra-Ar
Fn-
Fa- Fa-FF
Fa-
Fa--it
En-sRo
Rn-=Rt
Fn- Tr
En-VVi
Aug.
Millggaa
Traoofomaa (NAT)
Transfomner (AT)
454M
13.4x
13.0
62
30.0
10.6
25.2
23.0
27.2
14.6
17.5
13.1
19.5
5.3
15.0
0.4
15.2
13.1
27.8
Mubiiingua
"olafter
5IM
0,8x
0.5%
16.3
12.3
29.3
23.6
35.
38.9
92.2
38.4
25.0
18.9
33.6
28.4
26.3
21.3
19.1
14.8
15.2
11.4
30.3
25.2
22.1
27.3
SGPT MSP (AT)
19M
0.2x
11.6
24.1
31.7
32.5
20.7
29.6
19.2
17.9
11.7
24.4
22.3
XLM Ruce
At intisization
300
0.9%
3.9
17.6
26.4
22.6
14.1
20.1
15.8
10.6
6.5
18.4
16.0
SGA who,, dencising
6M
6.8%
8.7
19.4
28.3
23.8
16.5
25.8
18.8
13.5
7.2
22.8
18.4
Adqpnsion
SGA
3.0x
18.9
14.7
20.9
XLM-BBrw
15M
3.7x
11.3
19.5
13.1
21.3
21M
25.2
21.3
15.5
11.8
23,9
Table 1: Results of X->N and EN-XX translation. "Param." represents the total number of trainable parameters. "Speed" represents the inference speed when batch size is 1. Scores in bold represent the best performance in the Adapt PLM setting. Compared with the traditional fine-tuning method that directly adopts XLM-R as initialization, SGA brings large performance gains, with 8.2 BLUE on XLMM-Rase and 9.4 BLUE on XiaM-Riarge on X-EN, and with 4.9 BLEU on (LM-Raase and 4.6 BLEU on XLM-Rtarge EN- showing the effectiveness of SGA on adapting multilingual understanding models to multilingual generators.
decoder layers on the TED multilingual trans- lation datasets. Other hyperparameters shared with Transformer-big. To report X->En and En results, we train two m'Transformer models using all X-En and En +X data TED, respectively.
5.3 Main Results
SGA achieves large performance improvements over traditional initialization adaptation Table 1 presents the multilingual translation ex- periments. We find that initializing an autoregres- sive Transformer model from XLM-R only brings slight improvements by comparing with Trans- former (NAT). We speculate the different nature of AR and NAR leads to performance degrada- tion when using XLM-R as initializers. Compared with the traditional fine-tuning method that directly adopts XLM-R as initialization, SGA brings large performance gains, with 8.2 BLUE on XLM-Raaas and 9.4 BLUE on XLMM-R on X EN,, and with 4.9 BLEU on XLM-RRase and 4.6 BLEU on XLM- XLM-RRarge on EN-xX, showing the effectiveness of SGA on adapting multilingual understanding models to multilingual generators. With 2 train- able parameters, our method achieves comparable performance with bilingual counterparts and even better performance in several language directions (De, He, Tr). The bottom part presents translation results on the En->X XX directions.
m Transffrmer + adapter. We use language- specific adapters (Bapna and Firat, 2019) on top of our trained mTransformer. We append adapters to both encoder layers and decoder layers and use foed-forward layer dim of 1,024, which finally results in 50M extra pa- rameters for each language pair.
mGPT + MSP (Tan et al., 2022). mGPT + MSP introduces multi-stage prompting over multilingual GPT model with 560M param- eters. We implement this baseline following the same setting as the original paper.
XLM-R w. AT initialization. Under this setting, we initialize the encoder of au- toregressive Transformer with the weights of XLM-R, and fine-tune the whole parameters.
SGA shows better efficient inference over adap- tation baselines Compared with the original mTransformer baseline including the adapter set- ting, our method achieves 1.9/0.8 = 2.4x speedups with better performance. As a compari-
We use two variants of XLM-R: XLMMR Bassc with 270M parameters, and XLM-R Mi-Raarge with 550M parameters.

---

mance with multilingual Transformer and mGPT + MSP, mBART. All baselines including m mTrans- former are trained using the combined dataset.
25.0
Transformer
32.5
XLMRYSOR
SGAlshare)
Prompt sharing enables a compact X- trans- lation plugger SGA achieves better tradeoff between parameter size and inference performance by sharing prompts across all X->En directions, which achieves competitive performance with the bilingual Transformer. As adaptation methods, Fig- ure presents the tradeoff between parameter size and performance. Performance is evaluated by av- eraging the test set BLEU in all directions. SGA achieves better tradeoff performance than the AT counterpart, mGPT + MSP. With only 21M param- eters, SGA enables multilingual understanding LM unified and impressive X-+En translator.
37.5
25.0
22.5
MMSShrsPl
20.0
Parameters(M)
Figure 3: Tradeoff between parameter size and BLEU scores. "share" represents prompt sharing. Our pro- posed XLMAR+SGA with prompt sharing strategy can further reduce parameters without sacrificing much of the performance. As comparison, mGPT + MSP drops significantly.
Prompt sharing brings impressive zero-shot cross-lingual transfer Sharing prompts also em- power SGA with strong zero-shot cross-lingual transfer ability. We choose 5 languages (Kk, Be, Eu, Ms, and Bs) with the least training data in X->En directions in the TED dataset, and com- pare performance with multilingual Transformer, mGPT MSP and mBART. Table 2 presents the zero-shot cross-lingual transfer ability. (i) Trained only language directions, mTransformer out- performs mGPT MSP in the supervised language directions with large performance gap, while our +SGA
son, mGPT + MSP brings higher inference latency due to multi-stage prompting and AT decoding.
Denoising brings large performance gains in all directions On both XLM-R and XLM-R our proposed denoising technique brings an aver- age gain of 2.8 BLEU by increasing a very small amount of parameters. This confirms our conjec- ture that multilingual understanding models can be parameter-efficient language-specific denoisers due to the denoising pretraining nature of MLM.
method, XLM- Raagee RSGA is still superior to both
methods. (2) XLM-R+SGA achieves good perfor- mance in zero-shot X-En experiments, with sub- stantial performance improvement when compared with mGPT + MSP and mTrrasformer. (3) Al- though still lags behind the performance of mBART on supervised language directions, mBART sup- ports much fewer languages than XLM-R (50 vs. 100), which presents limitations in zero-shot cross- lingual performance.
XLM-R boots the performance of NAT Exist- ing NAT model (NAT+CTC) produces poor results in all language directions. It is because NAT gener- ally requires AT model generate distillation datasets (Gu al., 2018) which do provide in this paper. The NAR nature of XLM-R makes it possible to boost NAT performance. With SGA, XLM-R outperforms NAT basclines by a large mar- gin, indicating that NAR pre-training can be further explored in future work to make multilingual un- derstanding models better generators.
5.5 Other Generation Scenarios
In this section, we test the performance of our model in various generation tasks other than mul- tilingual translation to further explore the gener- ation ability of multilingual understanding mod- cls. Table 3 presents the results of both mono- lingual and cross-lingual results question gen- eration and story generation. For both tasks, we provide monolingual result in the En En direc- tion for reference. For the question generation task on XGLUE, since it only provides test sets in the
5.4 Prompt Sharing Analysis
We further reveal the potential of our proposed SGA by sharing prompts across languages. We combine datasets in the 10 Xn language direc- tions selected in Section 5.1, and compare perfor-
'MSP uses different tokenization processing scripts for evaluation, To have a fair comparison, we reproduce the mGPT + MSP results in all language directions based on their public onde.

---

Supervised Es
Unsupervised Eu"
Group
Model
Param.
De
It
Ms"
Bs"
mTrrasformer mBART-ine-tune
432M
32.5
39.2
37.5
35.5
1.2
2.1
2.1
1.4
1.8
Multilingual
610M
41.0
46.3
44.4
42.8
13.6
2.2
1.6
21.0
16.5
mGPT MSP
19M
27.6
35.2
33.2
32.0
6.7
17.1
10.7
19.3
14.3
Adapt PLM
XLM-Roaace SGA
8M
31.9
37.5
36.0
33.9
7.2
19.6
12.9
24.0
27.0
XIaM-Rtarge SGA
21M
34.2
39.3
38.3
36.2
11.4
26.8
20.7
30.1
34.0
Table 2: Zero-shot translation performance on TED in the X-=n directions. Our method achieves impressive performance in the zero-shot cross-lingual setting, with significant improvement in all unsupervised translation directions compared to mGPT MSP. represents that this language is not supported mBART.
Question Generation
Story Generation
Models
Para.
Speed
En-=En RL1
Ee=
En- RL
En-AEn
De-EBn
Fi-En
RL1
RL1
Meteort
RL
Metcort
RL1
Meteor
mGPT + MSP
19M
1.0%
36.3
17.5
17.7
16.7
16.1
13.7
13.7
14.2
13.8
XI.M-Rtarge
AT initialization
960M
1.7x
28.9
10.5
8.4
9.6
9.3
10.3
+ SGA w/o. demoising
15M
4.5x
34.4
16.9
19.2
15.7
15.5
14.3
14.4
15.3
13.9
+ SGGA
21M
4.3x
35.6
17.4
19.9
15.5
15.9
14.4
15.0
14.8
14.6
Table Results on question generation and story generation. RL represents the Fl-score of Rouge-L. represents zero-shot cross-lingual scenarios. SGA beats initialization-bal methods XLM-R in all cross-lingual scenarios with substantial improvement, and achieves comparable results with mGPT MSP.
X-* directions, we train all models on the train- ing set of En +X directions, and evaluate the model performance on the X->X directions for zero-shot cross-ling cross-lingual generation. For the story generation task on MTG, we test supervised cross-lingual gen- eration performance on the X-En direction. We report Rouge-L scores for both tasks, and report METEOR scores additionally for story generation. Table 3 presents the generation results. For for
Group
# Iter.
Meteort
Speed
2
14.5
14.1 sent's
wfo. denoising
4
15.5
8.3 sent's
8
15.8
5.2 sent/s
0
13.0
21.6 sent/s
2
15.3
11.8 sent/s
w: denoising
4
15.9
7.9 sent/s
8
16.1
4.8 sent/s
Table Trade-off between CTC-based denoiser and number of iterations on En-*En generation on story generation. Batch size is set to 1. Denoising brings bet- performance and presents better tradeoff between performance and inference speed.
free generation tasks, we use the aligmment stage, and we set the iteration num- ber to in this table. (i) Our proposed method XLM-R+SGA can achieve comparable perfor- mance while notable acceleration, when compared with an autoregressive- based model, mGPT + MSP, on almost all generation tasks. (ii) Using XLM-R initialize autoregressive Transformer totally loses the zero-shot cross-lingual ability. Although it performs moderately on the supervised monolin- gual direction (En->En) on Question Generation, it performs poorly on the zero-shot directions in- cluding En-xDe and En-*Fr. (iii) Our denoising technique is proven helpful in further improving the generation quality in both tasks without sacrificing much of the speed.
aligmment stage. Free generation introduces much more modalities than constrained generation tasks, specifically, translation, which intensifies the multi- modality problem NAR generation (Gu et al., 2018). Therefore, we use an iterative method, Mask-Predict, to improve the generation quality for the aligmment stage of our proposed SGA.
Although increasing the iteration number in the alignment stage can obviously lead to better perfor- mance, it will also intensify the latency problem. Our CTC-based denoiser can not only bring better performance, but also a better tradeoff between per- formance and speed, which is presented in Table 4. When the iterations of the alignment stage is set to the same, using the CTC-based denoiser leads to
Tradeoff between iterative prediction and CTC- based denoising free generation tasks For free-generation tasks, unlike translation, we use iterative mask prediction instead of CTC for the

---

better performance with slight sacrifice in speed. Using CTC with 4-step decoding can outperform 8-step decoding both in performance and speed. However, using CTC aligmment alone will lead to inferior performance (O-step decoding) because of the multi-modality problem.
9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLE) pages 1538- 1548, Hong Kong, China. Association for Computa- tional Linguistics.
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong. Dongdong Zhang, Jia Pan, Wenping Wang. and Furu Wei. 2021. Zero-shot cross- cross-lingual transfer of neu- machine translation with multilingual pretrained encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 15-26.
6 Conclusion
In this paper, we propose an effective approach adapt existing pre-trained multilingual under- standing models to multilingual generators. On translation tasks, experiments demonstrated that our proposed method achieves large performance improvements and notable accelcration with strong cross-lingual generation ability. On tree-generation tasks including question generation and story gen- eration, our method also achieves comparable per- formance with AT-based method with impressive speedups. Although still lagging behind pretrained multilingual AT models (c.g., mBART) in super- vised tieettnnig settings in translation, our pro- posed method show better zero-shot abilities and faster inference.
Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang. Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li, 2022. Mtg: A benchmark suite for multilingual text generation. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 2508- 2527.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal. Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
7 Limitations
Although our proposed method has achieved no- table speedups and performance improvements in the multilingual setting, we still lag behind in bilin- gual translation, especially in high-resource scenar- ios. In addition, there still remains a gap between NAR pre-trained models and AR pre-trained mod- els. Generally, XLM-R with fine-tuning is largely inferior to mBART fine-tuning. Despite the gap can be largely reduced with our method, the gap still exists. In future work, we would like explore pre- training methods to make pretrained multilingual NAR models better generators.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Asa Cooper Stickland, Xian Li, and Marjan Ghazvinine- jad. 2021. Recipes for adapting pre-trained monolin- gual and multilingual models to machine translation. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 3440-3453, Online. Association for Computational Linguisties.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ankur Bapna and Orhan Firat, 2019. Simple, seal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the

---

Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order- agnostic cross entropy for non-autoregressive ma- chine translation. In International Conference on Machine Learning, pages 2849-2859. PMLR.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. 2022a. Constrained sampling from language models via langevin dynamics in embedding spaces. arXiv e-prints, pages arXiv-2205
Angela Fan, Shruti Bhosale, Holger Schwenk. Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english- mul- tilingual machine translation. J. Mach. Learn. Res., 22(107):
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. 2022b. Gradient-based constrained sampling from language models. ArXiv preprint, abs/2205. 12558.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non con-eutoreeressivv neural se- quence modeling by iterative refinement. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1 1173-1 182, Brussels, Belgium. Association for Computational Linguisties.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515-3523. PMLR.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-pred fask-cte Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNN-P) pages 6112- 6121, Hong Kong, China. Association for Computa- tional Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. 2022. Diffuseq: Sequence to se- quence text generation with diffusion models. ArXiv preprint, abs/2210.08933.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto. 2022. Diffusion- Im improves controllable text generation. ArXiv preprint, abs/2205.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang. Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang. Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark datasetfor cross-lingual pre-training, un- derstanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008 6018, Online. Association for Computational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Vietor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In 6th International Con- ference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jindřich I Libovický and Jindřich Heicl. 2018. End-to- end non-autoregressive neural machine translation with connectionist temporal classification. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3016- 3021, Brussels, Belgium. Association for Computa- tional Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PML.R
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- rion Branches Ouf, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Xiao Shi Huang, Felipe Perez, and Maksims Volkows. 2021. Improving aut-eettreeresssvve translation mod- els without distillation. In International Conference on Learning Representations.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du Zhilin Yang, and Jie Tang. 2022. P-tuingg:

---

Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages I 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke 7ettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- rics, 8:726-742.
Ye Qi. Devendra Sachan, Matthicu Felix, Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neu- ral machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), New Orleans, Louisiana. Associa- tion for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv preprint, abs/1907.11em
pages
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang. Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1993-2003, Online. Association for Computational Linguistics.
Shuming Ma. Li Dong, Shaohan Huang. Dongdong Zhang. Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. arXiv e-prints, pages ar arXiv- 2106.
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. COLD decoding: Energy-based constrained text generation with langevin dynamics. CoRR, abs/2202. 1 1705.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Kirk-Kirkpa 2022. Mix and match: Learning- free controllable text generationusing energy lan- guage models. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 40) 401--15..
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive ma- chine translation with latent aligmments. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNIP)), P), pages 1098- 11081 Online. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 48-53.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski. Erich Elsen, and Aaron van den Oord. 2021. Step- unrolled denoising autoencoders for text generation. In International Conference on Learning Representa- tions.
Xuan Ouyang, Shuohuan Wang, Chao Pang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie- m: Enhanced multilingual representation by aligning cross- lingual semanties with monolingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 27- 38.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. ArXiv preprins, abs/2211.05100
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yixuan Su, Deng Cai, Yan Wang. David Vandyke, Si- mon Baker, Piji Li, and Nigel Collier. 2021. Non- autoregressive text generation with pre-trained lan- guage models. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 234- 243.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meering of the Association for Computational Linguistics, pages 4996-5001, Flo- rence, Italy. Association for Computational Linguis- tics.
Zhixing Tan, Jiacheng Zhang, Xuancheng Huang, Gang Chen, Shuo Wang, Maosong Sun, Huanbo Luan, and Yang Liu. 2020. Thumt: an open-souree toolkit for neural machine translation. In Proceedings of the 14th Conference of the Association for Machine

---

Translation the Americas (Volume 1: Research Track), pages 116-122.
Zhixing Tan, Xiangwen Zhang. Shuo Wang, and Yang Liu. 2022. Msp: Multi-stage prompting for making pre-trained language models better translators. In Pruceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 6131-6142
Ahmet Üstün, Alexandre Bérard, Laurent Besacier, and Matthias Gallé. 2021. Multilingual unsupervised neural machine translation with denoising adapters. Proceedings the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6650-6662.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages
Alex Wang and Kyunghyun Cho. 2019. BERT has mouth, and must speak: BERT as a Markov ran- dom field language model. In Proceedings of the Workshop Methods for Optimizing and Evaluat- ing Neural Language Generation, pages 90-36, Min- neapolis, Minnesota. Association for Computational Linguistics.
Haoran Xu, Benjamin Van Durme, and Kenton Mur- ray. 2021a, BERT, mBERT, or BiBERT? a study on contextualized embeddings for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 6663-6677 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021b. Vocabulary learning via optimal transport for neural machine translation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Natural Language Processing (Volume 1: Long Papers), pages 7361-7373, Online. Association for Computational Linguistics.
Joint Conference
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.

---

A
Language Code References
We provide the list languages and corresponding language codes used in our experiments in Table 5.
Name
Arabic
German
Spanish
French
Code
Ar
Es
Fr
Name Hebrew
Italian
Romanian Russian
Code
He
Ro
Ru
Name Turkish
Vietnamese
Kazakh
Belarusian
Code
Tr
Vi
Kk
Be
Name
Basque
Malay
Bosmian
Code
Eu
Ms
Bs
Table 5: Full names corresponding codes of
guages used in our experiments.
B Details of TED Dataset
Table 6 presents a rough statistics number of the chosen 15 languages in our main experiment.
Name
Ar
De
Es
He
Num.
211k
165k
193k
189k
208k
Name
It
Ro
Ru
Tr
Vi
Num.
201k
178k
205k
180k
169k
Name
Kk
Be
Eu
Ms
Bs
Num.
3,234
4,392
5,094
5,104
5,566
Table 6: A rough statistics of the chosen 15 languages (10 for supervised setting and 5 for zero-shot cross- lingual setting) for the number of samples TED datasct.


Reviews:
[{"review_id": "YnbKZOx8pH", "reviewer": "Reviewer_5rFu", "summary": "", "questions": "First see the \"reasons to reject\" section.\n\nA. Line 76. What does \"complicated\" mean right here? If there is a comparison of difficulty, please mention machine translation is complicated than what task. \n\nB. What is AT and NAT abbreviation for? Should it be AR and NAR?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable suggestions. Thanks for pointing out the typos and writing issues, and we will fix them accordingly.\n\n>**Weakness 1-1: This paper misses some important work in their discussion, e.g., BiBERT**\n\nThanks for pointing out BiBERT.  Although BiBERT also utilizes pre-trained encoders for generation, it is quite different from our works from the following perspectives. \n\n(a) BiBERT utilizes pre-trained encoders by initializing Transformer models with parameters of a bilingual BERT, while our work focuses on effective adaptation methods on pre-trained encoders. \n\n(b) BiBERT is still an autoregressive translation model, while we explore the non-autoregressive nature of pre-trained encoders. \n\n(c) BiBERT focuses on a bilingual setting, while our method targets on multilingual setting.\n\nTherefore, we do not include BiBERT in our experimental baselines in this version. We are glad to add this discussion about BiBERT in our next revision.\n\n>**Weakness 1-2: The authors should compare to doing prompt tuning on mBART as well to make it a fair comparison between the number of trained parameters.**\n\nFor a more comprehensive comparison with mBART, we agree that incorporating a baseline that employs prompt tuning on mBART would be beneficial, and we intend to include it in the next revision.\n\nIn this revision, the comparison between mBART with our model focuses solely on speed, which is a fair comparison.  The speed improvement is consistent since prompt tuning does not change the auto-regressive decoding nature of mBART.\n\n>**Weakness 2-1: Motivation Unclear. Works have demonstrated that standard encoder-decoder models can also be fast and comparable in performance by using a large encoder and a small decoder.**\n\nAlthough shallow decoder models can accelerate inference by reducing the decoding layers, the acceleration is orthogonal to the decoding strategy (AR or NAR). Works have pointed out that NAR methods could also benefit from decreasing the decoding layers [1].\n\n>**Weakness 2-2: The reviewer wants to see a more motivating argument for studying how to adapt encoder models for generation aside from performance and efficiency.**\n\nApart from performance and efficiency, our other motivation is the adaptation of foundation models. \n\nHow to adapt existing foundation models for general usage including translation is an important research direction recently. \n\nMany works have been devoted to adapting generative models to understanding tasks, but how to adapt an understanding model to generative tasks remains unexplored. Given that XLM-R has already been scaled up to 11B, an xx-large version, we would like to explore the possibility of exploring the general usage of understanding models apart from understanding tasks.\n\n>**Weakness 3-1: The paper is not very well written. It does not have a clear distinction between model architecture (encoder, decoder, enc-dec) and training objective (autoregressive, auto-encoding).**\n\nThanks for pointing out the writing issues, and we will fix them in the revision.\n\n>**Weakness 3-2: Encoder-decoder models can generate with permuted orders.**\n\nThe generation ability of decoder models or encoder-decoder models are limited in infilling tasks with strict length requirement, e.g. filling blanks with a restricted number of words or writing acrostics. \n\nEncoder-based models can naturally cope with tasks by pre-defining a given number of mask tokens, but autoregressive models (decoder only and enc-dec) are known to be weak in controlling sequence lengths without massive instruction tuning.\n\n>**Question A: Line 76. What does \"complicated\" mean right here? If there is a comparison of difficulty, please mention machine translation is more complicated than what task.**\n\nBy \"complicated\", we meant to clarify that existing learning-free methods perform inferior in generation tasks that require strict semantic alignment as well as word alignment and high generation quality, i.e. machine translation. \n\nCurrent learning-free or energy-based models, e.g. Mix and Match[2], are only proved effective on free generation tasks that only require semantic alignment, e.g. controllable debiasing. We will change \"complicated\" to a more appropriate word in future revisions.\n\n>**Question B: What is AT and NAT abbreviation for? Should it be AR and NAR?**\n\nIn our paper, we use AT to represent autoregressive translation, while NAT is for non-autoregressive translation. AR and NAR are decoding strategies.\n\n[1] Non-Autoregressive Neural Machine Translation: A Call for Clarity (Schmidt et al., EMNLP2022)\n\n[2] Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models(Mireshghallah et al., ACL2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem that this paper studies is important: how to modify encoder only models for generations as the non-autoregressive pre-train objective is hard to adapt for generation. The method is simple and generalizable to various tasks since we only need to train a few set of soft prompts.\n\n2. The experiments show that their method can outperform simply initializing the encoder with a pre-trained model, direct training a encoder-decoder model on the task albeit training on much less parameters due to the parameter efficient fine-tuning. \n\n3. The generation speed is accelerated because compared to decoder only models which needs to pass the generated output many times through the model, encoder only model generates multiple tokens simultaneously and iteratively denoises them. ", "reasons_to_reject": "1. This paper misses some important work in their discussion: The authors fail to compare with or at least cite some important work that utilizes pre-trained encoders for generation. e.g. BiBERT[1], which is the current state-of-the art on IWSLT 14 De-En translation. The authors also compare to mBART and show that although their method underperforms, they win in generation speed and trained parameters. I want to say that they should compare to doing prompt tuning on mBART as well to make it a fair comparison between number of trained parameters. If it outperforms, then it strengthens their claim. \n\n2. Motivation Unclear: The authors mention that an important reason to use encoders for generation is because of their speed. However, works [2] have demonstrated that standard encoder-decoder models can also be fast and comparable in performance by using a large encoder and a small decoder. On the other hand, if the argument is that encoder models excel in NLU tasks, then using Pattern Exploit Training ,encoder-decoder models can be comparable to standard encoder models in NLU tasks [3]. My point is, I want to see a more motivating argument for studying how to adapt encoder models for generation aside from performance and efficiency. \n \n3. The paper is not very well written. It does not have a clear distinction between model architecture (encoder, decoder, enc-dec) and training objective (autoregressive, auto-encoding). They use autoregressive (AR) and encoders interchangeably - which should be mentioned in the paper. Some of the facts need modifying: e.g. the authors mention that unlike AR models which can only generate from left to right, NAR models as the advantage of generating in arbitrary order. I don't think this is true since there has been work that generates with permuted orders with encoder-decoder models [4].\n\n\n[1] BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation (Xu et al., EMNLP 2021)\n\n[2] Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation (Kasai et al., ICLR 2021)\n\n[3] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., JMLR 2020)\n\n[4] GLM: General Language Model Pretraining with Autoregressive Blank Infilling (Du et al., ACL 2022) \n", "questions_to_authors": "First see the \"reasons to reject\" section.\n\nA. Line 76. What does \"complicated\" mean right here? If there is a comparison of difficulty, please mention machine translation is complicated than what task. \n\nB. What is AT and NAT abbreviation for? Should it be AR and NAR?\n", "paper_topic_and_main_contributions": "This paper proposes a three-stage approach to adapt multilingual encoders for generation tasks. During the first stage, the authors train soft prompts to allow a pre-trained encoder to extract representations; During the second stage, the authors train another series of soft prompts for the same encoder model to do alignment to the target sequence; During the third stage, train another series of soft prompts that iteratively refines the aligned sequence to the target sequence. Experiments on Machine Translation, Question Generation and Story Generation show that their method outperforms training decoder-only models from scratch and and naive soft prompt tuning on multilingual GPT models. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See \"the reasons to reject\" section."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "73: surprising latency -> surprisingly high latency\n\n144: with only -> with only training on \n\n146: tge -> the\n\n151: prompting based -> prompt tuning\n\n217 Abbreviation \"SGA\" first introduced here without any introduction before\n"}}, {"review_id": "W5AIXGIKLk", "reviewer": "Reviewer_qScF", "summary": "", "questions": "1) Why are the results of mTransformer on the same language direction different between Table 1 and Table 2? Why are the parameters of mTransformer not labeled in Table 1? \n2) The author seems to have not introduced how to deal with the problem of inconsistent input and output lengths, such as how to generate a target sentence shorter or longer than the source sentence?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review, and we appreciate your approval of our work.\n\n>**Question 1: Why are the results of mTransformer on the same language direction different between Table 1 and Table 2? Why are the parameters of mTransformer not labeled in Table 1?**\n\nBecause of the training setting, which are different in Table 1 and Table 2. \n\nIn Table 1, we use different soft prompts for each language direction, which represents bi-lingual translation results. \n\nIn Table 2, we focus more on the zero-shot cross-lingual ability, therefore we use a multilingual version of the soft prompt, i.e. the soft prompt is shared across all X $\\rightarrow$ En directions.\n\nWe do not label the parameters of mTransformer in Table 1, only label the number of tuned parameters in each baseline.\n\nBecause our method belongs to the adaption setting. mTransformer and XLM-R are pre-trained models and none of mTransformer parameters are fine-tuned in each language direction.\n\n>**Question 2: How to generate a target sentence shorter or longer than the source sentence?**\n\nFor generating a target sentence with inconsistent length with the source sentence, in our experiments, we follow the classical approach introduced in CTC[1]. In detail, we up-sample the length of the source sentence by a factor of two and use CTC loss to convert the optimization of token distributions to path distributions. Thanks for pointing out the writing issue. We will add a more detailed description in our revision.\n\n>**Weakness 2: It is not clear whether the authors will release their codes and models.**\n\nWe will release the code and the models.\n\n[1] End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification (Jindřich Libovický, Jindřich Helcl, EMNLP2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a novel approach to transform non-autoregressive generation into a step-by-step denoising process, utilizing the denoising ability of a multilingual pre-trained understanding model for implementation.\n\n2. Authors adopt prompt-based approaches to achieve high parameter-efficiency during semantic alignment and denoising procedures.\n\n3. The work is solid. Experiments demonstrate that the proposed method can significantly improve the performance of non-autoregressive generation with only a small number of parameters required. Additionally, this method can be extended to many other generation tasks and has better zero-shot knowledge transfer capability.", "reasons_to_reject": "1. Some parts are not very clear, which caused me confusion:\n  a) Why are the results of mTransformer on the same language direction different between Table 1 and Table 2? Why are the parameters of mTransformer not labeled in Table 1? \n  b) The author seems to have not introduced how to deal with the problem of inconsistent input and output lengths, such as how to generate a target sentence shorter or longer than the source sentence?\n\n2. It is not clear whether authors will release their codes and models.", "questions_to_authors": "1) Why are the results of mTransformer on the same language direction different between Table 1 and Table 2? Why are the parameters of mTransformer not labeled in Table 1? \n2) The author seems to have not introduced how to deal with the problem of inconsistent input and output lengths, such as how to generate a target sentence shorter or longer than the source sentence?", "paper_topic_and_main_contributions": "There are significant advantages to non-autoregressive generation in terms of both generation efficiency and flexibility. This paper is focused on proposing an efficient adaptation method to improve the generation quality of multilingual understanding models in a parameter-efficient way. The paper also presents a diffusion-like approach -- semantic-guided denoiser to further enhance the generation quality. Extensive experiments demonstrate that their proposed method outperforms the traditional methods by a large margin. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 271: Appendix -> Section"}, "extra_text": {}}, {"review_id": "IcZIUm8gsd", "reviewer": "Reviewer_Ne93", "summary": "", "questions": "Questions:\n\n(1) I'm not an expert in machine translation, do you know why the BLEU score pf X->en is much better than en->X for most languages (in Table 1)? any intuitive explanation for this?\n\n(2) in Table 1, row 2, Transformer (NAT), which model do you refer to? I think there're so many NAT models. Is this NAT model STOA?\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and we appreciate your approval of our work.\n\n>**Question 1: Why the BLEU score pf X->en is much better than en->X for most languages (in Table 1)?**\n\nThe generation ability of XLM-R is consistent with the understanding ability of XLM-R.\n\nOn understanding tasks, XLM-R also presents strong ability in English, Spanish, and French,  while inferior performance in other languages. Therefore, adapting XLM-R to English/Spanish/French decoders often yields better performance than adapting XLM-R to other language decoders.\n\n>**Question 2: Is the NAT model in Table1, row 2 STOA?**\n\nThe NAT model presented in Table 1 is introduced in [1]. It is not the SOTA model but a classical one. \n\nIn this paper, we intend to provide an efficient PLM adaptation method for pre-trained multilingual encoders, while previous NAT works focus on bilingual settings that are trained from scratch. \n\nTherefore we do not include other NAT models but only a representative baseline in Table 1.\n\n\n[1] Non-Autoregressive Neural Machine Translation (Gu et al., ICLR 2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper tackle an interesting topic and present a good solution to it, the experimental results look like strong.", "reasons_to_reject": "I didn't find any obvious reasons to reject this paper directly.", "questions_to_authors": "Questions:\n\n(1) I'm not an expert in machine translation, do you know why the BLEU score pf X->en is much better than en->X for most languages (in Table 1)? any intuitive explanation for this?\n\n(2) in Table 1, row 2, Transformer (NAT), which model do you refer to? I think there're so many NAT models. Is this NAT model STOA?\n", "paper_topic_and_main_contributions": "This paper explores methods to empower multilingual understanding models (encoder) the generation abilities to get a unified model.\n\nSpecifically, they started from a multilingual encoder (XLM-R) and propose a Semantic-Guided Alignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters. Experimental results on various tasks including machine translation, question generation demonstrate the effectiveness of their proposed method.\n\nExcept the above results, they also observed that an interesting phenomenon: XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explore to transform a encoder only model (in this case an XLM-R) into a generator model. This aims to use these models multilingual encoding capabilities. It is done with a “Semantic-Guided Alignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator”. This allows the model to use a small amount of new parameters for this adaption. The reviewers agree that the tackled problems is relevant to the field, and that the proposal is exciting. The main issues found where related with a unclear writing and some terminology confusions. Also the related work should be improved.
Decision: Accept-Findings