Question: What do they use to pre-train the encoder-decoder attention mechanism?
Evidence: The masked language modeling (MLM) BIBREF13 task, also known as the Cloze task BIBREF23, aims at predicting the randomly masked words according to their context. The objective pretrains the bidirectional encoder to obtain contextual representations. Following BIBREF13, we randomly mask 15% of the tokens in a monolingual sentence. For each masked token, we substitute it with a special token M, a random token, or the unchanged token with a probability of 0.8, 0.1, and 0.1, respectively. Let $x$ denote a sentence from the monolingual training corpus, and $M_{x}$ the set of randomly masked positions. The monolingual MLM loss is defined as: MLM(x) = -i Mxp( xi | xMx) where $x_{\setminus M_{x}}$ is the masked version of input $x$. Notice that language tags are fed into the model for all pre-training tasks.
We use the denoising auto-encoding (DAE) objective BIBREF24 to pretrain the encoder-decoder attention mechanism. Given sentence $x$ from the monolingual corpus, we use three types of noise to obtain the randomly perturbed text $\hat{x}$. First, the word order is locally shuffled. Second, we randomly drop tokens of the sentence with a probability of $0.1$. Third, we substitute tokens with the special padding token P with a probability of $0.1$. The pre-training objective is to recover the original sentence $x$ by conditioning on $\hat{x}$. The DAE loss is computed via: DAE(x) = -p(x|x) = -i = 1|x|p(xi | x, x<i) where $x_{<i}$ represents the tokens of previous time steps $x_1,\cdots ,x_{i-1}$.
Similar to monolingual MLM, the masked token prediction task can be extended to cross-lingual settings BIBREF5. To be specific, given a parallel corpus, we concatenate the pair of bilingual sentences $(x,y)$ to a whole sequence, and use it as the input of MLM. The language tags are also fed into the model to indicate the languages of tokens. During training, we adopt the same masking strategy as monolingual MLM. Apart from using monolingual context to predict the masked tokens, XMLM encourages the model to utilize the alignment of bilingual sentences, so that the model learns to map cross-lingual texts into a shared vector space. Similar to eq:mlm, the cross-lingual MLM loss is: XMLM(x,y) = -i Mxp( xi | xMx , yMy)
-i Myp( yi | xMx , yMy) where $M_x, M_y$ represent the masked positions of $x$ and $y$, respectively.
If only DAE is used as the pre-training task for the decoder, we found that the model ignores the target language tag while generating just the same language as the input, caused by the spurious correlation issue BIBREF25. In other words, the DAE loss captures the spurious correlation between the source language tag and the target sentences, but we expect the language of generated sentences can be controlled by the target language tag. To solve the above problem, we use machine translation as the cross-lingual auto-encoding (XAE) task, which decreases mutual information between the target sentences and the source language tag. XAE can be viewed as the multilingual-version DAE task in the sense that both of them recover the sentence by conditioning on the encoded representations. The cross-lingual auto-encoding loss is defined as: XAE(x,y) = -p(y|x) - p(x|y) where $(x,y)$ is a pair of sentences in the parallel corpus.
Denoising Auto-Encoding (DAE) We use the denoising auto-encoding (DAE) objective (Vincent et al. 2008) to pretrain the encoder-decoder attention mechanism.
Answer: The denoising auto-encoding (DAE) objective.