SlowGuess/ACL_2020_OCR huggingface dataset:
Arya D. McCarthy\* and Xian Li and Jiatao Gu and Ning Dong
$\spadesuit$ Johns Hopkins University
arya@jhu.edu, {xianl,jgu,dnn}  $@$  fb.com
This paper proposes a simple and effective approach to address the problem of posterior collapse in conditional variational autoencoders (CVAEs). It thus improves performance of machine translation models that use noisy or monolingual data, as well as in conventional settings. Extending Transformer and conditional VAEs, our proposed latent variable model measurably prevents posterior collapse by (1) using a modified evidence lower bound (ELBO) objective which promotes mutual information between the latent variable and the target, and (2) guiding the latent variable with an auxiliary bag-of-words prediction task. As a result, the proposed model yields improved translation quality compared to existing variational NMT models on WMT  $\mathrm{Ro}\leftrightarrow \mathrm{En}$  and  $\mathrm{De}\leftrightarrow \mathrm{En}$ . With latent variables being effectively utilized, our model demonstrates improved robustness over non-latent Transformer in handling uncertainty: exploiting noisy source-side monolingual data (up to  $+3.2$  BLEU), and training with weakly aligned web-mined parallel data (up to  $+4.7$  BLEU).
The conditional variational autoencoder (CVAE; Sohn et al., 2015) is a conditional generative model for structured prediction tasks like machine translation. This model, learned by variational Bayesian methods (Kingma and Welling, 2014), can capture global signal about the target in its latent variables. Unfortunately, variational inference for text generation often yields models that ignore their latent variables (Bowman et al., 2016), a phenomenon called posterior collapse.
In this paper, we introduce a new loss function for CVAEs that counteracts posterior collapse, motivated by our analysis of CVAE's evidence lower bound objective (ELBO). Our analysis (§2)
reveals that optimizing ELBO's second term not only brings the variational posterior approximation closer to the prior, but also decreases mutual information between latent variables and observed data. Based on this insight, we modify CVAE's ELBO in two ways (§3): (1) We explicitly add a principled mutual information term back into the training objective, and (2) we use a factorized decoder (Chen et al., 2017), which also predicts the target bag-of-words as an auxiliary decoding distribution to regularize our latent variables. Our objective is effective even without Kullback-Leibler term (KL) annealing (Bowman et al., 2016), a strategy for iteratively altering ELBO over the course of training to avoid posterior collapse.
In applying our method to neural machine translation (NMT; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014), we find that we have measurably mitigated posterior collapse. The latent variables are not ignored, even in the presence of a powerful Transformer decoder. By addressing this problem, the resulting NMT model has improved robustness and performance in low-resource scenarios. Noisy data like those scraped from the Internet (Smith et al., 2013; Michel and Neubig, 2018) present a challenge for NMT (Khayrallah and Koehn, 2018; Ott et al., 2018a); we are measurably more able to model this extrinsic uncertainty than the (non-latent) Transformer (Vaswani et al., 2017) or existing variational NMT with the CVAE architecture (Zhang et al., 2016). Finally, we extend the model to semi-supervised learning (Cheng et al., 2016) to more effectively learn from monolingual data.
In summary, our conditional text generation model overcomes posterior collapse by promoting mutual information. It can easily and successfully integrate noisy and monolingual data, and it does this without the cost of lower BLEU score than non-latent NMT in typical settings.
8512
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8512-8525 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2 Formalism and Mathematical Analysis
Here we review the standard framework for neural MT. Next, we connect this to the conditional variational autoencoder, a model with latent random variables whose distributions are learned by black-box variational Bayesian inference. Finally, we analyze the CVAE's objective to explain why these models will ignore their latent variables ("posterior collapse").
2.1 Neural Machine Translation
Problem instances in machine translation are pairs of sequences  $(\pmb{x} \triangleq [x_1, \dots, x_m], \pmb{y} \triangleq [y_1, \dots, y_n])$ , where  $\pmb{x}$  and  $\pmb{y}$  represent the source and target sentences, respectively. Conventionally, a neural machine translation model is a parameterized conditional distribution whose likelihood factors in an autoregressive fashion:
$$
p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid \boldsymbol {x}) = \prod_ {t = 1} ^ {n} p _ {\boldsymbol {\theta}} \left(y _ {t} \mid \boldsymbol {x}, \boldsymbol {y} _ {<   t}\right). \tag {1}
$$
The dominant translation paradigm first represents the source sentence as a sequence of contextualized vectors (using the encoder), then decodes this representation into a target hypothesis according to Equation 1. The parameters  $\theta$  are learned by optimizing the log-likelihood of training pairs with stochastic gradient methods (Bottou and Cun, 2004; Kingma and Ba, 2015). Decoding is deterministic, using an efficient approximate search like beam search (Tillmann and Ney, 2003). The Transformer architecture with multi-head attention has become the state of the art for NMT (Vaswani et al., 2017).
2.2 The Conditional Variational Autoencoder
Our NMT approach extends the conditional variational autoencoder (Sohn et al., 2015), which we identify as a generalization of Variational NMT (Zhang et al., 2016). It introduces a latent random variable  $z$  into the standard NMT conditional distribution from Equation  $1:^{1,2}$
$$
p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid \boldsymbol {x}) = \int_ {z} \underbrace {p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid z , \boldsymbol {x})} _ {\text {d e c o d e r}} \cdot \underbrace {p _ {\boldsymbol {\theta}} (\boldsymbol {z} \mid \boldsymbol {x})} _ {\text {e n c o d e r}} \mathrm {d} z. \tag {2}
$$
For a given source sentence  $\pmb{x}$ , first a latent variable  $\pmb{z}$  is sampled from the encoder, then the target sen
tence  $\pmb{y}$  is generated by the decoder:  $z\sim p_{\theta}(z\mid$ $\pmb {x}),\pmb {y}\sim p_{\theta}(\pmb {y}\mid \pmb {z},\pmb {x})$  3
It is intractable to marginalize Equation 2 over  $\mathcal{Z}$ . Instead, the CVAE training objective is a variational lower bound (the ELBO) of the conditional log-likelihood. It relies on a parametric approximation of the model posterior:  $q_{\phi}(z\mid x,y)$ . The variational family we choose for  $q$  is a neural network whose parameters  $\phi$  are shared (i.e., amortized) across the dataset.
The ELBO lower-bounds the log-likelihood, as can be proven with Jensen's inequality. Its form is:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C V A E}} = \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {y})} [ \log p _ {\boldsymbol {\theta}} (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {z}) ] \\ - \mathrm {D} _ {\mathrm {K L}} \left(q _ {\phi} (\boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {y}) \| p _ {\boldsymbol {\theta}} (\boldsymbol {z} \mid \boldsymbol {x})\right), \tag {3} \\ \end{array}
$$
where  $\mathrm{D}_{\mathrm{KL}}$  represents the Kullback-Leibler divergence between two distributions.
We use amortized variational inference to simultaneously perform learning and approximate posterior inference, updating both  $\theta$  and  $\phi$  with stochastic gradient methods. Improving  $\theta$  raises the lower bound, and improving  $\phi$  keeps the bound tight with respect to the model conditional log-likelihood. The same argument pertains to the joint maximization interpretation of the expectation-maximization (EM) algorithm (Neal and Hinton, 1998). (Our optimization is a variational generalization of EM.)
2.3 Posterior Collapse
Despite their success when applied to computer vision tasks, variational autoencoders in natural language generation suffer from posterior collapse, where the learnt latent code is ignored by a strong autoregressive decoder. This presents a challenge to conditional language generation tasks in NLP like machine translation.
The phenomenon can be explained mathematically by an analysis of the ELBO objective, as well as from the perspective of a powerful decoder that can model the true distribution without needing the latent code. We consider both in this subsection.
ELBO surgery Recall that the computed objective approximates the objective on the true data distribution  $p_{\mathcal{D}}$ , using a finite number of samples
3The sense of "encoder" in the context of variational autoencoders differs from the typical sense in neural machine translation, such that the NMT encoder is a component of both the VAE's encoder and decoder. We can separate these by computing a second, deterministic latent variable  $h$  from  $x$  to represent the NMT encoder outputs, used by both the VAE encoder and the NMT/VAE decoder.
<sup>1</sup>By contrast, the hidden states of a standard sequence-to-sequence model are deterministic latent variables. <sup>2</sup>In Equation 2 we assume a continuous latent variable. For the discrete case, replace integration with summation.
8513
Figure 1: Model architecture in training (with parallel data) and inference.
(see, e.g., Brown et al., 1992):
$$
\mathcal {L} = \mathbb {E} _ {p _ {\mathcal {D}} (\boldsymbol {x}, \boldsymbol {y})} \left[ \mathcal {L} _ {\mathrm {C V A E}} \left(\phi , \boldsymbol {\theta}; \boldsymbol {x}, \boldsymbol {y}\right) \right]. \tag {4}
$$
We can factor the KL term of Equation 3 (omitting parameter subscripts) as:
$$
\begin{array}{l} \mathbb {E} _ {p _ {\mathcal {D}} (\boldsymbol {x}, \boldsymbol {y})} [ \mathrm {D} _ {\mathrm {K L}} (q (\boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {y}) \| p (\boldsymbol {z} \mid \boldsymbol {x})) ] \\ = \underbrace {\operatorname {H} (\boldsymbol {x} , \boldsymbol {y}) - \operatorname {H} (\boldsymbol {x} , \boldsymbol {y} \mid \boldsymbol {z})} _ {\triangleq \mathrm {I} _ {q _ {\phi}} (\boldsymbol {z}; \boldsymbol {x}, \boldsymbol {y})} + \underbrace {\mathbb {E} _ {q (\boldsymbol {z})} \log \frac {q (\boldsymbol {z})}{p (\boldsymbol {z})}} _ {\triangleq \mathrm {D} _ {\mathrm {K L}} (q _ {\phi} (\boldsymbol {z}) \| p (\boldsymbol {z}))}, \tag {5} \\ \end{array}
$$
which we prove in Appendix A, following (Hoffman and Johnson, 2016).
As both the resulting mutual information and KL terms are non-negative (Cover and Thomas, 2006), the global minimum of Equation 5 is  $\mathrm{I}_{q_{\phi}}(z; x, y) = \mathrm{D}_{\mathrm{KL}}(q_{\phi}(z) \| p(z)) = 0$ . Unfortunately, at this point, the consequence of the optimization is that the latent variable  $z$  is conditionally independent of the data  $(x, y)$ .
A powerful decoder Revisiting Equation 3, we see that the decoder is conditioned on both the stochastic latent variable  $z$  and the source text  $x$ . A sufficiently high-capacity autoregressive decoder can model the conditional density directly, ignoring the latent variable and reducing inference to Equation 1. The KL term can then be reduced to its minimum (0) by equating the posterior to the prior. To prevent this, some work weakens the decoder in various ways. This is a challenge, because NMT requires a powerful decoder such as Transformer with direct attention to the encoder.
3 An Information-Infused Objective
We modify our training objective to explicitly retain mutual information between the latent variable  $z$  and the observation  $(x, y)$ . Further, we use an auxiliary decoder that only uses the latent variable, not the encoder states. We combine it with the existing decoder as a mixture of softmaxes (Yang et al., 2018a). The model is trained with amortized variational inference. When source-language monolingual text is available, we augment our modified CVAE objective with a similarly modified (non-conditional) VAE objective. The training and inference strategy is summarized in Figure 1.
3.1 Adding  $\mathrm{I}_{q_{\phi}}(z; x, y)$  to ELBO
To combat the optimization dilemma from Equation 5 (namely, that the objective discourages mutual information between the latent variable and the data), we explicitly add the mutual information term to the CVAE's ELBO and obtain a new training objective:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {M I C V A E}} = \mathcal {L} _ {\mathrm {C V A E}} + \mathrm {I} _ {q _ {\phi}} (\boldsymbol {z}; \boldsymbol {x}, \boldsymbol {y}) \\ = \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {y})} \log p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {z}) \\ - \mathrm {D} _ {\mathrm {K L}} \left(q _ {\phi} (\boldsymbol {z}) \| p (\boldsymbol {z})\right) \tag {6} \\ \end{array}
$$
The new training objective  $\mathcal{L}_{\mathrm{MICVAE}}$  aims to match the aggregated approximate posterior distribution of the latent variable  $q_{\phi}(z)$  (Hoffman and Johnson, 2016) to the aggregated-posterior prior distribution  $p_{\theta}(z)$ .<sup>4</sup>
<sup>4</sup>It can be seen as extending InfoVAE (Zhao et al., 2019) to conditional generative models, where we have overcome
8514
3.2 Guiding  $z$  to Encode Global Information
Several existing approaches weaken the decoder: limiting its capacity to encourage latent variables to be utilized (Bowman et al., 2016; Gulrajani et al., 2017). Here we propose a different approach: explicitly guiding the information encoded in  $z$  without reducing the decoder's capacity.
The decision to weaken the decoder can be understood in the context of Bits-Back Coding theory (Chen et al., 2017), which suggests that at optimality the decoder will model whatever it can locally, and only the residual will be encoded in the latent variable  $z$ . A consequence is that explicit information placement can give more powerful latent representations.
Inspired by this Bits-Back perspective, we add a global auxiliary loss for  $\mathbf{z}$  to encode information which cannot be modelled locally by the autoregressive decoder  $\prod_{t} p_{\theta}(y_t \mid x, y_{<t}, z)$ . We use bag-of-words (BoW) prediction as the auxiliary loss. It encodes global information while having a non-autoregressive factorization:  $\prod_{t} p_{\psi}(y_t \mid z)$ . (We choose not to condition it on the source sentence  $\mathbf{x}$ .) Further, it requires no additional annotated data. The auxiliary decoder complements the autoregressive decoder (which is locally factorized), interpolating predictions at the softmax layer, i.e.  $p(y_t \mid x, y_{<t}, z)$  is a mixture of softmaxes (Yang et al., 2018b):
$$
\begin{array}{l} p \left(y _ {t} \mid \cdot\right) = (1 - \lambda) \cdot p _ {\boldsymbol {\theta}} \left(y _ {t} \mid \boldsymbol {x}, \boldsymbol {y} _ {<   t}, \boldsymbol {z}\right) \\ + \lambda \cdot p _ {\psi} \left(y _ {t} \mid z\right), \tag {7} \\ \end{array}
$$
with mixing parameter  $\lambda$ . (We use  $\lambda = 0.1$  in this paper.) Thus, the bag-of-words objective regularizes the log-likelihood bound.
4 Implementing Latent Variable NMT
Our model uses discrete latent variables. These are used to select a latent embedding, which is concatenated to the decoder state.
Inference Network We use discrete latent variables with reparameterization via Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) to allow backpropagation through discrete sampling. Unlike the multivariate Gaussian distribution commonly used in VAE and CVAE, our parameterization can explicitly account for multiple
the mismatch between the (joint) data distribution  $p_{\mathcal{D}}(\boldsymbol{x}, \boldsymbol{y})$  and the (conditional) likelihood objective  $p_{\boldsymbol{\theta}}(\boldsymbol{y} \mid \boldsymbol{x})$ .
modes in the data. (See Rezende and Mohamed (2015) for a perspective on the value of multimodal distributions over latent variables.) To make our model more general, we introduce a set of discrete latent variables  $z = \{z_{1},\dots ,z_{K}\}$  which are independently sampled from their own inference networks  $\Phi_k$ . Specifically, each  $\Phi_k$  computes scaled dot product attention with encoder outputs  $h\in \mathbb{R}^d$  using latent code embedding  $e_k$ :
$$
\begin{array}{l} \boldsymbol {C} _ {k} = \text {A t t e n t i o n} \left(\boldsymbol {e} _ {k} \boldsymbol {W} ^ {k}, \boldsymbol {h} \boldsymbol {W} ^ {h}, \boldsymbol {h} \boldsymbol {W} ^ {h}\right) \\ = \operatorname {S o f t m a x} \left(\frac {e _ {k} \boldsymbol {W} ^ {k} (\boldsymbol {h} \boldsymbol {W} ^ {h}) ^ {\top}}{\sqrt {d}}\right) \boldsymbol {h} \boldsymbol {W} ^ {h}. \tag {8} \\ \end{array}
$$
We can now sample  $z_{k}$  by the Gumbel-Softmax reparameterization trick (Maddison et al., 2017; Jang et al., 2017):
$$
\begin{array}{l} z _ {k} \sim \operatorname {G u m b e l S o f t m a x} \left(\mathbf {C} _ {k}\right) (9) \\ = \operatorname {S o f t m a x} \left(\frac {\mathbf {C} _ {k} + \boldsymbol {g}}{\tau}\right), (10) \\ \end{array}
$$
where  $g = -\log (-\log (u))$ ,  $u \sim$  Uniform is the Gumbel noise and  $\tau$  is a fixed temperature. (We use  $\tau = 1$  in this paper.) At inference time, we use a discrete version by directly sampling from the latent variable distribution.
BoW Auxiliary Decoder Given an inferred sample  $z \sim \Phi_k(h)$ , the BoW decoder predicts all tokens at once without considering their order. We compute the cross-entropy loss for the predicted tokens over the output vocabulary space  $V$ :
$$
\mathcal {L} _ {\mathrm {B o W}} = \sum_ {i = 1} ^ {| V |} p _ {i} \log \hat {p} _ {\psi} (y _ {i} \mid z), \quad \sum_ {i = 1} ^ {| V |} p _ {i} = 1. \tag {11}
$$
We take the (unnormized) empirical distribution  $\tilde{p}_i$  to be a token's frequency within a sentence normalized by its total frequency within a minibatch, mitigating the effect of frequent (stop) words. This is then normalized over the sentence to sum to 1, giving values  $p_i$ . The model distribution  $\hat{p}_{\psi}$  is computed by conditioning on the latent code only, without direct attention to encoder outputs. We use scaled dot-product attention between the latent embeddings and the target embeddings (each of dimensionality  $d$ , represented as a matrix  $E_V$ ):
$$
p _ {\psi} \left(y _ {i} \mid z\right) = \operatorname {S o f t m a x} \left(\frac {\mathbf {e} (z) E _ {V} ^ {\top}}{\sqrt {d}}\right) _ {i}. \tag {12}
$$
8515
Algorithm 1 Training Strategy
1:  $\Phi_{enc},\Phi_{k = 1,\dots ,K},\Theta_{dec},\Theta_{BoW}\gets$  init.
2: while  $\Theta_{enc},\Theta_{dec},\Theta_{BoW},\Phi_{k = 1,\dots ,K}$  have not converged do
3: Sample  $(\mathbf{x},\mathbf{y})$  from  $D^{\mathrm{bitext}}$
4: Compute  $\mathcal{L}_{\mathrm{MICVAE}}$  with Equation 6
5: Train  $\Phi_{enc},\Theta_{dec},\Phi_{k = 1,\dots ,K}$  with  $\mathcal{L}_{\mathrm{MICVAE}}$
6: Compute  $\mathcal{L}_{\mathrm{BoW}}$  with Equation 12
7: Train  $\Phi_{enc}, \Theta_{BoW}, \Phi_{k=1,\dots,K}$  with  $\mathcal{L}_{\mathrm{BoW}}$
8: if self_training then
9: Sample  $\mathbf{x}$  from  $D^{\mathrm{mono}}$
10: Compute  $\mathcal{L}_{\mathrm{Mono}}$  with Equation 13
11: Train  $\Phi_{enc},\Phi_{k = 1,\dots ,K}$  with  $\mathcal{L}_{\mathrm{Mono}}$
12: end if
13: end while
For training with parallel data, we optimize  $\mathcal{L}_{\mathrm{MICVAE}}$ . We draw samples  $z$  from the approximate posterior  $q_{\phi}(z \mid x, y)$  parameterized by the inference network, then feed the samples to both the autoregressive and auxiliary (BoW) decoders to get a Monte Carlo estimate of the gradient.
Estimating aggregated distributions We estimate  $p_{\theta}(z)$  and  $q_{\phi}(z)$  over each minibatch, following Zhao et al. (2018).
Semi-supervised learning We apply the same modification to VAE's ELBO, following Zhao et al. (2019). For jointly training with source-side monolingual data, we add  $\mathrm{I}_{q_{\phi}}(z; x)$  to the ELBO, and for target-side monolingual data, we add  $\mathrm{I}_{q_{\phi}}(z; y)$ . The joint objective sums the modified CVAE and VAE objectives:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {M o n o}} = \log p (\boldsymbol {x} \mid \boldsymbol {z}) \\ + D _ {\mathrm {K L}} \left(\frac {1}{L} \sum_ {\ell = 1} ^ {L} q _ {\phi} \left(\boldsymbol {z} ^ {(\ell)} \mid \boldsymbol {x} ^ {(\ell)}\right) \left| \left| \frac {1}{L} \sum_ {\ell = 1} ^ {L} p \left(\boldsymbol {z} ^ {(\ell)}\right) \right. \right. \right. \tag {13} \\ \end{array}
$$
$$
\mathcal {L} _ {\text {J o i n t}} = \mathcal {L} _ {\text {M I C V A E}} + \mathcal {L} _ {\text {M o n o}}, \tag {14}
$$
where  $L$  is the number of monolingual examples. Algorithm 1 describes the overall training strategy.
Here we present empirical results on the Transformer architecture. We evaluate our model on four standard datasets and compare against three baselines. We use four measures to quantify posterior collapse, then examine translation quality (BLEU score) in standard fully supervised settings, a semi-supervised setting, and a fully supervised setting with noisy source text. Hyperparameters, regularization choices, and subword vocabulary information can be found in §5.3.
The results show that we have effectively addressed posterior collapse: latent variables are no longer ignored despite the presence of a powerful decoder. As a result, we outperform both the standard Transformer and the Transformer-based variational NMT approach, when using noisy data or source-language monolingual data.
First, we evaluate our models on a standard high-resource and low-resource benchmark dataset from WMT. Second, we focus on situations where noisy or monolingual data is available. We note that low-resource scenarios and noisy data are two representative challenges in MT (Lopez and Post, 2013).
WMT14 German-English We use data from the WMT14 news translation shared task, which has 3.9M sentence pairs for training with the same BPE tokenization as in Gu et al. (2018).
WMT16 Romanian-English We use data from the WMT16 news translation shared task. We use the same BPE-preprocessed (Sennrich et al., 2016b) train, dev and test splits as in Gu et al. (2018) with 608k sentence pairs for training.
FLORES Sinhala-English For this low-resource benchmark, we use the same preprocessed data as in Guzmán et al. (2019). There are 646k sentence pairs.
MT for Noisy Text (MTNT) French-English
This dataset pairs web-scraped text from Reddit with professional translations. We use 30k subword units built jointly from source and target sentences and only keep sentences with less than 100 tokens. For training, there are 34,380 sentence pairs for English-French and 17,616 sentence pairs for French-English (Michel and Neubig, 2018). We also used
5 Learning to copy the target text has proven useful for low-resource NMT (Currey et al., 2017).
18,676 monolingual sentences per language from the same data source (Reddit).
5.2 Baselines
We compare our model to three baselines:
Non-latent This is a standard Transformer model without latent variables.
VNMT A CVAE model with Gaussian distribution as proposed in Variational NMT by Zhang et al. (2016), which we reimplement using Transformer. (Zhang et al. (2016) use a GRU-based recurrent model.)
DCVAE A CVAE model with the same discrete latent variable parameterization as ours but without the new objective (i.e., the mutual information term and bag-of-words regularizer).