SlowGuess/ACL_2020_OCR huggingface dataset:
Tom Pelsmaeker
ILCC
t.l.pelsmaeker@sms.ed.ac.uk
Wilker Aziz
ILLC
w.aziz@uva.nl
Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.
Deep generative models (DGMs) are probabilistic latent variable models parameterised by neural networks (NNs). Specifically, DGMs optimised with amortised variational inference and reparameterised gradient estimates (Kingma and Welling, 2014; Rezende et al., 2014), better known as variational auto-encoders (VAEs), have spurred much interest in various domains, including computer vision and natural language processing (NLP).
In NLP, VAEs have been developed for word representation (Rios et al., 2018), morphological analysis (Zhou and Neubig, 2017), syntactic and
semantic parsing (Corro and Titov, 2018; Lyu and Titov, 2018), document modelling (Miao et al., 2016), summarisation (Miao and Blunsom, 2016), machine translation (Zhang et al., 2016; Schulz et al., 2018; Eikema and Aziz, 2019), language and vision (Pu et al., 2016; Wang et al., 2017), dialogue modelling (Wen et al., 2017; Serban et al., 2017), speech modelling (Fraccaro et al., 2016), and, of course, language modelling (Bowman et al., 2016; Goyal et al., 2017). One problem remains common to the majority of these models, VAEs often learn to ignore the latent variables.
We investigate this problem, dubbed posterior collapse, in the context of language models (LMs). In a deep generative LM (Bowman et al., 2016), sentences are generated conditioned on samples from a continuous latent space, an idea with various practical applications. For example, one can constrain this latent space to promote generalisations that are in line with linguistic knowledge and intuition (Xu and Durrett, 2018). This also allows for greater flexibility in how the model is used, for example, to generate sentences that live—in latent space—in a neighbourhood of a given observation (Bowman et al., 2016). Despite this potential, VAEs that employ strong generators (e.g. recurrent NNs) tend to ignore the latent variable. Figure 1 illustrates this point: neighbourhood in latent space does not correlate to patterns in data space, and the model behaves just like a standard LM.
Recently, many techniques have been proposed to address this problem (§3 and §7) and they range from modifications to the objective to changes to the actual model. Some of these techniques have only been tested under different conditions and under different evaluation criteria, and some of them have only been tested outside NLP. This paper contributes: (1) a novel strategy based on constrained optimisation towards a pre-specified upper-bound on mutual information; (2) multimodal priors that
Work done while the first author was at the University of Amsterdam. Code is available at https://github.com/tom-pelsmaeker/deep-generative-lm
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7220-7236 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
by design promote increased mutual information between data and latent code; last and, arguably most importantly, (3) a systematic comparison—in terms of resources dedicated to hyperparameter search and sensitivity to initial conditions—of strategies to counter posterior collapse, including some never tested for language models (e.g. InfoVAE, LagVAE, soft free-bits, and multimodal priors).
2 Density Estimation for Text
Density estimation for written text has a long history (Jelinek, 1980; Goodman, 2001), but in this work we concentrate on neural network models (Bengio et al., 2003), in particular, autoregressive ones (Mikolov et al., 2010). Following common practice, we model sentences independently, each a sequence  $x = \langle x_1, \ldots, x_n \rangle$  of  $n = |x|$  tokens.
2.1 Language models
A language model (LM) prescribes the generation of a sentence as a sequence of categorical draws parameterised in context, i.e.  $P(x|\theta) =$
$$
\prod_ {i = 1} ^ {| x |} P \left(x _ {i} \mid x _ {<   i}, \theta\right) = \prod_ {i = 1} ^ {| x |} \operatorname {C a t} \left(x _ {i} \mid f \left(x _ {<   i}; \theta\right)\right). \tag {1}
$$
To condition on all of the available context, a fixed NN  $f(\cdot)$  maps from a prefix sequence (denoted  $x_{<i}$ ) to the parameters of a categorical distribution over the vocabulary. We estimate the parameters  $\theta$  of the model by searching for a local optimum of the log-likelihood function  $\mathcal{L}(\theta) = \mathbb{E}_X[\log P(x|\theta)]$  via stochastic gradient-based optimisation (Robbins and Monro, 1951; Bottou and Cun, 2004), where the expectation is taken w.r.t. the true data distribution and approximated with samples  $x \sim \mathcal{D}$  from a data set of i.i.d. observations. Throughout, we refer to this model as RNNLM alluding to a particular choice of  $f(\cdot;\phi)$  that employs a recurrent neural network (Mikolov et al., 2010).
2.2 Deep generative language models
Bowman et al. (2016) model observations as draws from the marginal of a DGM. An NN maps from a latent sentence embedding  $z \in \mathbb{R}^{d_z}$  to a distribution  $P(x|z,\theta)$  over sentences,
$$
\begin{array}{l} P (x | \theta) = \int p (z) P (x | z, \theta) \mathrm {d} z \\ = \int \mathcal {N} (z | 0, I) \prod_ {i = 1} ^ {| x |} \operatorname {C a t} \left(x _ {i} \mid f \left(z, x _ {<   i}; \theta\right)\right) \mathrm {d} z, \tag {2} \\ \end{array}
$$
where  $z$  follows a standard Gaussian prior. Generation still happens one word at a time without Markov assumptions, but  $f(\cdot)$  now conditions on  $z$  in addition to the observed prefix. The conditional  $P(x|z,\theta)$  is commonly referred to as generator or decoder. The quantity  $P(x|\theta)$  is the marginal likelihood, essential for parameter estimation.
This model is trained to assign a high (marginal) probability to observations, much like standard LMs. Unlike standard LMs, it employs a latent space which can accommodate a low-dimensional manifold where discrete sentences are mapped to, via posterior inference  $p(z|x,\theta)$ , and from, via generation  $P(x|z,\theta)$ . This gives the model an explicit mechanism to exploit neighbourhood and smoothness in latent space to capture regularities in data space. For example, it may group sentences according to latent factors (e.g. lexical choices, syntactic complexity, etc.). It also gives users a mechanism to steer generation towards a specific purpose. For example, one may be interested in generating sentences that are mapped from the neighbourhood of another in latent space. To the extent this embedding space captures appreciable regularities, interest in this property is heightened.
Approximate inference Marginal inference for this model is intractable and calls for variational inference (VI; Jordan et al., 1999), whereby an auxiliary and independently parameterised model  $q(z|x,\lambda)$  approximates the true posterior  $p(z|x,\theta)$ . When this inference model is itself parameterised by a neural network, we have a case of amortised inference (Kingma and Welling, 2014; Rezende et al., 2014) and an instance of what is known as a VAE. Bowman et al. (2016) approach posterior inference with a Gaussian model
$$
\begin{array}{l} Z | \lambda , x \sim \mathcal {N} (\mathbf {u}, \operatorname {d i a g} (\mathbf {s} \odot \mathbf {s})) \tag {3} \\ [ \mathbf {u}, \mathbf {s} ] = g (x; \lambda) \\ \end{array}
$$
whose parameters, i.e. a location vector  $\mathbf{u} \in \mathbb{R}^D$  and a scale vector  $\mathbf{s} \in \mathbb{R}_{>0}^{D}$ , are predicted by a neural network architecture  $g(\cdot; \lambda)$  from an encoding of the complete observation  $x$ . In this work, we use a bidirectional recurrent encoder. Throughout the text we will refer to this model as SENVAE.
Parameter estimation We can jointly estimate the parameters of both models (i.e. generative and
<sup>1</sup>We use uppercase  $P(\cdot)$  for probability mass functions and lowercase  $p(\cdot)$  for probability density functions.
2We use boldface for deterministic vectors and  $\odot$  for elementwise multiplication.
7221
Decoding	Generated sentence
Greedy	The company said it expects to report net income of $UNK-NUM million
Sample	They are getting out of my own things?
IBM also said it will expect to take next year.
(a) Greedy generation from prior samples (top) yields the same sentence every time, showing that the latent code is ignored. Yet, ancestral sampling (bottom) produces good sentences, showing that the recurrent decoder learns about the structure of English sentences.
The two sides hadn’t met since Oct. 18.
I don’t know how much money will be involved.
The specific reason for gold is too painful.
The New Jersey Stock Exchange Composite Index gained 1 to 16.
And some of these concerns aren’t known.
Prices of high-yield corporate securities ended unchanged.
(b) Homotopy: ancestral samples mapped from points along a linear interpolation of two given sentences as represented in latent space. The sentences do not seem to exhibit any coherent relation, showing that the model does not exploit neighbourhood in latent space to capture regularities in data space.
Figure 1: Sentences generated from Bowman et al. (2016)'s VAE trained without special treatment.
inference) by locally maximising a lower-bound on the log-likelihood function (ELBO)
$$
\begin{array}{l} \mathcal {E} (\theta , \lambda) = \mathbb {E} _ {X} \left[ \mathbb {E} _ {q (z | x, \lambda)} [ \log P (x | z, \theta) ] \right. \\ \left. - \operatorname {K L} (q (z | x, \lambda) | | p (z)) \right]. \\ \end{array}
$$
For as long as we can reparameterise samples from  $q(z|x,\lambda)$  using a fixed random source, automatic differentiation (Baydin et al., 2018) can be used to obtain unbiased gradient estimates of the ELBO (Kingma and Welling, 2014; Rezende et al., 2014).
3 Posterior Collapse
In VI, we make inferences using an approximation  $q(z|x,\lambda)$  to the true posterior  $p(z|x,\theta)$  and choose  $\lambda$  as to minimise the KL divergence  $\mathbb{E}_X[\mathrm{KL}(q(z|x,\lambda)||p(z|x,\theta))]$ . The same principle yields a lower-bound on log-likelihood used to estimate  $\theta$  jointly with  $\lambda$ , thus making the true posterior  $p(z|x,\theta)$  a moving target. If the estimated conditional  $P(x|z,\theta)$  can be made independent of  $z$ , which in our case means relying exclusively on  $x_{<i}$  to predict the distribution of  $X_i$ , the true posterior will be independent of the data and equal to the prior. Based on such observation, Chen et al. (2017) argue that information that can be modelled by the generator without using latent variables will be modelled that way—precisely because when no information is encoded in the latent variable the true posterior equals the prior and it is then trivial to reduce  $\mathbb{E}_X[\mathrm{KL}(q(z|x,\lambda)||p(z|x,\theta))]$  to 0. This is typically diagnosed by noting that after training  $\mathrm{KL}(q(z|x,\lambda)||p(z))\to 0$  for most  $x$ : we say that the true posterior collapses to the prior. Alemi et al. (2018) show that the rate,  $R = \mathbb{E}_X[\mathrm{KL}(q(z|x,\lambda)||p(z))]$ , is an upperbound to  $I(X;Z|\lambda)$ , the mutual information (MI) between  $X$  and  $Z$ . Thus, if  $\mathrm{KL}(q(z|x,\lambda)||p(z))$  is
close to zero for most training instances, MI is either 0 or negligible. They also show that the distortion,  $D = -\mathbb{E}_X[\mathbb{E}_{q(z|x,\lambda)}[\log P(x|z,\theta)]]$ , relates to a lower-bound on MI (the lower-bound being  $H - D$ , where  $H$  is the unknown data entropy).
A generator that makes no Markov assumptions, such as a recurrent LM, can potentially achieve  $X_{i} \perp Z \mid x_{< i}, \theta$ , and indeed many have noticed that VAEs whose observation models are parameterised by such strong generators (or strong decoders) tend to ignore the latent representation (Bowman et al., 2016; Higgins et al., 2017; Sønderby et al., 2016; Zhao et al., 2018b). For this reason, a strategy to prevent posterior collapse is to weaken the decoder (Yang et al., 2017; Semeniuta et al., 2017; Park et al., 2018). In this work, we are interested in employing strong generators, thus we do not investigate weaker decoders. Other strategies involve changes to the optimisation procedure and manipulations to the objective that target local optima of the ELBO with non-negligible MI.
Annealing Bowman et al. (2016) propose "KL annealing", whereby the KL term in the ELBO is incorporated into the objective in gradual steps. This way the optimiser can focus on reducing distortion early on in training, potentially by increasing MI. They also propose to drop words from  $x_{<i}$  at random to weaken the decoder—intuitively the model would have to rely on  $z$  to compensate for missing history. We experiment with a slight modification of word dropout whereby we slowly vary the dropout rate from  $1 \rightarrow 0$ . In a sense, we "anneal" from a weak to a strong generator.
Targeting rates Another idea is to target a prespecified rate (Alemi et al., 2018). Kingma et al. (2016) replace the KL term in the ELBO with  $\max(r, \mathrm{KL}(q(z|x,\lambda)||p(z)))$ , dubbed free bits (FB) because it allows encoding the first
3This follows trivially from the definition of posterior:  $p(z|x) = \frac{p(z)P(x|z)}{P(x)}\stackrel {X\perp Z}{=}\frac{p(z)P(x)}{P(x)} = p(z).$
7222
$r$  nats of information "for free". As long as  $\mathrm{KL}(q(z|x,\lambda)||p(z)) < r$ , this does not optimise a proper ELBO (it misses the KL term), and the max introduces a discontinuity. Chen et al. (2017) propose soft free bits (SFB), that instead multiplies the KL term in the ELBO with a weighing factor  $0 < \beta \leq 1$  that is dynamically adjusted based on the target rate  $r$ :  $\beta$  is incremented (or reduced) by  $\omega$  if  $R > \gamma r$  (or  $R < \varepsilon r$ ). Note that this technique requires hyperparameters (i.e.  $\gamma, \varepsilon, \omega$ ) besides  $r$  to be tuned in order to determine how  $\beta$  is updated.
Change of objective We may also seek alternatives to the ELBO as an objective and relate them to quantities of interest such as MI. A simple adaptation of the ELBO weighs its KL-term by a constant factor ( $\beta$ -VAE; Higgins et al., 2017). Setting  $\beta < 1$  promotes increased MI. Whilst being a useful counter to posterior collapse, low  $\beta$  might lead to variational posteriors becoming point estimates. InfoVAE (Zhao et al., 2018b) mitigates this with a term aimed at minimising the divergence from the aggregated posterior  $q(z|\lambda) = \mathbb{E}_X[q(z|x,\lambda)]$  to the prior. Following Zhao et al. (2018b), we approximate this with an estimate of maximum mean discrepancy (MMD; Gretton et al., 2012) in our experiments. Lagrangian VAE (LagVAE; Zhao et al., 2018a) casts VAE optimisation as a dual problem; it targets either maximisation or minimisation of (bounds on)  $I(X;Z|\lambda)$  under constraints on the InfoVAE objective. In MI-maximisation mode, LagVAE maximises a weighted lower-bound on MI,  $-\alpha D$ , under two constraints, a maximum -ELBO and a maximum MMD, that prevent  $p(z|x,\theta)$  from degenerating to a point mass. Reasonable values for these constraints have to be found empirically.
4 Minimum Desired Rate
We propose minimum desired rate (MDR), a technique to attain ELBO values at a pre-specified rate  $r$  that does not suffer from the gradient discontinuities of FB, and does not introduce the additional hyperparameters of SFB. The idea is to optimise the ELBO subject to a minimum rate constraint  $r$ :
$$
\max  _ {\theta , \lambda} \mathcal {E} (\theta , \lambda), \tag {5}
$$
$$
\mathrm {s . t .} \mathbb {E} _ {X} \left[ \operatorname {K L} (q (z | x, \lambda) | | p (z)) \right] > r.
$$
Because constrained optimisation is generally intractable, we optimise the Lagrangian (Boyd and Vandenberghe, 2004)  $\Phi (\theta ,\lambda ,u) =$
$$
\mathcal {E} (\theta , \lambda) - u (r - \mathbb {E} _ {X} [ \mathrm {K L} (q (z | x, \lambda) | | p (z)) ]) \tag {6}
$$
where  $u\in \mathbb{R}_{\geq 0}$  is a positive Lagrangian multiplier. We define the dual function  $\phi (u) =$ $\max_{\theta ,\lambda}\Phi (\theta ,\lambda ,u)$  and solve the dual problem  $\min_{u\in \mathbb{R}_{\geq 0}}\phi (u)$  . Local minima of the resulting min-max objective can be found by performing stochastic gradient descent with respect to  $u$  and stochastic gradient ascent with respect to  $\theta ,\lambda$
4.1 Relation to other techniques
It is insightful to compare MDR to the various techniques we surveyed in terms of the gradients involved in their optimisation. The losses minimised by KL annealing,  $\beta$ -VAE, and SFB have the form  $\ell_{\beta}(\theta, \lambda) = D + \beta R$ , where  $\beta \geq 0$ . FB minimises the loss  $\ell_{\mathrm{FB}}(\theta, \lambda) = D + \max(r, R)$ , where  $r > 0$  is the target rate. Last, with respect to  $\theta$  and  $\lambda$ , MDR minimises the loss  $\ell_{\mathrm{MDR}}(\theta, \lambda) = D + R + u(r - R)$ , where  $u \in \mathbb{R}_{\geq 0}$  is the Lagrangian multiplier. And with respect to  $u$ , MDR minimises  $\phi(u) = -D - R - u(R - r)$ .
Let us inspect gradients with respect to the parameters of the VAE, namely,  $\theta$  and  $\lambda$ . FB's gradient  $\nabla_{\theta,\lambda}\ell_{\mathrm{FB}}(\theta,\lambda) =$
$$
\nabla_ {\theta , \lambda} D + \left\{ \begin{array}{l l} 0 & \text {i f} R \leq r \\ \nabla_ {\theta , \lambda} R & \text {o t h e r w i s e} \end{array} \right. \tag {7a}
$$
is discontinuous, that is, there is a sudden 'jump' from zero to a (possibly) large gradient w.r.t.  $R$  when the rate dips above  $r$ . KL annealing,  $\beta$ -VAE, and SFB do not present such discontinuity
$$
\nabla_ {\theta , \lambda} \ell_ {\beta} (\theta , \lambda) = \nabla_ {\theta , \lambda} D + \beta \nabla_ {\theta , \lambda} R, \tag {7b}
$$
for  $\beta$  scales the gradient w.r.t.  $R$ . The gradient of the MDR objective is
$$
\nabla_ {\theta , \lambda} \ell_ {\mathrm {M D R}} (\theta , \lambda) = \nabla_ {\theta , \lambda} D + (1 - u) \nabla_ {\theta , \lambda} R \tag {7c}
$$
which can be thought of as  $\nabla_{\theta, \lambda} \ell_{\beta}(\theta, \lambda)$  with  $\beta$  dynamically set to  $1 - u$  at every gradient step.
Hence, MDR is another form of KL weighing, albeit one that targets a specific rate. Compared to  $\beta$ -VAE, MDR has the advantage that  $\beta$  is not fixed but estimated to meet the requirements on rate. Compared to KL-annealing, MDR dispenses with a fixed schedule for updating  $\beta$ , not only annealing schedules are fixed, they require multiple decisions (e.g. number of steps, linear or exponential increments) whose impact on the objective are not directly obvious. Most similar then, seems SFB. Like MDR, it flexibly updates  $\beta$  by targeting
7223
a rate. However, differences between the two techniques become apparent when we observe how  $\beta$  is updated. In case of SFB:
$$
\beta^ {(t + 1)} = \beta^ {(t)} + \left\{ \begin{array}{l l} \omega & \text {i f} R > \gamma r \\ - \omega & \text {i f} R <   \varepsilon r \end{array} \right. \tag {8a}
$$
where  $\omega, \gamma$  and  $\varepsilon$  are hyperparameters. In case of MDR (not taking optimiser-specific dynamics into account):
$$
u ^ {(t + 1)} = u ^ {(t)} - \rho \frac {\partial \phi (u)}{\partial u} = u ^ {(t)} + \rho (R - r) (8 b)
$$
where  $\rho$  is a learning rate. From this, we conclude that MDR is akin to SFB, but MDR's update rule is a direct consequence of Lagrangian relaxation and thus dispenses with the additional hyperparameters in SFB's handcrafted update rule. $^4$
5 Expressive Priors
Suppose we employ a multimodal prior  $p(z|\theta)$ , e.g. a mixture of Gaussians, and suppose we employ a unimodal posterior approximation, e.g. the typical diagonal Gaussian. This creates a mismatch between the prior and the posterior approximation families that makes it impossible for  $\mathrm{KL}(q(z|x,\lambda)||p(z|\theta))$  to be precisely 0. For the aggregated posterior  $q(z|\lambda)$  to match the prior, the inference model would have to—on average—cover all of the prior's modes. Since the inference network is deterministic, it can only do so as a function of the conditioning input  $x$ , thus increasing  $I(X;Z|\lambda)$ . Admittedly, this conditioning might still only capture shallow features of  $x$ , and the generator may still choose to ignore the latent code, keeping  $I(X;Z|\theta)$  low, but the potential seems to justify an attempt. This view builds upon Alemi et al. (2018)'s information-theoretic view which suggests that the prior regularises the inference model capping  $I(X;Z|\lambda)$ . Thus, we modify SENVAE to employ a more complex, ideally multimodal, parametric prior  $p(z|\theta)$  and fit its parameters.
MoG Our first option is a uniform mixture of Gaussians (MoG), i.e.  $p(z|\theta) =$
$$
\frac {1}{C} \sum_ {c = 1} ^ {C} \mathcal {N} (z | \boldsymbol {\mu} ^ {(c)}, \operatorname {d i a g} (\boldsymbol {\sigma} ^ {(c)} \odot \boldsymbol {\sigma} ^ {(c)})) \tag {9}
$$
where the Gaussian parameters are optimised along with other generative parameters. Note that though we give this prior up to  $C$  modes, the optimiser might merge some of them (by learning approximately the same location and scale).
VampPrior Motivated by the fact that, for a fixed posterior approximation, the prior that optimises the ELBO equals  $\mathbb{E}_X[q(z|x,\lambda)]$ , Tomczak and Welling (2018) propose the VampPrior, a variational mixture of posteriors:
$$
p (z | \theta) = \frac {1}{C} \sum_ {c = 1} ^ {C} q (z | v ^ {(c)}, \lambda) \tag {10}
$$
where  $v^{(c)}$  is a learned pseudo input—in their case a continuous vector. Again the parameters of the prior, i.e.  $\{v^{(c)}\}_{c=1}^{C}$ , are optimised in the ELBO. In our case, the input to the inference network is a discrete sentence, which is incompatible with the design of the VampPrior. Thus, we propose to bypass the inference network's embedding layer and estimate a sequence of word embeddings, which makes up a pseudo input. That is,  $v^{(c)}$  is a sequence  $\langle \mathbf{v}_1^{(c)}, \ldots, \mathbf{v}_{l_c}^{(c)} \rangle$  where  $\mathbf{v}_i^{(c)}$  has the dimensionality of our embeddings, and  $l_c$  is the length of the sequence (fixed at the beginning of training). Note, however, that for this prior to be multimodal, the inference model must already encode information in  $Z$ , thus there is some gambling in its design.
Our goal is to identify which techniques are effective in training VAEs for language modelling. Our evaluation concentrates on intrinsic metrics: negative log-likelihood (NLL), perplexity per token (PPL), rate  $(R)$ , distortion  $(D)$ , the number of active units (AU; Burda et al., 2015)) and gap in the accuracy of next word prediction (given gold prefixes) when decoding from a posterior sample versus decoding from a prior sample ( $\mathrm{Acc}_{\mathrm{gap}}$ ).
For VAE models, NLL (and thus PPL) can only be estimated. We use importance sampling (IS)
$$
P (x | \theta) = \int p (z, x | \theta) \mathrm {d} z \stackrel {\mathrm {I S}} {=} \int q (z | x) \frac {p (z , x | \theta)}{q (z | x)} \mathrm {d} z
$$