SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\stackrel {\mathrm {M C}} {\approx} \frac {1}{S} \sum_ {s = 1} ^ {S} \frac {p \left(z ^ {(s)} , x \mid \theta\right)}{q \left(z ^ {(s)} \mid x\right)} \text {w h e r e} z ^ {(s)} \sim q (z \mid x) \tag {11}
$$
Note that if we set  $\gamma = 1$ ,  $\varepsilon = 1$ , and  $\omega = \rho(R - r)$  at every step of SFB, we recover MDR.
A latent unit (a single dimension of  $z$ ) is denoted active when its variance with respect to  $x$  is larger than 0.01.
7224
Technique	Hyperparameters
KL annealing	increment γ (2 × 10-5)
Word dropout (WD)	decrement γ (2 × 10-5)
FB and MDR	target rate r (5)
SFB	r (6.46), γ (1.05), ε (1), ω (0.01)
β-VAE	KL weight β (0.66)
InfoVAE	β (0.7), λ (31.62)
LagVAE	α (-21.7), target MMD (0.0017)
target -ELBO (100.8)
with our trained approximate posterior as importance distribution (we use  $S = 1000$  samples).
We first report on experiments using the English Penn Treebank (PTB; Marcus et al., 1993).<sup>6</sup>
RNNLM The baseline RNNLM generator is a building block for all of our SENVAEs, thus we validate its performance as a strong standalone generator. We highlight that it outperforms an external baseline that employs a comparable number of parameters (Dyer et al., 2016) and that this performance boost is mostly due to tying embeddings with the output layer.<sup>7</sup> Appendix A.1 presents the complete architecture and a comparison.
Bayesian optimisation The techniques we compare are sensitive to one or more hyperparameters (see Table 1), which we tune using Bayesian optimisation (BO) towards minimising estimated NLL of the validation data. For each technique, we ran 25 iterations of BO, each iteration encompassing training a model to full convergence. This was sufficient for the hyperparameters of each technique to converge. See Appendix A.2 for details.
On optimisation strategies First, we assess the effectiveness of techniques that aim at promoting local optima of SENVAE with better MI tradeoff. As for the architecture, the approximate posterior  $q(z|x,\lambda)$  employs a bidirectional recurrent encoder, and the generator  $P(x|z,\theta)$  is essentially our RNNLM initialised with a learned projection of  $z$  (complete specification in A.1). We train with Adam (Kingma and Ba, 2014) with default parameters and a learning rate of  $10^{-3}$  until convergence five times for each technique.
Results can be found in Table 2. First, note how
Table 1: Techniques and their hyperparameters.
Mode	D	R	PPL↓	AU↑	Accgap
RNNLM	-	-	107.1±0.5	-	-
Vanilla	118.4	0.0	105.7±0.4	0	0.0
Annealing	115.3	3.3	103.7±0.3	17	6.0
WD	117.6	0.0	102.5±0.6	0	0.0
FB	113.3	5.0	101.9±0.8	14	5.8
SFB	112.0	6.4	101.0±0.5	18	7.0
MDR	113.5	5.0	102.1±0.5	13	6.2
β-VAE	113.0	5.3	101.7±0.5	11	6.1
InfoVAE	113.5	4.3	100.8±0.4	10	5.2
LagVAE	112.1	6.5	101.6±0.7	24	6.9
Table 2: Performance (avg±std across 5 independent runs) of SENVAE on the PTB validation set. Standard deviations for  $D$  and  $R$  are at most 0.2.
the vanilla VAE (no special treatment) encodes no information in latent space  $(R = 0)$ . Then note that all techniques converged to VAEs that attain better PPL than the RNNLM and vanilla VAE, and all but annealed word dropout did so at non-negligible rate. Notably, the two most popular techniques, word dropout and KL annealing, perform sub-par to the other techniques. The techniques that work well at non-negligible rates can be separated into two groups: one based on a change of objective (i.e.,  $\beta$ -VAE, InfoVAE and LagVAE), another based on targeting a specific rate (i.e., FB, SFB, and MDR). InfoVAE, LagVAE and SFB all require tuning of multiple hyperparameters. InfoVAE and LagVAE, in particular, showed poor performance without this careful tuning. In the first group, consider LagVAE, for example. Though Zhao et al. (2018a) argue that the magnitude of  $\alpha$  is not particularly important (in MI-maximisation mode, they fixed it to  $-1$ ), we could not learn a useful SENVAE with LagVAE until we allowed BO to also estimate the magnitude of  $\alpha$ . Once BO converges to the values in Table 1, the method does perform quite well.
Generally, it is hard to believe that hyperparameters transfer across data sets, thus it is fair to expect that this exercise will have to be repeated every time. We argue that the rate hyperparameter common to the techniques in the second group is more interpretable and practical in most cases. For example, it is easy to grid-search against a handful of values. Hence, we further investigate FB and MDR by varying the target rate further (from 5 to 50). SFB is left out, for MDR generalises SFB's handcrafted update rule. We observe that FB and MDR attain essentially the same PPL across rates,
Though here we show annealed word dropout, to focus on techniques that do not weaken the generator, standard word dropout also converged to negligible rates.
<sup>6</sup>We report on Dyer et al. (2016)'s pre-processing, rather than Mikolov et al. (2010)'s. Whereas our findings are quantitatively similar, qualitative analysis based on generations are less interesting with Mikolov's far too small vocabulary.
7Stronger RNN-based models can be designed (Melis et al., 2018), but those use vastly more parameters.
7225
Figure 2: Sensitivity of output distributions to posterior samples measured in terms of symmetrised KL (JS). We obtain 51 (top) validation and 84 (bottom) test instances of length 20 and report on their output distributions per time step. To account for expected variability, we report  $\mathrm{JS}(\mathrm{Cat}(\pi_i)||\mathrm{Cat}(\eta_i)) - \mathrm{JS}(\mathrm{Cat}(\pi_i)||\mathrm{Cat}(\pi_i'))$  where  $\eta_{i}$  conditions on a prior sample, and  $\pi_{i}$  and  $\pi_{i}'$  condition on different posterior samples, averaged over 10 experiments.
Model	D	R	PPL↓	AU↑	Accgap
RNNLM	-	-	84.5±0.5	-	-
N/N	103.5	5.0	81.5±0.5	13	5.4
MoG/N	103.3	5.0	81.4±0.5	32	5.8
Vamp/N	103.1	5.0	81.2±0.4	22	5.8
Table 3: Performance on the PTB test set for different priors  $(\mathcal{N},\mathrm{MoG},\mathrm{Vamp})$ . Standard deviations of  $D,R,$  and  $\mathrm{Acc}_{\mathrm{gap}}$  are at most 0.1.
though MDR attains the desired rate earlier on in training, especially for higher targets (where FB fails at reaching the specified rate). Importantly, at the end of training, the validation rate is closer to the target for MDR. Appendix B supports these claims. Though  $\mathrm{Acc}_{\mathrm{gap}}$  already suggests it, Figure 2 shows more visibly that MDR leads to output Categorical distributions that are more sensitive to the latent encoding. We measure this sensitivity in terms of symmetrised KL between output distributions obtained from a posterior sample and output distributions obtained from a prior sample for the same time step given an observed prefix.
On expressive priors Second, we compare the impact of expressive priors. This time, prior hyperparameters were selected via grid search and can be found in Appendix A.1. All models are trained with a target rate of 5 using MDR, with settings otherwise the same as the previous experiment. In Table 3 it can be seen that more expressive priors do not improve perplexity further, though
they seem to encode more information in the latent variable—note the increased number of active units and the increased gap in accuracy. One may wonder whether stronger priors allow us to target higher rates without hurting PPL. This does not seem to be the case: as we increase rate to 50, all models perform roughly the same, and beyond 20 performance degrades quickly. $^{10}$  The models did, however, show a further increase in active units (VampPrior) and accuracy gap (both priors). Again, Appendix B contains plots supporting these claims.
Generated samples Figure 3 shows samples from a well-trained SENVAE, where we decode greedily from a prior sample—this way, all variability is due to the generator's reliance on the latent sample. Recall that a vanilla VAE ignores  $z$  and thus greedy generation from a prior sample is essentially deterministic in that case (see Figure 1a). Next to the samples we show the closest training instance, which we measure in terms of an edit distance (TER; Snover et al., 2006). This "nearest neighbour" helps us assess whether the generator is producing novel text or simply reproducing something it memorised from training. In Figure 4 we show a homotopy: here we decode greedily from points lying between a posterior sample conditioned on the first sentence and a posterior sample conditioned on the last sentence. In contrast to the vanilla VAE (Figure 1b), neighbourhood in latent space is now used to capture some regularities in data space. These samples add support to the quantitative evidence that our DGMs have been trained not to neglect the latent space. In Appendix B we provide more samples.
Other datasets To address the generalisability of our claims to other, larger, datasets, we report results on the Yahoo and Yelp corpora (Yang et al., 2017) in Table 4. We compare to the work of He et al. (2019), who proposed to mitigate posterior collapse with aggressive training of the inference network, optimising the inference network multiple steps for each step of the generative network.[12] We report on models trained with the standard prior as well as an MoG prior both op
10We also remark that, without MDR, the MoG model attains validation rate of about 2.5.
This distance metric varies from 0 to 1, where 1 indicates the sentence is completely novel and 0 indicates the sentence is essentially copied from the training data.
To enable direct comparison we replicated the experimental setup from (He et al., 2019) and built our methods into their codebase.
Here we remark that best runs (based on validation performance) do show an advantage, which stresses the need to report multiple runs as we do.
7226
Model	Yahoo	Yelp
R	NLL↓	PPL↓	AU↑	R	NLL↓	PPL↓	AU↑
RNNLM	-	328.0±0.3	-	-	-	358.1±0.6	-	-
Lagging	5.7±0.7	326.7±0.1	-	15.0±3.5	3.8±0.2	355.9±0.1	-	11.3±1.0
β-VAE (β = 0.4)	6.3±1.7	328.7±0.1	-	8.0±5.2	4.2±0.4	358.2±0.3	-	4.2±3.8
Annealing	0.0±0.0	328.6±0.0	-	0.0±0.0	0.0±0.0	357.9±0.1	-	0.0±0.0
Vanilla	0.0±0.0	328.9±0.1	61.4±0.1	0.0±0.0	0.0±0.0	358.3±0.2	40.8±0.1	0.0±0.0
N/N	5.0±0.0	328.1±0.1	60.8±0.1	4.0±0.7	5.0±0.0	357.5±0.2	40.4±0.1	4.2±0.4
MoG/N	5.0±0.1	327.5±0.2	60.5±0.1	5.0±0.7	5.0±0.0	359.5±0.5	41.2±0.3	2.2±0.4
Table 4: Performance on the Yahoo/Yelp corpora. Top rows taken from (He et al., 2019)
Sample	Closest training instance	TER
For example, the Dow Jones Industrial Average fell almost 80 points to close at 2643.65.	By futures-related program buying, the Dow Jones Industrial Average gained 4.92 points to close at 2643.65.	0.38
The department store concern said it expects to report profit from continuing operations in 1990.	Rolls-Royce Motor Cars Inc. said it expects its U.S. sales to remain steady at about 1,200 cars in 1990.	0.59
The new U.S. auto makers say the accord would require banks to focus on their core businesses of their own account.	International Minerals said the sale will allow Mallinckrodt to focus its resources on its core businesses of medical products, specialty chemicals and flavors.	0.78
Figure 3: Samples from SENVAE (MoG prior) trained via MDR: we sample from the prior and decode greedily. We also show the closest training instance in terms of a string edit distance (TER).
The inquiry soon focused on the judge.  
The judge declined to comment on the floor.  
The judge was dismissed as part of the settlement.  
The judge was sentenced to death in prison.  
The announcement was filed against the SEC.  
The offer was misstated in late September.  
The offer was filed against bankruptcy court in New York.  
The letter was dated Oct. 6.
Figure 4: Latent space homotopy from a properly trained SENVAE. Note the smooth transition of topic and grammatically of the samples.
timised with MDR, and a model trained without optimisation techniques. $^{13}$  It can be seen that MDR compares favourably to other optimisation techniques reported in (He et al., 2019). Whilst aggressive training of the inference network performs slightly better in terms of NLL and leads to more active units, it slows down training by a factor of 4. The MoG prior improves results on Yahoo but not on Yelp. This may indicate that a multimodal prior does offer useful extra capacity to the latent space, $^{14}$  at the cost of more instability in optimisation. This confirms that targeting a pre-specified rate leads to VAEs that are not collapsed without hurting NLL.
Recommendations We recommend targeting a specific rate via MDR instead of annealing (or word dropout). Besides being simple to implement, it is fast and straightforward to use: pick a rate by plotting validation performance against a handful of values. Stronger priors, on the other hand, while showing indicators of higher mutual information (e.g. AU and  $\mathrm{Acc}_{\mathrm{gap}}$ ), seem less effective than MDR. Use IS estimates of NLL, rather than single-sample ELBO estimates, for model selection, for the latter can be too loose of a bound and too heavily influenced by noisy estimates of KL.[15] Use many samples for a tight bound.[16] Inspect sentences greedily decoded from a prior (or posterior) sample as this shows whether the generator is at all sensitive to the latent code. Retrieve nearest neighbours to spot copying behaviour.
In NLP, posterior collapse was probably first noticed by Bowman et al. (2016), who addressed it via word dropout and KL scaling. Further investigation revealed that in the presence of strong generators,
<sup>15</sup>This point seems obvious to many, but enough published papers report exponentiated loss or distortion per token, which, besides unreliable, make comparisons across papers difficult.
<sup>16</sup>We use 1000 samples. Compared to a single sample estimate, we have observed differences up to 5 perplexity points in non-collapsed models. From 100 to 1000 samples, differences are in the order of 0.1 suggesting our IS estimate is close to convergence.
<sup>13</sup>We focus on MoG since the PTB experiments showed the VampPrior to underperform in terms of AU.
<sup>14</sup>We tracked the average KL divergence between any two components of the prior and observed that the prior remained multimodal.
7227
the ELBO itself becomes the culprit (Chen et al., 2017; Alemi et al., 2018), as it lacks a preference regarding MI. Posterior collapse has also been ascribed to approximate inference (Kim et al., 2018; Dieng and Paisley, 2019). Beyond the techniques compared and developed in this work, other solutions have been proposed, including modifications to the generator (Semeniuta et al., 2017; Yang et al., 2017; Park et al., 2018; Dieng et al., 2019), side losses based on weak generators (Zhao et al., 2017), factorised likelihoods (Ziegler and Rush, 2019; Ma et al., 2019), cyclical annealing (Liu et al., 2019) and changes to the ELBO (Tolstikhin et al., 2018; Goyal et al., 2017).
Exploiting a mismatch in correlation between the prior and the approximate posterior, and thus forcing a lower-bound on the rate, is the principle behind  $\delta$ -VAEs (Razavi et al., 2019) and hyperspherical VAEs (Xu and Durrett, 2018). The generative model of  $\delta$ -VAEs has one latent variable per step of the sequence, i.e.  $z = \langle z_1, \ldots, z_{|x|} \rangle$ , making it quite different from that of the SENVAEs considered here. Their mean-field inference model is a product of independent Gaussians, one per step, but they construct a correlated Gaussian prior by making the prior distribution over the next step depend linearly on the previous step, i.e.  $Z_i | z_{i-1} \sim \mathcal{N}(\alpha z_{i-1}, \sigma)$  with hyperparameters  $\alpha$  and  $\sigma$ . Hyperspherical VAEs work on the unit hypersphere with a uniform prior and a nonuniform VonMises-Fisher posterior approximation (Davidson et al., 2018). Note that, though in this paper we focused on Gaussian (and mixture of Gaussians, e.g. MoG and VampPrior) priors, MDR is applicable for whatever choice of prescribed prior. Whether its benefits stack with the effects due to different priors remains an empirical question.
GECO (Rezende and Viola, 2018) casts VAE optimisation as a dual problem, and in that it is closely related to our MDR and the LagVAE. GECO targets minimisation of  $\mathbb{E}_X[\mathrm{KL}(q(z|x,\lambda)||p(z))]$  under constraints on distortion, whereas LagVAE targets either maximisation or minimisation of (bounds on)  $I(X;Z|\lambda)$  under constraints on the InfoVAE objective. Contrary to MDR, GECO focuses on latent space regularisation and offers no explicit mechanism to mitigate posterior collapse.
Recently Li et al. (2019) proposed to combine FB, KL scaling, and pre-training of the inference network's encoder on an auto-encoding objective. Their techniques are complementary to ours in so
far as their main finding—the mutual benefits of annealing, pre-training, and lower-bounding KL—is perfectly compatible with ours (MDR and multimodal priors).
SENVAE is a deep generative model whose generative story is rather shallow, yet, due to its strong generator component, it is hard to make effective use of the extra knob it offers. In this paper, we have introduced and compared techniques for effective estimation of such a model. We show that many techniques in the literature perform reasonably similarly (i.e. FB, SFB,  $\beta$ -VAE, InfoVAE), though they may require a considerable hyperparameter search (e.g. SFB and InfoVAE). Amongst these, our proposed optimisation subject to a minimum rate constraint is simple enough to tune (as FB it only takes a pre-specified rate and unlike FB it does not suffer from gradient discontinuities), superior to annealing and word dropout, and require less resources than strategies based on multiple annealing schedules and/or aggressive optimisation of the inference model. Other ways to lower-bound rate, such as by imposing a multimodal prior, though promising, still require a minimum desired rate.
The typical RNNLM is built upon an exact factorisation of the joint distribution, thus a well-trained architecture is hard to improve upon in terms of log-likelihood of gold-standard data. Our interest in latent variable models stems from the desire to obtain generative stories that are less opaque than that of an RNNLM, for example, in that they may expose knobs that we can use to control generation and a hierarchy of steps that may award a degree of interpretability to the model. The SENVAE is not that model, but it is a crucial building block in the pursue for hierarchical probabilistic models of language. We hope this work, i.e. the organised review it contributes and the techniques it introduces, will pave the way to deeper—in statistical hierarchy—generative models of language.
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825299 (GoURMET).
7228
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. 2018. Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168.
The GPyOpt authors. 2016. GPyOpt: A bayesian optimization framework in python. http://github.com/SheffieldML/GPyOpt.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18:1-43.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155.
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305.
Léon Bottou and Yann L. Cun. 2004. Large scale online learning. In S. Thrun, L. K. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16, pages 217-224. MIT Press.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21.
Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In International Conference on Machine Learning.
Caio Corro and Ivan Titov. 2018. Differentiable perturb-and-parse: Semi-supervised parsing with a structured variational autoencoder. In ICLR.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. 2018. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891.
Adji Dieng and John Paisley. 2019. Reweighted expectation maximization. Technical report.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. 2019. Avoiding latent variable collapse with generative skip models. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 2397-2405. PMLR.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209. Association for Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 124–141, Florence, Italy. Association for Computational Linguistics.
Marco Fraccaro, Soren Kaele Sønderby, Ulrich Paquet, and Ole Winther. 2016. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pages 2199-2207.
Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a Bayesian approximation: Insights and applications. In Deep Learning Workshop, ICML.
Joshua T. Goodman. 2001. A bit of progress in language modeling. Comput. Speech Lang., 15(4):403-434.
Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. 2017. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pages 6713-6723.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander Smola. 2012. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In Proceedings of ICLR.
Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. betaVAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
Frederick Jelinek. 1980. Interpolated estimation of markov source parameters from sparse data. In Proc. Workshop on Pattern Recognition in Practice, 1980.
MichaelI. Jordan, Zoubin Ghahramani, TommiS. Jaakkola, and LawrenceK. Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233.
7229