SlowGuess/NAACL_2021_OCR huggingface dataset:
From Tables 1 and 2, we observe that FlowPrior consistently outperforms the baselines in test set perplexity, sometimes by large margins. This is not surprising since the  $\mathbf{M}_{\mathrm{IS}}$  term in our training objective directly targets the perplexity metric because the expressions are identical (differing only in the number of samples used). While FB typically improves models on PTB, and helps FlowPrior to reach a higher AU and KL on the other datasets, it does not lead to better test PPL and reconstruction. We report additional results on measuring the impact of FB in the Appendix.
Another finding is that simply enriching the parametric family of the prior is not sufficient to improve our evaluation metrics. Tables 1 and 2 show mixed results when moving from the VAE with its standard Gaussian prior to the MoG- or Vamp-VAE. Though these priors have the potential to be multimodal, they could still be unimodal after training. For example, the MoG-VAE might learn a mixture in which all Gaussians have the same location and
Table 2: Language modeling results on other datasets.
Prior	PPL(↓)	KL	AU(↑)
PTB
Standard	101.8 / 101.4 / 98.4	0.0 / 0.0 / 3.2	0 / 0 / 2
MoG	101.9 / 98.2 / 96.7	0.0 / 0.0 / 0.0	0 / 0 / 0
Vamp	101.7 / 98.3 / 96.1	0.0 / 0.0 / 3.1	0 / 0 / 4
Real NVP	102.5 / 98.4 / 94.7	0.0 / 0.0 / 3.3	0 / 0 / 2
Yahoo
Standard	65.6 / 65.8 / 63.9	0.0 / 0.0 / 2.7	0 / 0 / 1
MoG	65.6 / 64.6 / 62.7	0.0 / 0.0 / 0.5	0 / 0 / 1
Vamp	78.5 / 74.8 / 62.9	0.0 / 0.0 / 1.5	0 / 0 / 2
Real NVP	65.6 / 65.8 / 62.5	0.0 / 0.0 / 1.4	0 / 0 / 4
Yelp
Standard	35.4 / 35.1 / 33.2	0.0 / 0.0 / 2.9	0 / 0 / 2
MoG	36.0 / 35.2 / 34.9	0.0 / 0.0 / 0.0	0 / 0 / 0
Vamp	38.0 / 35.0 / 33.7	0.0 / 0.0 / 4.1	0 / 0 / 1
Real NVP	35.6 / 35.1 / 31.8	0.0 / 0.0 / 4.2	0 / 0 / 2
SNLI
Standard	27.4 / 26.0 / 25.3	0.0 / 0.0 / 1.2	0 / 0 / 3
MoG	27.2 / 28.1 / 24.3	0.0 / 0.4 / 4.2	0 / 1 / 5
Vamp	27.6 / 26.0 / 23.7	0.0 / 0.0 / 2.8	0 / 0 / 2
Real NVP	27.7 / 26.1 / 22.4	0.0 / 0.0 / 3.8	0 / 0 / 3
Table 3: Comparing training objectives with several choices for priors. Each cell has three results: training with  $\mathbf{M}_{\mathrm{IS}}$  only, ELBO only, and the combination of ELBO +  $\mathbf{M}_{\mathrm{IS}}$ . The combination consistently improves performance across models and datasets.
scale. Also, the complexity of the prior learned by the Vamp-VAE is dependent upon the inference network, so if the inference network does not learn anything useful, the learned prior may not be useful either.
Impact of selection of objectives. The learned prior baselines (MoG-VAE and Vamp-VAE) fail to learn to use the latent variable, as shown by the small numbers (nearly zero) for the AU and MI metrics in Tables 1-2. Similar observations were made by Pelsmaeker and Aziz (2020). We argue that only improving the prior might not be sufficient, as the ELBO objective is difficult to optimize and little information may be learnable for the prior from the ELBO alone. To measure the utility of the  $\mathbf{M}_{\mathrm{IS}}$  term, we introduce this term to standard-VAE, MoG-VAE, and Vamp-VAE and evaluate the improved models under the same language model metrics.
Table 3 compares models trained with  $\mathbf{M}_{\mathrm{IS}}$ , the ELBO, and the combined training objective (Eq. 7). The combined objective is beneficial to all metrics for all priors and datasets. Our results are consistent with the observations of Rainforth et al. (2018) that tighter bounds are preferable for training the gener
Vamp-VAE + MIS
Man in a blue shirt and jeans is sitting on a bench.
Man in a blue shirt and jeans is sitting on a bench
Women in a white dress and a man in a black shirt are standing in front of a microphone.
two men are playing basketball
FlowPrior
The dog is running through the snow.
Two young boys are playing in the snow.
There is a man in a blue shirt and a woman in a black shirt and black pants.
Three people are sitting on a bench.
two men are standing on a bench
A girl is sitting on a bench
A young girl is sitting on a bench.
A young man is sitting on a bench.
A woman in a black shirt is sitting on a bench.
A woman is sitting on a bench.
Table 4: Interpolation from the prior on SNLI dataset. In each cell, the first and last sentences correspond to two sampled latent codes and between are linearly interpolated samples.
ative network, while looser bounds are preferable for training the inference network. Still, FlowPrior (real NVP +  $\mathbf{M}_{\mathrm{IS}}$ ) performs the best in PPL and MI compared to other models, showing the flexibility and the power of the real NVP architecture.
For the "Standard" setting in Table 3, the prior is fixed and not learned while in the other three settings the prior is learned. The combination of ELBO and  $\mathbf{M}_{\mathrm{IS}}$  is helpful across all settings.
6.2 Interpolations Between Prior Samples
One appealing aspect of VAEs for sentence modeling is the potential for learning a smooth, interpretable space for sentences. A qualitative way to explore the latent space is to interpolate between samples from the prior distribution. We randomly sample two latent vectors from the prior and linearly interpolate between them with evenly divided intervals (Bowman et al., 2016).<sup>7</sup> We use greedy
(a) MoG-VAE / ELBO
(b) MoG-VAE / ELBO+MIS
(c) Vamp-VAE / ELBO
(d) Vamp-VAE / ELBO+MIS
(e) Real NVP-VAE / ELBO
(f) FlowPrior
Figure 1: Densities of 4 dimensions of learned priors (SNLI dataset).
decoding in generation. Table 4 shows linear interpolation between prior samples in FlowPrior and Vamp-VAE + MIS (i.e., Vamp-VAE with the combined training objective). We observe substantial improvement with FlowPrior, as it can generate sentences with smooth semantic evolution while maintaining plausible generations in terms of fluency. This semantic evolution may reflect the complex structure in the learned prior distribution. Interpolations with MoG-VAE + MIS and Vamp-VAE + MIS have more repetitions and do not transit smoothly from one to the other. (Results with MoG-VAE are in the appendix.)
6.3 Visualization of Learned Priors
We randomly select 4 dimensions from the learned priors per model and plot their densities in Fig. 1.
In MoG-VAE, each dimension is a Gaussian mixture with 100 components. When only using the ELBO for training (Fig. 1(a)), the four visualized components all have similar shapes. After adding  $\mathbf{M}_{\mathrm{IS}}$  (Fig. 1(b)), different dimensions have similar locations but different scales.
Vamp-VAE permits relatively complex components because the means and variances are acquired from the inference network applied to learned
latent with Eq. 4. We also experiment with interpolating the two samples after mapping, namely interpolating in the VAE latent space, and find similar results.
<sup>8</sup>We additionally tried various sampling methods for decoding. This leads to more noise and becomes harder to interpret. Generations can be found in the Appendices.
For the MoG setting, we also performed experiments with setting the number of Gaussian components  $K = 1$  and observed comparable or slightly worse test PPL under all 3 choices of training loss than Standard setting.
FlowPrior is slightly different. Instead of directly sampling from the latent variable of VAE (in MoG-VAE and VampVAE), FlowPrior samples from the base distribution of real NVP, interpolates in the base distribution, and maps to the
MoG-VAE	MoG-VAE + MIS
The man is wearing a black shirt.	An older gentleman in a white shirt is walking in a parking lot.
A man is standing in front of a building.	A woman is walking in a field.
A man is standing in front of a building.	A young girl in a red shirt is playing with a toy.
Vamp-VAE	Vamp-VAE + MIS
A man is playing a guitar.	Man in a blue shirt and jeans is sitting on a bench.
A man is playing a guitar.	The man is wearing a black shirt.
A man is playing a guitar.	People are walking down the street.
VAE	FlowPrior
A man is sitting on a bench.	Man in a blue shirt and blue jeans is sitting on a rock with a hammer.
A man is sitting on a bench.	Two young boys are playing in the snow.
A man is sitting on a bench.	A dog is running through the snow.
pseudo-inputs. Fig. 1(c) shows that Vamp-VAE trained without  $\mathbf{M}_{\mathrm{IS}}$  does not show much difference compared to MoG-VAE. However, when training with  $\mathbf{M}_{\mathrm{IS}}$  (Fig. 1(d)), the distributions in several dimensions appear to be multimodal.
The real NVP prior learns little information when training without  $\mathbf{M}_{\mathrm{IS}}$ , as all dimensions are akin to standard normal distributions. When training with  $\mathbf{M}_{\mathrm{IS}}$ , different dimensions show distinct placement and shape. The prior in FlowPrior is highly multimodal overall and smooth in each dimension.
6.4 Generations from Prior Samples
Sampling from Prior. To measure the expressiveness of the prior and the richness of the learned latent variable, we randomly sample 5000 times from the prior distribution and evaluate their greedy-decoded generations qualitatively and quantitatively. Table 5 shows greedy generations from prior samples. We observe substantial improvements in term of generation diversity when adding  $\mathbf{M}_{\mathrm{IS}}$  in the training objective. Note that these diverse samples are achieved with a purely deterministic decoding. A diverse set of samples implies that (1) richer latent codes and a highly multimodal distribution is learned by the model; (2) and the generator is trained to attend to the latent codes.
Sample Mundanity and Coverage. A strongly-performing generative model should be able to generate samples that comport to the training data distribution. We use the forward and reverse PPL to estimate the similarity between the training data and samples. We can consider F-PPL as a generation "precision" as it reflects the amount of information in the samples that is relevant to the actual text. Analogously, we can consider R-PPL
Table 5: Generations from prior samples with greedy decoding (SNLI dataset).
Yelp	SNLI
F-PPL	R-PPL	SB	F-PPL	R-PPL	SB
VAE	4	30248	96	4	51127	100
VAE+MIS	5	10818	30	4	19047	73
Vamp-VAE	4	32504	100	4	56050	100
Vamp-VAE+MIS	7	5280	10	5	8420	29
FlowPrior	209	1677	3	42	5725	13
Table 6: Forward PPL (F-PPL), Reverse PPL (R-PPL), and Self-BLEU (SB) of greedy-decoded prior samples.
as a generation "recall" as it reflects how much the samples as a whole provide coverage of the actual text. Moreover, both F-PPL and R-PPL reflect whether the decoder is able to attend to the latent variable in generation.
Table 6 shows the F-PPL and R-PPL with greedy generation from prior samples. While Fang et al. (2019) treats a lower F-PPL as an indicator of better samples, we argue that it is not necessarily true. A model could achieve a low F-PPL by simply generating identical (or nearly-identical) high-probability sequences, like those observed from the VAE, MoG-VAE, and Vamp-VAE in Table 5. This reflects how an overly-simplified or restrictive assumption in the prior can lead to less diversity in samples.
Indeed, we find that models with very low F-PPL values often have very high R-PPL values. A lower R-PPL indicates the distribution of generated samples matches the distribution of the training data. From Table 6 we observe that adding  $\mathrm{M}_{\mathrm{IS}}$  is beneficial as it leads to a lower R-PPL. FlowPrior has the best R-PPL, and shows the capability of capturing characteristics of the target distribution that are not captured by simpler priors.
Generation Diversity. To identify which model has richer usage of latent variables, we use self-BLEU to measure the diversity of a set of samples. We observe significant improvements in FlowPrior in Table 6, which implies a diverse latent representation and a better utilization of the latent variable.
When considering the parameterized family of VAE models, expressive latent components (i.e., posterior and prior) have been widely studied in computer vision (Dinh et al., 2015, 2016; Kingma and Dhariwal, 2018). However, multimodal priors have been seldom applied to language, with some exceptions (Serban et al., 2017; He et al., 2018; Ziegler and Rush, 2019; Ma et al., 2019; Lee et al., 2020).
Chen et al. (2017) use autoregressive flow for the prior and posterior and experiment with images. Ziegler and Rush (2019) propose several autoregressive NF architectures and characterize performance on character-level language modeling. Ma et al. (2019) design priors using the Glow architecture to improve the performance of non-autoregressive neural machine translation. Lee et al. (2020) empirically characterize the performance of NF and simple Gaussian priors in token-level latent variable models, and observe that flexible priors yield higher log-likelihoods but not better BLEU scores on machine translation tasks.
Our work differs from that of Ziegler and Rush (2019) and Chen et al. (2017) as we are using a non-autoregressive flow-based architecture for the prior, while they are using autoregressive NF. Also, we focus on models with a single latent variable for an entire sentence, while similar prior work has focused on token-level latent variables (Ziegler and Rush, 2019; Ma et al., 2019; Lee et al., 2020).
Several others have employed NF for flexible modeling in NLP. Setiawan et al. (2020) present a variational translation model that uses NF in the approximate posterior while keeping the prior as Gaussian. Wang and Wang (2019) apply NF to a variational Wasserstein autoencoder in order to make the posterior more flexible. Jin et al. (2019) use transformed distributions via NF to model the emission density, which improves parsing performance as compared to Gaussian baselines.
We proposed a method, FlowPrior, that uses normalizing flow to define the prior in a sentence VAE
and adds the importance-sampled marginal likelihood  $(\mathbf{M}_{\mathrm{IS}})$  as a second term to the standard VAE objective. Our empirical results show FlowPrior yields a substantial improvement in language modeling and generation tasks as compared to prior work. Adding  $\mathbf{M}_{\mathrm{IS}}$  improves performance for other models as well, especially in settings when the prior parameters are being learned.
We would like to thank Sam Wiseman, Qingming Tang, and Mingda Chen for helpful discussions, and the anonymous reviewers for their comments that improved this paper.
Johannes Balle, Valero Laparra, and Eero P Simoncelli. 2016. Density modeling of images using a generalized normalization transformation. In International Conference on Learning Representations.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In International Conference on Learning Representations.
Mingda Chen, Qingming Tang, Karen Livescu, and Kevin Gimpel. 2018. Variational sequential labelers for semi-supervised learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 215-226, Brussels, Belgium. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In International Conference on Learning Representations.
Chris Cremer, Quaid Morris, and David Duvenaud. 2017. Reinterpreting importance-weighted autoencoders. In International Conference on Learning Representations (Workshop Track).
Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. Nice: Non-linear independent components estimation. In International Conference on Learning Representations.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016. Density estimation using Real NVP. In International Conference on Learning Representations.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3946-3956, Hong Kong, China. Association for Computational Linguistics.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating KL vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 240-250, Minneapolis, Minnesota. Association for Computational Linguistics.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881-889.
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lapping inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Representations.
Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland. Association for Computational Linguistics.
Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. betaVAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. 2019. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning.
Matthew D Hoffman and Matthew J Johnson. 2016. ELBO surgery: yet another way to carve up the variational evidence lower bound. In *Workshop in Advances in Approximate Bayesian Inference*, NIPS, volume 1, page 2.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning.
Lifeng Jin, Finale Doshi-Velez, Timothy Miller, Lane Schwartz, and William Schuler. 2019. Unsupervised learning of PCFGs with normalizing flow. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2442-2452, Florence, Italy. Association for Computational Linguistics.
Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. 2018. Semi-amortized variational autoencoders. In Proceedings of the 35th International Conference on Machine Learning.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational Bayes. In International Conference on Learning Representations.
Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible  $1 \times 1$  convolutions. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4743-4751. Curran Associates, Inc.
Jason Lee, Dustin Tran, Orhan Firat, and Kyunghyun Cho. 2020. On the discrepancy between density estimation and sequence generation. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pages 84-94, Online. Association for Computational Linguistics.
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3603-3614, Hong Kong, China. Association for Computational Linguistics.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xi-jun Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4678-4699, Online. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Nonautoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4282-4292, Hong Kong, China. Association for Computational Linguistics.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Tom Pelsmaeker and Wilker Aziz. 2020. Effective estimation of deep generative language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7220-7236, Online. Association for Computational Linguistics.
Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. 2018. Tighter variational bounds are not necessarily better. In International Conference on Machine Learning, pages 4277-4285. PMLR.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-VAEs. In International Conference on Learning Representations.
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In International Conference on Learning Representations.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 627-637, Copenhagen, Denmark. Association for Computational Linguistics.
Iulian Vlad Serban, Alexander Ororbia II, Joelle Pineau, and Aaron Courville. 2017. Piecewise latent variables for neural variational text processing. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 52-62, Copenhagen, Denmark. Association for Computational Linguistics.