SlowGuess/NAACL_2021_OCR huggingface dataset:
Hendra Setiawan, Matthias Sperber, Udhyakumar Nallasamy, and Matthias Paulik. 2020. Variational neural machine translation with normalizing flows. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7771-7777, Online. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830-6841.
Jakub M Tomczak and Max Welling. 2018. VAE with a VampPrior. In Proceedings of AISTATS.
Prince Zizhuang Wang and William Yang Wang. 2019. Riemannian normalizing flow on variational Wasserstein autoencoder for text modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 284-294, Minneapolis, Minnesota. Association for Computational Linguistics.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3881-3890. JMLR.org.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2017. InfoVAE: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. SIGIR.
Zachary M Ziegler and Alexander M Rush. 2019. Latent normalizing flows for discrete sequences. In International Conference on Learning Representations.
A Normalizing Flow
Change of Variable Formula. We start with a base density  $p_{\epsilon}(\epsilon)$ , and define a bijection function  $f: \epsilon \to Z$ , that maps from  $\epsilon \sim p_{\epsilon}$  to the target  $z \sim p_{Z}$ . According to the change of variable formula:
$$
\log p _ {Z} (z) = \log p _ {\epsilon} (f ^ {- 1} (z)) + \log \left| \det \left(\frac {\partial f ^ {- 1} (z)}{\partial z}\right) \right|
$$
where  $\frac{\partial f^{-1}(z)}{\partial z}$  is the Jacobian of  $f$  at  $z$ .
Normalizing Flows. A normalizing flow is a sequence of invertible, deterministic transformations. By repeatedly applying the rule for change of variables, the base density is transformed into a more complex one. Networks parameterized using NF can be trained through exact maximum log-likelihood computation. Exact sampling is performed by drawing sample from the base distribution and performing the chain of transformations. Our work uses NF because it allows a flexible functional form, and it is capable of capturing data complexity and performing exact likelihood computation and sampling.
Real NVP. Computing the Jacobian of functions with high dimension and the determinants of large matrices (i.e., the two main computation in NF) are very expensive. Prior work has addressed this challenge by introducing efficient transformations (Dinh et al., 2015, 2016; Germain et al., 2015; Kingma et al., 2016; Kingma and Dhariwal, 2018; Ho et al., 2019).
Our flow-based prior is based on real-valued non-volume preserving (real NVP; Dinh et al., 2016) which is efficient in both training and sampling. The main building block of real NVP transformation is the affine coupling layer.
An affine coupling layer is a bijective transformation  $f_{i}:z_{i - 1}\to z_{i}$  that follows the equations:
$$
\begin{array}{l} z _ {i} ^ {(1: d)} = z _ {i - 1} ^ {(1: d)} \\ z _ {i} ^ {(d + 1: D)} = z _ {i} ^ {(d + 1: D)} \odot \exp (s (z _ {i - 1} ^ {(1: d)})) + t (z _ {i - 1} ^ {(1: d)}) \\ \end{array}
$$
where  $D$  is the dimensionality,  $z_{i}^{(1:d)}$  stands for the first  $d$  dimensions of  $z_{i}$ $(d < D)$ ;  $s$  and  $t$  denote the functions for scale and translation operations that map from  $\mathbb{R}^d\to \mathbb{R}^{D - d}$ ; and  $\odot$  denotes elementwise product.
The Jacobian determinant and inverse of the affine coupling layer are easy to compute. The transformation is flexible because its computation of the Jacobian determinant and inverse do not require any operation with the functions  $s$  and  $t$ , so these two functions could be designed to be arbitrarily complex.
The statistic of our dataset is in Table 7. For Yelp and SNLI, we follow Li et al. (2019) and create the
# Train	# Dev	# Test	Avg L	Max L	# Vocabulary
PTB	42,068	3,370	3,761	21	82	10,002
Yahoo	100,000	10,000	10,000	68	100	19,982
Yelp	100,000	10,000	10,000	9	15	9,389
SNLI	100,000	10,000	10,000	12	82	19,978
Table 7: Statistics of the datasets. # Train/Dev/Test is the number of train/dev/test instances. Avg L and Max L are the average and maximum length of the sequences in the training sets. # Vocabulary is the size of the vocabulary including ⟨unk⟩, ⟨sos⟩, ⟨eos⟩, and ⟨pad⟩.
PTB	Yahoo	SNLI	Yelp
Word Embedding	256	512	128	128
Encoder Hidden States	256	1024	512	512
Decoder Hidden States	256	1024	512	512
Table 8: The size of word embeddings and hidden states in VAE models used in this paper, which are adopted from prior work.
dataset with downsampling. For Yahoo, we truncate sentences to length 100 due to computational constraints.
We use a batch size of 32 and train using SGD without momentum. The optimizer is initialized with learning rate 1 or 0.5, and the learning rate is decayed by  $1/2$  if the dev loss is not improved in two consecutive epochs. The training stops early after 5 learning rate decay operations. We use a linear annealing schedule that increases the weight from 0 to 1 in the first 10 or 20 epoch for the weight of both KL and  $\mathbf{M}_{\mathrm{IS}}$  term if they are in the training objective. When training with the combined objective, we start adding  $\mathbf{M}_{\mathrm{IS}}$  after training ELBO objective 10 epochs. For each model variation, we experiment with 5 different random seeds and report the median numbers in the paper.
D Hyperparameter Settings
Across all the experiments for our implemented baselines (i.e., standard VAE, MoG-VAE, VampVAE) and our proposed model FlowPrior, we follow prior work (Kim et al., 2018; He et al., 2019; Li et al., 2019) and use a single-layer LSTM encoder and decoder with a 32-dimensional latent variable. We follow the prior work (He et al., 2019; Li et al., 2019) and set the embedding dimension as in Table 8. We set a dropout rate of 0.5 to both the
For other baselines, we use their open source implementations.
Model	PPL(↓)	KL	AU(↑)	MI(↑)
PTB
VAE	101.4 / 101.6	0.00 / 4.46	0 / 32	0.0 / 0.1
VAE+MIS	96.9 / 95.8	1.57 / 6.34	24 / 32	0.6 / 1.5
MoG-VAE	98.2 / 96.8	0.00 / 2.35	0 / 32	0.00 / 0.68
Vamp-VAE	98.3 / 97.3	0.00 / 2.31	0 / 32	0.00 / 0.72
FlowPrior	94.7 / 93.6	3.28 / 7.21	2 / 31	2.3 / 2.8
Yahoo
VAE	65.8 / 64.6	0.00 / 4.88	0 / 32	0.0 / 0.9
VAE+MIS	63.9 / 61.7	2.72 / 13.31	1 / 32	2.0 / 1.7
MoG-VAE	64.6 / 67.3	0.00 / 1.83	0 / 32	0.0 / 0.6
Vamp-VAE	74.8 / 75.9	0.01 / 1.24	0 / 32	0.0 / 0.6
FlowPrior	62.5 / 68.3	1.43 / 10.99	4 / 25	1.6 / 0.6
Yelp
VAE	35.1 / 37.5	0.00 / 3.59	0 / 32	0.0 / 1.0
VAE+MIS	33.2 / 39.6	2.91 / 4.16	28 / 32	0.9 / 2.2
MoG-VAE	35.2 / 39.8	0.01 / 1.81	0 / 32	0.0 / 0.6
Vamp-VAE	35.0 / 39.4	0.00 / 1.78	0 / 32	0.0 / 0.6
FlowPrior	31.8 / 39.0	4.15 / 10.13	2 / 32	2.5 / 2.6
SNLI
VAE	26.0 / 30.5	0.00 / 1.84	0 / 32	0.0 / 0.9
VAE+MIS	25.3 / 17.8	1.23 / 15.48	23 / 32	0.5 / 2.0
MoG-VAE	28.1 / 27.5	0.44 / 2.28	1 / 32	0.4 / 0.7
Vamp-VAE	26.0 / 29.3	0.00 / 5.11	0 / 32	0.0 / 0.8
FlowPrior	22.4 / 26.2	3.83 / 7.59	3 / 32	1.0 / 3.2
Table 9: Results when comparing standard KL and FB KL for several models. The left part in each cell shows training with standard KL and the right part shows using FB KL instead.
input embeddings and the output embeddings before the softmax layer in the decoder. All the parameters are initialized with a uniform distribution  $U(-0.01, 0.01)$ . For both MoG and Vamp-VAE we use 100 components/pseudo-inputs in the prior. For real NVP, we use 10 affine coupling layers with 3-layer MLP networks for the parameterized scale and translation operations with the dimensionality of 32. We follow Dinh et al. (2016) to compose the affine coupling layers in an alternative pattern and add batch normalization (Balle et al., 2016) between adjacent affine coupling layers. For models trained with FB KL, we set the target rate as 2, 4, or 8.
E Additional Results with Free Bits KL
Using the Free Bits method can help achieve a consistently better AU and higher KL as shown in the overall results in the main text. We report additional empirical comparisons to focus on measuring the impact of FB for three models in Table 9.
Though adding FB yields higher AU and MI, it
Yelp	SNLI
KL	FB KL	KL	FB KL
VAE	0.26	0.49	1.66	2.18
VAE+MIS	0.65	0.71	1.91	3.57
FlowPrior	1.50	0.74	4.87	3.64
Table 10: Test set reconstruction BLEU scores.
Yelp	SNLI
F-PPL	R-PPL	SB	F-PPL	R-PPL	SB
VAE	4	30248	96	4	51127	100
VAE+MIS	5	10818	30	4	19047	73
MoG-VAE	4	30413	100	3	45979	77
MoG-VAE+MIS	4	33624	100	6	5257	26
Vamp-VAE	4	32504	100	4	56050	100
Vamp-VAE+MIS	7	5280	10	5	8420	29
FlowPrior	209	1677	3	42	5725	13
VAE+FB	7	7517	29	4	22536	42
VAE+MIS+FB	8	5713	13	4	24204	48
FlowPrior+FB	8	5179	9	15	4876	11
Table 11: Forward PPL (F-PPL), Reverse PPL (R-PPL), and Self-BLEU (SB) of greedy-decoded prior samples.
is not always true that it leads to a better test PPL and reconstruction. This phenomenon has been pointed out by Razavi et al. (2019) that adding FB makes the objective non-smooth which can lead to optimization difficulties. Possible solutions could be changing to a better training procedure. Li et al. (2019) remedy this issue by combining pretraining with FB, namely using a pretrained autoencoder to initialize the inference network before starting training the VAE networks. This suggests that it may be necessary to pretrain the inference network and decoder to unilaterally benefit from FB.
Table 10 considers standard VAEs and FlowPrior when comparing the use of standard KL to FB KL. Using FB KL does not lead to a higher BLEU score in FlowPrior, though FB does improve BLEU when combined with standard VAE and  $\mathrm{VAE} + \mathrm{M}_{\mathrm{IS}}$
F Additional Results with FB and  $\mathbf{M}_{\mathrm{IS}}$
Table 11 shows the impact of  $\mathbf{M}_{\mathrm{IS}}$  and FB on F-PPL, R-PPL, and self-BLEU with greedy generation from prior samples.
G Reconstruction Results with Sampling
Tables 12-13 show the reconstruction performance with standard sampling and nucleus sampling with  $p = 0.9$  (Holtzman et al., 2020). We observe the trends are consistent with the results that use greedy decoding.
Yelp	SNLI
ELBO	ELBO+MIS	ELBO	ELBO+MIS
Standard	0.07	0.15	0.43	0.57
MoG	0.08	0.00	0.66	2.36
Vamp	0.06	0.32	0.55	0.76
Real NVP	0.28	0.60	0.97	1.91
KL	FB KL	KL	FB KL
VAE	0.07	0.13	0.43	0.74
VAE+MIS	0.15	0.28	0.57	0.95
MoG	0.08	0.08	0.66	0.54
Vamp	0.06	0.10	0.55	0.78
FlowPrior	0.60	0.34	1.91	0.91
Table 12: Test set reconstruction BLEU scores using standard sampling in decoding.
Yelp	SNLI
ELBO	ELBO+MIS	ELBO	ELBO+MIS
Standard	0.08	0.20	0.56	0.72
MoG	0.09	0.06	0.80	2.66
Vamp	0.08	0.44	0.71	1.01
Real NVP	0.30	0.72	1.31	2.40
KL	FB KL	KL	FB KL
VAE	0.08	0.17	0.56	1.03
VAE+MIS	0.20	0.36	0.72	1.22
MoG	0.09	0.16	0.80	0.82
Vamp	0.08	0.13	0.71	1.07
FlowPrior	0.72	0.40	2.40	1.23
H Interpolation with Sampling
Table 14 shows more examples of interpolation-based generation with greedy decoding. We show results with sampling methods for decoding in Tables 15 and 16. The results with greedy decoding provide a lower-variance way to interpret the learned latent space. The additional results with sampling methods provide a richer picture as they also capture the randomness in the relationship between the latent variable and the text. This is especially helpful when we observe repetition in neighboring samples with greedy decoding, as we see with MoG-VAE and Vamp-VAE in Table 14. Even with sampling, FlowPrior shows a smoother semantic evolution in the latent space than MoGVAE and Vamp-VAE, at least in terms of aspects of the subjects of the generated sentences.
I Sampling from Priors
Table 17 shows more greedy generations from prior samples. We observe substantial improvements in term of generation diversity in FlowPrior and Flow-
Table 13: Test set reconstruction BLEU scores using nucleus sampling in decoding.
Mog-VAE
The man is wearing a black shirt .
The man is wearing a black shirt .
The man is wearing a black shirt .
A man is standing in front of a building .
A man is standing in front of a building .
A man is standing in front of a building .
A man is standing in front of a building .
A man is standing in front of a building .
A man is standing in front of a building .
Vamp-VAE
Two young boys are playing in the snow.
There is a man in a blue shirt and a woman in a black shirt and black pants.
two men are standing on a bench
A girl is sitting on a bench
A woman in a black shirt is sitting on a bench.
Table 14: Interpolation between two prior samples with greedy decoding. Dataset used is SNLI. In each cell, the first sentence and the last sentence correspond to the two sampled latent codes, and between are linearly interpolated samples.
Prior + FB. While standard VAE always yields identical samples because the latent variable is ignored, using FB KL yields better sample diversity as it encourages more information encoded into latent variable during training. This can be easily observed by comparing the samples from standard VAE with those from VAE+FB, comparing MoG-VAE with MoG-VAE +  $\mathbf{M}_{\mathrm{IS}}$  ,comparing Vamp-VAE with Vamp-VAE  $^+$  MIs.
J More Visualizations of Real NVP Prior and FlowPrior
Visualizations of each dimension alone and all dimensions together are in Figures 2 and 3.
(a) dim 1
(b) dim 2
(c) dim 3
(d) dim 4
(e) all 32 dims
Figure 2: Visualization of dimensions of learned prior when using real NVP on SNLI dataset. Plots from left to right are first dimension alone, second dimension alone, third dimension alone, fourth dimension alone, and all 32 dimensions together.
(a) dim 1
Figure 3: Visualization of dimensions of learned prior when using real NVP with  $\mathbf{M}_{\mathrm{IS}}$  (i.e., FlowPrior) on SNLI dataset. Plots from left to right are first dimension alone, second dimension alone, third dimension alone, fourth dimension alone, and all 32 dimensions together.
K Reproducibility
We train our models on 1080 TI and 2080 TI GPUs. The number of parameters in each model is listed in Table 18. We report the average runtime for each approach on the PTB dataset in Table 19. We report the validation performance in Table 20.
The datasets are downloaded via https://github.com/jxhe/ vae-lagging-encoder/blob/master/prepare_data.py, https://nlp.stanford.edu/projects/snli/, https://github.com/fangleai/ Implicit-LVM/tree/master/lang_model_ptb/data, and https://github. com/fangleai/Implicit-LVM/tree/ master/lang_model_yelp/data.
The links to open source implementation of the other baselines follow: CycVAE (Fu et al., 2019): github.com/snakeztc/NeuralDialog-CVAE; Lag-VAE (He et al., 2019): github.com/jxhe/vae-lagging-encoder; Pre-VAE+FB (Li et al., 2019): github.com/bohanli/vae-pretraining-encoder.
MoG-VAE
A girl is laughing at the beach.
The dog is walking in the water
Man in khaki jacket painting an elephant.
The man is breakdancing.
People are outside on a sunny day.
Two men are working in a lab.
Five soccer players playing soccer.
Men stand on a pier.
A person is riding a bicycle at a parade.
A brown and white dog with a brown collar is climbing over its hind legs while lying on the floor, talking and fabric in the grass.
A little girl wearing a yellow shirt looks at a fountain while a man is kneeling next to her and a child.
Two men play dominoes
Young lady in blue dress waits at a bus.
A woman carrying a small child looking through a window
. Three men are fighting with swords .
The little boy is riding his scooter down the paved road.
A man wearing a yellow suit eats a hotdog on a wooden table.
A group of friends are smiling.
three dogs are in the water
3 people walking down the street with their hands in the air
Man getting a picture
There are two dogs standing near each other.
Two men in white uniforms are cleaning on a mess of an escalator.
Two girls are getting some exercise together.
Two women working in a restaurant outdoors.
The children are riding.
The child is standing on the sidewalk in front of an apartment building.
Table 15: Interpolation between two prior samples with nucleus sampling for decoding. Dataset used is SNLI. In each cell, the first sentence and the last sentence correspond to the two sampled latent codes, and between are linearly interpolated samples.
A girl is attending a birthday of popcorn.
The dog is walking in the green snow.
Man lady on the beach.
People are outside on a sunny day
Some people are rooting in an outdoor restaurant.
Two men both face out for directions on a street.
Big hikers .
Five soccer players playing soccer after fifty finish in a field
Men stand on a doorstep.
Two farmers share a drink, while one looks at a woolen
A couple in black and white with a long blue scarf are standing in a store wearing a yellow hat.
A three people are riding a white bike through the desert along a road.
A tribesman near a playground.
People all gathered in the street looking at something on a woman.
A couple of angel making corn on a mountainside in a city.
A man holding a sign can and the young female.
The man playing a guitar concert festival
The shirtless woman and woman performing on sand . dog is outside .
A dog is jumping through the air to catch a Frisbee in the air.
A brown and white dog chewing on a red disc.
A crowd of people are blowing in a brown down balloon.
A woman is pushing her cart
A person is skiing through a snowy mountain.
A woman carrying a small child is playing with her friend on a busy street.
A man with a black shirt and brown long-sleeve shirt is standing near a graffiti that has come poles off around two.
A man, dressed with purple and black stands in bottoms while bandannas disbelief.
A man is looking at shoulder on a rack.
Table 16: Interpolation between two prior samples with standard sampling for decoding. Dataset used is SNLI. In each cell, the first sentence and the last sentence correspond to the two sampled latent codes, and between are linearly interpolated samples.
VAE	VAE + FB
A man is sitting on a bench .	Two men are playing basketball .
A man is sitting on a bench .	A man is playing a guitar .
A man is sitting on a bench .	The man is wearing a blue shirt .
A man is sitting on a bench .	The man is wearing a blue shirt .
A man is sitting on a bench .	Two men are playing basketball .
MoG-VAE	MoG-VAE + MIS
The man is wearing a black shirt .	An older gentleman in a white shirt is walking in a parking lot .
A man is standing in front of a building .	A dog is running .
A man is standing in front of a building .	A woman is walking in a field .
A man is standing in front of a building .	A young girl in a red shirt is playing with a toy .
A man is standing in front of a building .	An older gentleman in a white shirt and white pants is standing on a ladder with a large ladder on his right hand
Vamp-VAE	Vamp-VAE + MIS
A man is playing a guitar .	Women in a white dress and a man in a black shirt are standing in front of a microphone .
A man is playing a guitar .	Man in a blue shirt and jeans is sitting on a bench .
A man is playing a guitar .	The man is wearing a black shirt .
A man is playing a guitar .	People are walking down the street .
A man is playing a guitar .	Two men are playing a game of chess .
FlowPrior	FlowPrior + FB
Man in a blue shirt and blue jeans is sitting on a rock with a hammer .	Children are standing in the middle of a building with a man in a blue shirt and black pants .
Two young boys are playing in the snow .	A man is standing in the middle of a large building .
A dog is running through the snow .	The man is wearing a black shirt .
Two men are standing on a boat .	Two men are playing basketball .
A young man is sitting on a bench .	Girl in a blue shirt and black jacket standing on a bench .
Table 17: Greedy generation from prior samples.
VAE	MoG-VAE	Vamp-VAE	FlowPrior
PTB	8917074	8922174	8922174	9023314
Yahoo	55319136	55325536	55325536	55425376
Yelp	10260973	10267373	10367213	10263073
SNLI	18403914	18406814	18406814	18510154
Table 18: Number of parameters in each model.
VAE	MoG-VAE	Vamp-VAE	FlowPrior
PTB	4550	5100	7000	8550
Yahoo	40110	49115	87253	92526
Yelp	1640	2652	3510	5620
SNLI	1242	2190	4081	9720
Table 19: Average runtime of each approach (s).
VAE	MoG-VAE	Vamp-VAE	FlowPrior
PTB	101.50	101.28	101.27	100.14
Yahoo	332.30	332.12	344.06	330.01
Yelp	35.03	35.05	35.05	32.10
SNLI	41.83	41.83	41.86	41.17
Table 20: Corresponding validation performance (NLL on validation set) for each reported test result.