SlowGuess/ACL_2020_OCR huggingface dataset:
Chen Wu $^{1}$ , Prince Zizhuang Wang $^{2}$ , William Yang Wang $^{2}$
$^{1}$ Department of Foreign Languages and Literatures, Tsinghua University  $^{2}$ Department of Computer Science, University of California, Santa Barbara
wu-c16@mails.tsinghua.edu.cn, zizhuang_wang@ucsb.edu, william@cs.ucsb.edu
Variational autoencoders (VAEs) combine latent variables with amortized variational inference, whose optimization usually converges into a trivial local optimum termed posterior collapse, especially in text modeling. By tracking the optimization dynamics, we observe the encoder-decoder incompatibility that leads to poor parameterizations of the data manifold. We argue that the trivial local optimum may be avoided by improving the encoder and decoder parameterizations since the posterior network is part of a transition map between them. To this end, we propose Coupled-VAE, which couples a VAE model with a deterministic autoencoder with the same structure and improves the encoder and decoder parameterizations via encoder weight sharing and decoder signal matching. We apply the proposed Coupled-VAE approach to various VAE models with different regularization, posterior family, decoder structure, and optimization strategy. Experiments on benchmark datasets (i.e., PTB, Yelp, and Yahoo) show consistently improved results in terms of probability estimation and richness of the latent space. We also generalize our method to conditional language modeling and propose Coupled-CVAE, which largely improves the diversity of dialogue generation on the Switchboard dataset. $^{1}$
The variational autoencoder (VAE) (Kingma and Welling, 2014) is a generative model that combines neural latent variables and amortized variational inference, which is efficient in estimating and sampling from the data distribution. It infers a posterior distribution for each instance with a shared inference network and optimizes the evidence lower bound (ELBO) instead of the intractable marginal
log-likelihood. Given its potential to learn representations from massive text data, there has been much interest in using VAE for text modeling (Zhao et al., 2017; Xu and Durrett, 2018; He et al., 2019).
Prior work has observed that the optimization of VAE suffers from the posterior collapse problem, i.e., the posterior becomes nearly identical to the prior and the decoder degenerate into a standard language model (Bowman et al., 2016; Zhao et al., 2017). A widely mentioned explanation is that a strong decoder makes the collapsed posterior a good local optimum of ELBO, and existing solutions include weakened decoders (Yang et al., 2017; Semeniuta et al., 2017), modified regularization terms (Higgins et al., 2017; Wang and Wang, 2019), alternative posterior families (Rezende and Mohamed, 2015; Davidson et al., 2018), richer prior distributions (Tomczak and Welling, 2018), improved optimization strategies (He et al., 2019), and narrowed amortization gaps (Kim et al., 2018).
In this paper, we provide a novel perspective for the posterior collapse problem. By comparing the optimization dynamics of VAE with deterministic autoencoders (DAE), we observe the incompatibility between a poorly optimized encoder and a decoder with too strong expressiveness. From the perspective of differential geometry, we show that this issue indicates poor chart maps from the data manifold to the parameterizations, which makes it difficult to learn a transition map between them. Since the posterior network is a part of the transition map, we argue that the posterior collapse would be mitigated with better parameterizations.
To this end, we propose the Coupled-VAE approach, which couples the VAE model with a deterministic network with the same structure. For better encoder parameterization, we share the encoder weights between the coupled networks. For better decoder parameterization, we propose a signal matching loss that pushes the stochastic decod
1Our code is publicly available at https://github. com/ChenWu98/Coupled-VAE.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3449-3464 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ing signals to the deterministic ones. Notably, our approach is model-agnostic since it does not make any assumption on the regularization term, the posterior family, the decoder architecture, or the optimization strategy. Experiments on PTB, Yelp, and Yahoo show that our method consistently improves the performance of various VAE models in terms of probability estimation and the richness of the latent space. The generalization to conditional modeling, i.e., Coupled-CVAE, largely improves the diversity of dialogue generation on the Switchboard dataset. Our contributions are as follows:
- We observe the encoder-decoder incompatibility in VAE and connect it to the posterior collapse problem.
- We propose the Coupled-VAE, which helps the encoder and the decoder to learn better parameterizations of the data manifold with a coupled deterministic network, via encoder weight sharing and decoder signal matching.
- Experiments on PTB, Yelp, and Yahoo show that our approach improves the performance of various VAE models in terms of probability estimation and richness of the latent space. We also generalize Coupled-VAE to conditional modeling and propose Coupled-CVAE, which largely improves the diversity of dialogue generation on the Switchboard dataset.
2.1 Variational Inference for Text Modeling
The generative process of VAE is first to sample a latent code  $z$  from the prior distribution  $\mathcal{P}(z)$  and then to sample the data  $x$  from  $P(x|z;\theta)$  (Kingma and Ba, 2015). Since the exact marginalization of the log-likelihood is intractable, a variational family of posterior distributions  $\mathcal{Q}(z|x;\phi)$  is adopted to derive the evidence lower bound (ELBO), i.e.,
$$
\begin{array}{l} \log P (x; \theta) \geq \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {Q} (\boldsymbol {z} | x; \phi)} [ \log P (x | \boldsymbol {z}; \theta) ] \tag {1} \\ - \operatorname {K L} [ \mathcal {Q} (\boldsymbol {z} | x; \phi) \| \mathcal {P} (\boldsymbol {z}) ] \\ \end{array}
$$
For training, as shown in Figure 1(a), the encoded text  $e$  is transformed into its posterior via a posterior network. A latent code is sampled and mapped to the decoding signal  $h$ . Finally, the decoder infers the input with the decoding signal. The objective can be viewed as a reconstruction loss  $\mathcal{L}_{rec}$  plus a regularization loss  $\mathcal{L}_{reg}$  (whose form varies), i.e.,
$$
\mathcal {L} = \mathcal {L} _ {\text {r e c}} + \mathcal {L} _ {\text {r e g}} \tag {2}
$$
(a) Variational autoencoder
(b) Deterministic autoencoder
Figure 1: VAE and DAE for text modeling.
However, the optimization of the VAE objective is challenging. We usually observe a very small  $\mathcal{L}_{reg}$  and a  $\mathcal{L}_{rec}$  similar to a standard language model, i.e., the well-known posterior collapse problem.
2.2 Deterministic Autoencoders
An older family of autoencoders is the deterministic autoencoder (DAE) (Rumelhart et al., 1986; Ballard, 1987). Figure 1(b) shows an overview of DAE for text modeling, which is composed of a text encoder, an optional MLP, and a text decoder. The reconstruction loss of DAE is usually much lower than that of VAE after convergence.
3 Encoder-Decoder Incompatibility in VAE for Text Modeling
To understand the posterior collapse problem, we take a deeper look into the training dynamics of VAE. We investigate the following questions. How much backpropagated gradient does the encoder receive from reconstruction? How much does it receive from regularization? How much information does the decoder receive from the encoded text?
3.1 Tracking Training Dynamics
To answer the first question, we study the gradient norm of the reconstruction loss w.r.t. the encoded text, i.e.,  $\| \partial \mathcal{L}_{rec} / \partial e\| _2$  , which shows the magni
(a) Gradient norm of the reconstruction loss w.r.t. the encoded text
(b) Gradient norm of the regularization loss w.r.t. the encoded text
Figure 2: Training dynamics of DAE, VAE, and the proposed Coupled-VAE on the Yelp test set. Please find the analysis in Section 3 and Section 5.7. Best viewed in color (yet the models are distinguished by line markers).
(c) Gradient norm of the decoding signal w.r.t. the encoded text
tude of gradients received by the encoder parameters. From Figure 2(a), we observe that it constantly increases in DAE, while in VAE it increases marginally in the early stage and then decreases continuously. It shows that the reconstruction loss actively optimizes the DAE encoder, while the VAE encoder lacks backpropagated gradients after the early stage of training.
We seek the answer to the second question by studying the gradient norm of the regularization loss w.r.t. the encoded text, i.e.,  $\| \partial \mathcal{L}_{reg} / \partial e\| _2$  . In a totally collapsed posterior, i.e.,  $\mathcal{Q}(\pmb {z}|x;\phi) = \mathcal{P}(\pmb {z})$  for each  $x$ $\| \partial \mathcal{L}_{reg} / \partial e\| _2$  would be zero. Thus,  $\| \partial \mathcal{L}_{reg} / \partial e\| _2$  can show how far the posterior of each instance is from the aggregate posterior or the prior. Figure 2(b) shows a constant decrease of the gradient norm in VAE from the 2.5K step until convergence, which shows that the posterior collapse is aggravated as the KL weight increases.
For the third question, we compute the normalized gradient norm of the decoding signal w.r.t. the encoded text, i.e.,  $\| \partial h / \partial e\| _F / \| h\| _2$ . As this term shows how relatively the decoding signal changes with the perturbation of the encoded text, it reflects the amount of information passed from the encoder to the decoder. Figure 2(c) shows that for DAE, it constantly increases. For VAE, it at first increases even faster than DAE, slows down, and finally decreases until convergence, indicating that the VAE decoder, to some extent, ignores the encoder in the late stage of training.
3.2 Encoder-Decoder Incompatibility
Based on the training dynamics in Section 3.1 and the observations in previous work (Bowman et al., 2016; Zhao et al., 2017), text VAE has three features, listed as follows. First, the encoder is poorly
optimized, as shown by the low  $\| \partial \mathcal{L}_{rec} / \partial e\| _2$  .Second, the decoder degenerates into a powerful language model. Third,  $h$  contains less information from  $e$  in VAE than in DAE, which is indicated by the lower  $\| \partial h / \partial e\| _F / \| h\| _2$  . We call these features as encoder-decoder incompatibility.
To bridge the incompatibility and posterior collapse, we start with the manifold hypothesis which states that real-world data concentrates near a manifold with a lower dimensionality than the ambient space (Narayanan and Mitter, 2010; Bengio et al., 2013). In our case, we denote the manifold of text data as  $\mathcal{X} \subset \bigcup_{l \in \mathbb{N}} \mathcal{V}^l$  where  $\mathcal{V}$  is the vocabulary. In the language of differential geometry, the encoded text  $e \in \mathcal{E} \subset \mathbb{R}^d$  and the decoding signal  $h \in \mathcal{H} \subset \mathbb{R}^d$  can be viewed as the parameterizations (or coordinates) of  $x \in \mathcal{X}$  under two different charts (or coordinate systems). Formally, we denote the chart maps as  $\varphi_e : \mathcal{X} \to \mathcal{E}$  and  $\varphi_h : \mathcal{X} \to \mathcal{H}$ , which satisfy  $e = \varphi_e(x)$  and  $h = \varphi_h(x)$  for any  $x \in \mathcal{X}$ . Given the two charts, the map from  $\mathcal{E}$  to  $\mathcal{H}$  is called the transition map  $\varphi_h \circ \varphi_e^{-1} : \mathcal{E} \to \mathcal{H}$  between the two charts.
In DAE, the two chart maps and the transition map between them are learned simultaneously via the single reconstruction loss, which we rewrite as
$$
\mathcal {L} _ {r e c} = \mathbb {E} _ {x \in \mathcal {X}} \left[ \mathcal {L} \left(x, \varphi_ {h} ^ {- 1} \left(\varphi_ {h} \circ \varphi_ {e} ^ {- 1} \left(\varphi_ {e} (x)\right)\right)\right) \right] \tag {3}
$$
where  $\varphi_{e},\varphi_{h}\circ \varphi_{e}^{-1}$  , and  $\varphi_h^{-1}$  are modeled as the encoder, the MLP, and the decoder (strictly speaking, in text modeling, the range of  $\varphi_h^{-1}$  is not  $\mathcal{X}$  but distributions on  $\mathcal{X}$  ), as illustrated in Figure 3.
In VAE, as discussed before, both  $\varphi_{e}$  and  $\varphi_{h}$  inadequately parameterize the data manifold. We argue that the inadequate parameterizations make it harder to find a smooth transition map in VAE than in DAE, as shown by the lower  $\| \partial h / \partial e\| _F / \| h\| _2$
Figure 3: Left: DAE and VAE interpreted as manifold parameterizations and a transition map. Right: A graphical overview of the proposed Coupled-VAE. The upper path is deterministic, and the lower path is stochastic.
Since the posterior network is a part of the transition map, it consequently seeks to map each instance to the prior (discussed in Section 3.1) rather than learning the transition map.
4 Coupling Variational and Deterministic Networks
Based on the above analysis, we argue that posterior collapse could be alleviated by learning chart maps (i.e.,  $\varphi_{e}$  and  $\varphi_{h}$ ) that better parameterize the data manifold. Inspired by the chart maps in DAE, we propose to couple the VAE model with a deterministic network, outlined in Figure 3. Modules with a subscript  $c$  are deterministic networks that share the structure with those in the stochastic network. Sampling is disabled in the deterministic network, e.g., in the case of Gaussian posterior, we use the predicted mean vector for later computation. Please find details for other posterior families in Appendix B. Similar to DAE, the coupled deterministic network is optimized solely by the coupled reconstruction loss  $\mathcal{L}_{rec}^{c}$ , which is the same autoregressive cross-entropy loss as  $\mathcal{L}_{rec}$ .
To learn a well-optimized  $\varphi_{e}$ , we share the encoder between the stochastic and the deterministic networks, which leverages the rich gradients backpropagated from  $\mathcal{L}_{rec}^{c}$ . To learn better  $\varphi_{h}$ , we propose to guide  $\varphi_{h}$  with a well-learned chart map, i.e., the one characterized by  $\mathrm{Decoder}^{c}$ . Thus, we introduce a signal matching loss  $\mathcal{L}_{match}$  that pushes the  $h$  to  $h^{c}$ . The objective of our approach is
$$
\mathcal {L} = \mathcal {L} _ {r e c} + \mathcal {L} _ {r e g} + \lambda_ {r} \mathcal {L} _ {r e c} ^ {c} + \lambda_ {m} \mathcal {L} _ {\text {m a t c h}} \tag {4}
$$
where  $\lambda_r$  and  $\lambda_m$  are hyperparameters $^2$ ,  $\mathcal{L}_{rec}^c$  is the coupled reconstruction loss, and the signal matching loss  $\mathcal{L}_{match}$  is essentially a distance function
between  $\pmb{h}$  and  $h^c$ . We evaluate both the Euclidean distance and the Rational Quadratic kernel<sup>3</sup>, i.e.,
$$
\mathcal {L} _ {\text {m a t c h}} = \left\{ \begin{array}{l l} \| \boldsymbol {h} - \operatorname {D e t a c h} \left(\boldsymbol {h} ^ {c}\right) \| ^ {2} & \text {E u c l} \\ \sum_ {s} \frac {- s \cdot C}{s \cdot C + \| \boldsymbol {h} - \operatorname {D e t a c h} \left(\boldsymbol {h} ^ {c}\right) \| ^ {2}} & R Q \end{array} \right. \tag {5}
$$
where  $s \in \{0.1, 0.2, 0.5, 1, 2, 5, 10\}$ ,  $C$  is a hyperparameter, and Detach prevents gradients to be propagated into  $h^c$  since we would like  $h^c$  to guide  $h$  but not the opposite.
One would question the necessity of sharing the structure of the posterior network by resorting to universal approximation (Hornik et al., 1989). Specifically, a common question is: why not using an MLP as Posterior  $c$ ? We argue that each structure has a favored distribution of  $\mathcal{H}$  in  $\mathbb{R}^d$ , so structure sharing facilitates the optimization when we are learning by gradient descent. For example, the latent space learned by planar flows (Rezende and Mohamed, 2015) has compression and expansion, and vMF-VAE (Xu and Durrett, 2018), which is supported on a sphere, may significantly influence the distribution of  $\mathcal{H}$  in its ambient space  $\mathbb{R}^d$ .
We conduct the experiments on three commonly used datasets for text modeling, i.e., the Penn Treebank (PTB) (Marcus et al., 1993), Yelp (Xu et al., 2016), and Yahoo. The training/validation/test splits are 42K/3370/3761 for PTB, 63K/7773/8671 for Yelp, and 100K/10K/10K for Yahoo. The vocabulary size for PTB/Yelp/Yahoo is 10K/15K/20K. We discard the sentiment labels in Yelp.
We evaluate the proposed Coupled-VAE approach by applying it to various VAE models, which in
3To avoid heavy hyperparameter tuning, we use the Rational Quadratic kernel unless otherwise specified.
2To avoid heavy hyperparameter tuning, we set  $\lambda_r = 1.0$  unless otherwise specified.
PTB	Yelp	Yahoo
NLL (KL)	PPL	NLL (KL)	PPL	NLL (KL)	PPL
GRU-LM*	105.8 (-)	125.3	196.3 (-)	57.3	347.9 (-)	78.0
VAE	103.6 (8.6)	112.9	193.7 (7.2)	54.3	344.5 (12.4)	74.7
Coupled-VAE	103.1 (9.5)	110.5	191.2 (8.0)	51.6	342.4 (12.8)	72.8
β(0.8)-VAE	103.8 (11.0)	113.9	193.8 (10.2)	54.5	344.9 (16.1)	75.1
Coupled-β(0.8)-VAE	103.3 (12.1)	111.5	191.5 (12.2)	51.9	342.8 (17.0)	73.2
β(1.2)-VAE	103.7 (7.8)	113.3	193.7 (6.0)	54.3	345.3 (10.5)	75.5
Coupled-β(1.2)-VAE	102.9 (8.6)	109.6	191.2 (6.9)	51.6	342.3 (11.3)	72.7
vMF-VAE	103.6 (2.0)	113.2	195.4 (0.0)	56.3	344.5 (2.5)	74.7
Coupled-vMF-VAE	103.0 (3.0)	110.1	191.2 (2.8)	51.6	342.2 (4.0)	72.5
CNN-VAE	118.5 (29.6)	222.6	194.2 (12.8)	54.8	344.3 (19.7)	74.5
Coupled-CNN-VAE	118.2 (30.2)	219.7	193.9 (13.7)	54.6	343.3 (22.4)	73.6
WAE	103.7 (11.0)	113.3	193.7 (10.7)	54.3	344.7 (16.6)	74.9
Coupled-WAE	103.2 (12.5)	110.9	191.3 (12.5)	51.7	343.3 (18.2)	73.6
VAE-NF	103.3 (5.5)	111.3	193.9 (5.3)	54.5	344.3 (8.1)	74.5
Coupled-VAE-NF	102.6 (5.7)	108.1	191.8 (5.6)	52.2	342.6 (8.8)	73.0
WAE-NF	103.4 (6.7)	111.9	194.1 (7.0)	54.7	344.3 (10.6)	74.5
Coupled-WAE-NF	102.7 (7.4)	108.4	192.1 (7.4)	52.5	342.7 (11.0)	73.1
CycAnn-VAE	104.2 (1.6)	116.3	192.5 (1.2)	53.0	345.4 (3.9)	75.5
Coupled-CycAnn-VAE	103.7 (2.4)	113.3	190.8 (2.0)	51.1	342.4 (4.4)	72.7
PreFB-VAE	103.4 (14.6)	111.9	190.4 (14.1)	50.7	341.4 (17.6)	71.8
Coupled-PreFB-VAE	103.3 (15.6)	111.4	189.9 (14.4)	50.3	341.3 (17.9)	71.7
SA-VAE†	100.7 (7.7)	98.7	183.5 (3.8)	44.0	327.5 (7.2)‡	60.4‡
Lagging-VAE†	98.8 (6.0)	90.7	182.5 (1.2)	43.1	326.7 (6.0)	59.7
Coupled-Lagging-VAE†	98.7 (11.0)	90.4	182.3 (3.8)	42.9	326.2 (7.4)	59.3
Table 1: Language modeling results. NLL is estimated with importance sampling. PPL is based on the estimated NLL. KL and MI are approximated by their Monte Carlo estimates. Coupled- stands for "with the coupled deterministic network". The better results in each block are shown in **bold**. *The exact NLL is reported.†Modifying open-source implementation which does not follow our setup and evaluation.‡Previously reported.
clude VAE (Kingma and Welling, 2014),  $\beta$ -VAE (Higgins et al., 2017), vMF-VAE (Xu and Durrett, 2018; Davidson et al., 2018) with learnable  $\kappa$ , CNN-VAE (Yang et al., 2017), WAE (Tolstikhin et al., 2018), VAE with normalizing flows (VAENF) (Rezende and Mohamed, 2015), WAE with normalizing flows (WAE-NF), VAE with cyclic annealing schedule (CycAnn-VAE) (Fu et al., 2019), VAE with encoder pretraining and the free bits objective (PreFB-VAE) (Li et al., 2019), and Lagging-VAE (He et al., 2019). We also show the result of GRU-LM (Cho et al., 2014) and SA-VAE (Kim et al., 2018). We do not apply our method to SA-VAE since it does not follow amortized variational inference. Please find more details in Appendix C and previous footnotes.
5.3 Language Modeling Results
We report negative log-likelihood (NLL), KL divergence, and perplexity as the metrics for language
modeling. NLL is estimated with importance sampling, KL is approximated by its Monte Carlo estimate, and perplexity is computed based on NLL. Please find the metric details in Appendix D.
Table 1 displays the language modeling results. For all models, our proposed approach achieves smaller negative log-likelihood and lower perplexity, which shows the effectiveness of our method to improve the probability estimation capability of various VAE models. Larger KL divergence is also observed, showing that our approach helps address the posterior collapse problem.
5.4 Mutual Information and Reconstruction
Language modeling results only evaluate the probability estimation ability of VAE. We are also interested in how rich the latent space is. We report the mutual information (MI) between the text  $x$  and the latent code  $z$  under  $\mathcal{Q}(z|x)$ , which is approximated with Monte Carlo estimation. Better
PTB	Yelp	Yahoo
MI	BLEU-1/2	MI	BLEU-1/2	MI	BLEU-1/2
VAE	10.48	23.2 / 4.4	8.28	28.7 / 5.3	15.43	21.2 / 3.6
Coupled-VAE	11.99	23.4 / 4.5	9.65	30.4 / 5.8	16.44	23.1 / 4.1
β(0.8)-VAE	15.43	24.5 / 4.9	13.52	30.6 / 6.0	24.16	24.0 / 4.3
Coupled-β(0.8)-VAE	18.13	24.3 / 4.8	17.69	32.6 / 6.6	28.03	26.4 / 4.9
β(1.2)-VAE	9.16	22.8 / 4.3	6.60	28.0 / 5.0	11.83	18.2 / 2.9
Coupled-β(1.2)-VAE	10.28	22.9 / 4.2	7.90	29.8 / 5.6	13.51	22.4 / 3.8
vMF-VAE	1.74	15.2 / 2.0	0.03	22.4 / 2.8	2.06	8.5 / 1.1
Coupled-vMF-VAE	2.37	16.1 / 2.3	2.60	25.1 / 4.0	3.37	10.3 / 1.4
CNN-VAE	78.49	32.0 / 7.8	17.26	32.9 / 7.1	30.18	24.9 / 5.3
Coupled-CNN-VAE	80.54	31.8 / 7.7	19.15	33.4 / 7.3	37.62	26.9 / 5.9
WAE	15.09	24.8 / 5.1	15.08	30.7 / 6.1	24.73	24.2 / 4.5
Coupled-WAE	18.51	24.7 / 5.1	18.56	32.5 / 6.6	30.08	27.7 / 5.3
VAE-NF	5.63	19.2 / 3.3	5.64	25.6 / 4.5	8.02	13.7 / 2.1
Coupled-VAE-NF	5.86	19.4 / 3.3	6.06	26.3 / 4.6	9.14	15.3 / 2.5
WAE-NF	7.18	19.7 / 3.5	7.95	26.0 / 4.6	11.43	13.8 / 2.2
Coupled-WAE-NF	8.10	20.7 / 3.7	8.53	27.2 / 5.0	12.56	14.9 / 2.5
CycAnn-VAE	1.55	16.3 / 2.3	1.18	22.6 / 3.2	3.09	8.3 / 1.1
Coupled-CycAnn-VAE	2.27	16.7 / 2.6	2.01	23.1 / 3.4	3.89	10.9 / 1.5
PreFB-VAE	20.6	25.5 / 5.7	20.3	33.1 / 6.8	26.2	27.2 / 5.2
Coupled-PreFB-VAE	23.2	25.8 / 5.8	21.0	33.3 / 6.8	27.0	27.2 / 5.3
Lagging-VAE†	2.90	-	0.96	-	3.04	-
Coupled-Lagging-VAE†	3.29	-	2.36	-	3.06	-
Table 2: Mutual information (MI) and reconstruction.  ${}^{ \dagger  }$  Modifying the open-source implementation.
reconstruction from the encoded text is another way to show the richness of the latent space. For each text  $x$ , we sample ten latent codes from  $\mathcal{Q}(z|x)$  and decode them with greedy search. We report the BLEU-1 and BLEU-2 scores between the reconstruction and the input. Please find the metric details in Appendix E. In Table 2, we observe that our approach improves MI on all datasets, showing that our approach helps learn a richer latent space. BLEU-1 and BLEU-2 are consistently improved on Yelp and Yahoo, but not on PTB. Given that text samples in PTB are significantly shorter than those in Yelp and Yahoo, we conjecture that it is easier for the decoder to reconstruct on PTB by exploiting its autoregressive expressiveness, even without a rich latent space.
5.5 Hyperparameter Analysis: Distance Function,  $\lambda_r$ , and  $\lambda_m$
We investigate the effect of key hyperparameters. Results are shown in Table 3. Note that the lowest NLL does not guarantee the best other metrics, which shows the necessity to use multiple metrics for a more comprehensive evaluation.
For the distance function, we observe that the
Euclidean distance (denoted as Eucl in Table 3) is more sensitive to  $\lambda_{m}$  than the Rational Quadratic kernel (denoted as RQ in Table 3).
The first and the third block in Table 3 show that, with larger  $\lambda_{m}$ , the model achieves higher KL divergence, MI, and reconstruction metrics. Our interpretation is that by pushing the stochastic decoding signals closer to the deterministic ones, we get latent codes with richer text information. We leave the analysis of  $\lambda_{m} = 0.0$  in Section 5.6.