SlowGuess/ACL_2020_OCR huggingface dataset:
1: Initialize  $\phi, \theta$  and  $\kappa$ .
2: for  $i = 1,2,\dots$  Until Convergence do
3: Sample a mini-batch  $\mathbf{x},\mathbf{y}$
4:  $\mu_q, \log \sigma_q^2 = f_\phi(\mathbf{x}, \mathbf{y})$  and  $\mu_p, \log \sigma_p^2 = f_\theta(\mathbf{x})$ .
5:  $\mu_q' = BN_{\gamma, \beta}(\mu_q)$ .
6: Sample  $\mathbf{z} \sim \mathcal{N}(\mu_q', \sigma_q^2)$  and reconstruct  $\mathbf{y}$  from  $f_{\kappa}(\mathbf{z}, \mathbf{x})$ .
7: Compute gradients  $\mathbf{g}_{\phi ,\theta ,\kappa}\gets \nabla_{\phi ,\theta ,\kappa}\mathcal{L}^{\prime}$
8: Update  $\phi, \theta, \kappa$  using  $\mathbf{g}_{\phi, \theta, \kappa}$ .
9: end for
<sup>1</sup>We perform empirical study on this method and find that the neural network can always find a small  $KL$  value in this situation.
Model	Yahoo	Yelp
NLL	KL	MI	AU	NLL	KL	MI	AU
Without a pretrained AE encoder
CNN-VAE	≤332.1	10.0	-	-	≤359.1	7.6	-	-
LSTM-LM	328	-	-	-	351.1	-	-	-
VAE	328.6	0.0	0.0	0.0	357.9	0.0	0.0	0.0
β-VAE (0.4)	328.7	6.3	2.8	8.0	358.2	4.2	2.0	4.2
cyclic*	330.6	2.1	2.0	2.3	359.5	2.0	1.9	4.1
Skip-VAE*	328.5	2.3	1.3	8.1	357.6	1.9	1.0	7.4
SA-VAE	327.2	5.2	2.7	9.8	355.9	2.8	1.7	8.4
Agg-VAE	326.7	5.7	2.9	15.0	355.9	3.8	2.4	11.3
FB (4)	331.0	4.1	3.8	3.0	359.2	4.0	1.9	32.0
FB (5)	330.6	5.7	2.0	3.0	359.8	4.9	1.3	32.0
δ-VAE (0.1)*	330.7	3.2	0.0	0.0	359.8	3.2	0.0	0.0
vMF-VAE (13)*	327.4	2.0	-	32.0	357.5	2.0	-	32.0
BN-VAE (0.6)*	326.7	6.2	5.6	32.0	356.5	6.5	5.4	32.0
BN-VAE (0.7)*	327.4	8.8	7.4	32.0	355.9	9.1	7.4	32.0
With a pretrained AE encoder
cyclic*	333.1	25.8	9.1	32.0	361.5	20.5	9.3	32.0
FB (4)*	326.2	8.1	6.8	32.0	356.0	7.6	6.6	32.0
δ-VAE (0.15)*	331.0	5.6	1.1	11.2	359.4	5.2	0.5	5.9
vMF-VAE (13)*	328.4	2.0	-	32.0	357.0	2.0	-	32.0
BN-VAE (0.6)*	326.7	6.4	5.8	32.0	355.5	6.6	5.9	32.0
BN-VAE (0.7)*	326.5	9.1	7.6	32.0	355.7	9.1	7.5	32.0
Table 1: Results on Yahoo and Yelp datasets. We report mean values across 5 different random runs. * indicates the results are from our experiments, while others are from He et al. (2019); Li et al. (2019). We only show the best performance of every model for each dataset. More results on various parameters can be found in the Appendix.
5.1 VAE for Language Modeling
Setup: We test our approach on two benchmark datasets: Yelp and Yahoo corpora (Yang et al., 2017). We use a Gaussian prior  $\mathcal{N}(0, I)$ , and the approximate posterior is a diagonal Gaussian. Following previous work (Burda et al., 2016; He et al., 2019), we report the estimated negative log likelihood (NLL) from 500 importance weighted samples, which can provide a tighter lower bound compared to the ELBO and shares the same information with the perplexity (PPL). Besides the NLL, we also report the KL, the mutual information (MI)  $I_q$  (Alemi et al., 2017) and the number of activate units (AU) (Burda et al., 2016) in the latent space. The  $I_q$  can be calculated as:
$$
\begin{array}{l} I _ {q} = \mathrm {E} _ {p _ {d} (\mathbf {x})} [ K L (q _ {\phi} (\mathbf {z} | \mathbf {x}) | | p (\mathbf {z})) ] - \\ K L \left(q _ {\phi} (\mathbf {z}) | | p (\mathbf {z})\right), \tag {13} \\ \end{array}
$$
where  $p_d(\mathbf{x})$  is the empirical distribution. The aggregated posterior  $q_{\phi}(\mathbf{z}) = \mathrm{E}_{p_d(\mathbf{x})}[q_{\phi}(\mathbf{z}|\mathbf{x})]$  and  $KL(q_{\phi}(\mathbf{z})||p(\mathbf{z}))$  can be approximated with Monte Carlo estimations. The AU is measured as  $A_z = Cov(\mathrm{E}_{\mathbf{z}\sim q(\mathbf{z}|\mathbf{x})}[\mathbf{z}])$ . We set the threshold of 0.01, which means if  $A_{zi} > 0.01$ , the unit  $i$  is active.
Configurations: We use a 512-dimension word embedding layer for both datasets. For the encoder and the decoder, a single layer LSTM with 1024
hidden size is used. We use  $\mathbf{z}$  to generate the initial state of the encoder following Kim et al. (2018); He et al. (2019); Li et al. (2019). To optimize the objective, we use mini-batch SGD with 32 samples per batch. We use one NVIDIA Tesla v100 for the experiments. For all experiments, we use the linear annealing strategy that increases the KL weight from 0 to 1 in the first 10 epochs if possible.
Compared methods: We compare our model with several strong baselines and methods that hold the previous state-of-the-art performance on text modeling benchmarks.
- Baselines, including neural autoregressive models (the LSTM language model).
- Methods with weakening the decoder: CNN-VAE (Yang et al., 2017).
- Methods with a modified model structure: SkipVAE (Dieng et al., 2019).
Methods with a modified training objective:
- VAE with annealing (Bowman et al., 2016).
-  $\beta$ -VAE (Higgins et al., 2017).
- Cyclic annealing (Fu et al., 2019), we use the default cyclic schedule.
Methods with a lower bound for KL values:
- Free-bits (FB) (Kingma et al., 2016).
-  $\delta$ -VAE (Razavi et al., 2019).
- vMF-VAE (Xu and Durrett, 2018)
- Methods with a modified training strategy.
Model	Yahoo	Yelp
Hours	Ratio	Hours	Ratio
VAE	3.83	1.00	4.50	1.00
SA-VAE	52.99	12.80	59.37	12.64
Agg VAE	11.76	2.84	21.44	4.56
AE+FB	7.70	2.01	9.22	2.05
BN-VAE	3.98	1.04	4.60	1.02
- Semi-amortized VAE (SA-VAE) (Kim et al., 2018).
- VAE with an aggressive training (Agg-VAE) (He et al., 2019).
- FB with a pretrained inference network (AE+FB) (Fu et al., 2019)
Main results: Table 1 shows the results. We further split the results into two different settings, one for models with a pretrained inference network and one without it. Our approach achieves the best NLL in the setting without a pretrained inference network on both datasets and is competitive in the setting with a pretrained encoder. Moreover, we can observe that:
-  $\delta$ -VAE does not perform well in both settings, which shows that constraining the parameters in a small interval is harmful to the model. In vMF-VAE, data points share the same KL value. Our approach is flexible and gets better performance.
- Although Agg-VAE and SA-VAE both get good performance, they require additional updates on the inference network and cost more training efforts, which are validated in the next part.
- Cyclic annealing with a pretrained inference network achieves the highest KL, but it may not be a good generative model.
- Paired with a pretrained inference network, all methods except cyclic annealing can someday boost the performance. This phenomenon indicates that the lagging problem (He et al., 2019) is important in VAE training. When leveraging the pretrained inference network, our approach achieves the smallest performance gap compared with other methods. In other words, our approach can alleviate the lagging problem efficiently.
Training time: Table 2 shows the training time (until convergence) and the relative ratio of the basic VAE, our approach and the other best three models in Table 1. SA-VAE is about 12 times slower than our approach due to the local update for each data point. Agg-VAE is 2-4 times slower
Table 2: Comparison of training time to convergence. We report both the absolute hours and relative speed.
#label	100	500	1k	2k	10k
AE	81.1	86.2	90.3	89.4	94.1
VAE	66.1	82.6	88.4	89.6	94.5
δ-VAE	61.8	61.9	62.6	62.9	93.8
Agg-VAE	80.9	85.9	88.8	90.6	93.7
cyclic	62.4	75.5	80.3	88.7	94.2
FB (9)	79.8	84.4	88.8	91.12	94.7
AE+FB (6)	87.6	90.2	92.0	93.4	94.9
BN-VAE (0.7)	88.8	91.6	92.5	94.1	95.4
Table 3: Accuracy on Yelp.
Model	CVAE	CVAE (BOW)	BN-VAE
PPL	36.40	24.49	30.67
KL	0.15	9.30	5.18
BLEU-4	10.23	8.56	8.64
A-bow Prec	95.87	96.89	96.64
A-bow Recall	90.93	93.95	94.43
E-bow Prec	86.26	83.55	84.69
E-bow Recall	77.91	81.13	81.75
Table 4: Comparison on dialogue generation.
than ours because it requires additional training for the inference network. AE+FB needs to train an autoencoder before the VAE. However, our approach is fast since we only add one-layer batch normalization, and thus the training cost is almost the same as the basic VAE. More results about the training behavior can be found in Section 3 of the Appendix.
Performance on a downstream task - Text classification: The goal of VAE is to learn a good representation of the data for downstream tasks. Here, we evaluate the quality of latent representations by training a one-layer linear classifier based on the mean of the posterior distribution. We use a downsampled version of the Yelp sentiment dataset (Shen et al., 2017). Li et al. (2019) further sampled various labeled data to train the classifier. To compare with them fairly, we use the same samples in Li et al. (2019). Results are shown in Table 3. Our approach achieves the best accuracy in all the settings. For 10k training samples, all the methods get a good result. However, when only using 100 training samples, different methods vary a lot in accuracy. The text classification task shows that our approach can learn a good latent representation even without a pretrained inference network.
5.2 CVAE for Dialogue Generation
Setup: For dialogue generation, we test our approach in the setting of CVAE. Following previous work (Zhao et al., 2017b), we use the Switchboard (SW) Corpus (Godfrey and Holliman, 1997), which contains 2400 two-sided telephone conversations.
Model	Fluency	Relevance	Informativeness
Avg	#Accept	#High	Avg	#Accept	#High	Avg	#Accept	#High
CVAE	2.11 (0.58)	87%	23%	1.90 (0.49)	82%	8%	1.39 (0.59)	34%	5%
CVAE (BOW)	2.08 (0.73)	84%	23%	1.86 (0.58)	75%	11%	1.54 (0.65)	46%	8%
BN-CVAE	2.16 (0.71)	88%	27%	1.92 (0.67)	80%	12%	1.54 (0.67)	43%	10%
Table 5: Human evaluation results. Numbers in parentheses is the corresponding variance on 200 test samples.
Topic: ETHICS IN GOVERNMENT
Context: have trouble drawing lines as to what's illegal and what's not
Target (statement): well i mean the other problem is that they're always up for
CVAE	CVAE (BOW)	BN-CVAE
1. yeah	1. yeah	1. it's not a country
2. yeah	2. oh yeah they're not	2. it is the same thing that's what i think is about the state is a state
3. yeah	3. no it's not too bad	3. yeah it's
Table 6: Sampled generated responses. Only the last sentence in the context is shown here.
We use a bidirectional GRU with hidden size 300 to encode each utterance and then a one-layer GRU with hidden size 600 to encode previous  $k - 1$  utterances as the context. The response decoder is a one-layer GRU with hidden size 400. The latent representation  $\mathbf{z}$  has a size of 200. We use the evaluation metrics from Zhao et al. (2017b): (1) Smoothed Sentence-level BLEU (Chen and Cherry, 2014); (2) Cosine Distance of Bag-of-word Embedding, which is a simple method to obtain sentence embeddings. We use the pretrained Glove embedding (Pennington et al., 2014) and denote the average method as A-bow and the extreme method as E-bow. Higher values indicate more plausible responses. We compared our approach with CVAE and CVAE with bag-of-words (BOW) loss (Zhao et al., 2017b), which requires the decoder in the generation network to predict the bag-of-words in the response  $\mathbf{y}$  based on  $\mathbf{z}$ .
Automatic evaluation: Table 4 shows the results of these three approaches. From the KL values, we find that CVAE suffers from posterior collapse while CVAE (BOW) and our approach avoid it effectively. For BLEU-4, we observe the same phenomenon in the previous work (Fu et al., 2019; Zhao et al., 2017b) that CVAE is slightly better than the others. This is because CVAE tends to generate the most likely and safe responses repeatedly with the collapsed posterior. As for precision, these three models do not differ much. However, CVAE (BOW) and our BN-VAE outperform CVAE in recall with a large margin. This indicates that BN-VAE can also produce diverse responses with good quality like CVAE (BOW).
Human evaluation: We conduct the human evaluation by asking five annotators from a commercial annotation company to grade 200 sampled conver
sations from the aspect of fluency, relevance and informativeness on a scale of 1-3 (see Section 4 of the Appendix for more details on the criteria). We also report the proportion of acceptable/high scores  $(\geq 2$  and  $= 3)$  on each metric. Table 5 shows the annotation results. Overall, our approach beats the other two compared methods in relevance and fluency with more informative responses. Also, our approach has the largest proportion of responses whose scores are High. This indicates that our model can produce more meaningful and relevant responses than the other two.
Case study: Table 6 shows the sampled responses generated by the three methods (more can be found in the Appendix). By maintaining a reasonable KL, responses generated by our approach are more relevant to the query with better diversity compared to the other two. We test the three methods in the simplest setting of dialogue generation. Note that the focus of this work is to improve the CVAE itself by avoiding its KL vanishing problem but not to hack the state-of-the-art dialogue generation performance. To further improve the quality of generated responses, we can enhance our approach by incorporating knowledge such as dialogue acts (Zhao et al., 2017b), external facts (Ghazvininejad et al., 2018) and personal profiles (Zhang et al., 2018).
In this paper, we tackle the posterior collapse problem when VAE is paired with autoregressive decoders. Instead of considering the KL individually, we make it follow a distribution  $D_{KL}$  and show that keeping the expectation of  $D_{KL}$  positive is sufficient to prevent posterior collapse. We propose Batch Normalized VAE (BN-VAE), a simple but effective approach to set a lower bound of  $D_{KL}$
by regularization the approximate posterior's parameters. Our approach can also avoid the recently proposed lagging problem efficiently without additional training efforts. We show that our approach can be easily extended to CVAE. We test our approach on three real applications, language modeling, text classification and dialogue generation. Experiments show that our approach outperforms strong baselines and is competitive with more complex methods which keeping substantially faster.
We leverage the Gaussian prior as the example to introduce our method in this work. The key to our approach to be applicable is that we can get a formula for the expectation of the KL. However, it is hard to get the same formula for some more strong or sophisticated priors, e.g., the Dirichlet prior. For these distributions, we can approximate them by the Gaussian distributions (such as in Srivastava and Sutton (2017)). In this way, we can batch normalize the corresponding parameters. Further study in this direction may be interesting.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. 2018. Fixing a broken elbo. In ICML.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2017. Deep variational information bottleneck. In ICLR.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CONLL.
Yuri Burda, Roger B. Grosse, and Ruslan R. Salakhutdinov. 2016. Importance weighted autoencoders. In ICLR.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In  $ICLR$ .
Chris Cremer, Xuechen Li, and David Duvenaud. 2018. Inference suboptimality in variational autoencoders. In ICML.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. 2018. Hyperspherical variational auto-encoders. In UAI.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. 2019. Avoiding latent variable collapse with generative skip models. In AISTATS.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In EMNLP-IJCNLP.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating KL vanishing. In NAACL.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In AAAI.
J Godfrey and E Holliman. 1997. Switchboard-1 release 2: Linguistic data consortium. In SWITCHBOARD: A User's Manual.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. In Transactions of the Association of Computational Linguistics. MIT Press.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In *ICLR*.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. In Neural computation. MIT Press.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.
Yoon Kim, Sam Wiseman, Andrew C. Miller, David A Sontag, and Alexander M. Rush. 2018. Semi-amortized variational autoencoders. In ICML.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In ICLR.
Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In ICLR.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. In NeurlIPS.
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In EMNLP-IJCNLP.
Xuezhe Ma, Chunting Zhou, and Eduard Hovy. 2019. MAE: Mutual posterior-divergence regularization for variational autoencoders. In ICLR.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In ICML.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. 2017. Neural discrete representation learning. In NeurlIPS.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-VAEs. In *ICLR*.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In ICML.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018. How does batch normalization help optimization? In NeurIPS.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. In EMNLP.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NeurlIPS.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In NeurIPS.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder variational autoencoders. In NeurlIPS.
Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In ICLR.
Akash Srivastava and Charles Sutton. 2018. Variational inference in pachinko allocation machines. In arXiv preprint arXiv:1804.07944.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. 2018. Wasserstein autoencoders. In ICLR.
Jakub M. Tomczak and Max Welling. 2018. Vae with a vampprior. In AISTATS.
Jiacheng Xu and Greg Durrett. 2018. Spherical latent spaces for stable variational autoencoders. In EMNLP.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. 2017. Variational autoencoder for semi-supervised text classification. In AAAI.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In ICML.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2017a. Infovae: Information maximizing variational autoencoders. In arXiv preprint arXiv:1706.02262.
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In ACL.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017b. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL.
Qile Zhu, Zheng Feng, and Xiaolin Li. 2018. Graphbtm: Graph enhanced autoencoded variational inference for biterm topic model. In EMNLP.
2645
A.1 Experiments on Synthetic Data
We follow the Agg-VAE and construct the synthetic data to validate whether our approach can avoid the lagging problem. VAE used in this synthetic task has a LSTM encoder and a LSTM decoder. We use a scalar latent variable because we need to compute  $\mu_{x,\theta}$  which is approximated by discretization of  $p_{\theta}(z|x)$ . To visualize the training progress, we sample 500 data points from the validation set and show them on the mean space.
We plot the mean value of the approximate posterior and the model posterior during training for the basic VAE and BN-VAE. As shown the first column in Fig. 1, all points have the zero mean of the model posterior (the x-axis), which indicates that  $\mathbf{z}$  and  $\mathbf{x}$  are independent at the beginning of training. For the basic VAE, points start to spread in the x-axis during training while sharing almost the same y value, since the model posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  is well learned with the help of the autoregressive decoder. However, the inference posterior  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is lagging behind  $p_{\theta}(\mathbf{z}|\mathbf{x})$  and collapses to the prior in the end. Our regularization approximated by BN, on the other hand, pushes the inference posterior  $q_{\phi}(\mathbf{z}|\mathbf{x})$  away from the prior  $(p(\mathbf{z}))$  at the initial training stage, and forces  $q_{\phi}(\mathbf{z}|\mathbf{x})$  to catch up with  $p_{\theta}(\mathbf{z}|\mathbf{x})$  to minimize  $KL(q_{\phi}(\mathbf{z}|\mathbf{x})||p_{\theta}(\mathbf{z}|\mathbf{x}))$  in Eq. 9. As in the second row of Fig. 1, points spread in both directions and towards the diagonal.
We also report the results on different  $\gamma$ 's with different batch sizes (32 in Fig. 1). Fig. 2 shows the training dynamics. Both settings of  $\gamma$  avoid posterior collapse efficiently. A larger  $\gamma$  produces more diverse  $\mu$ 's which spread on the diagonal. However, a small  $\gamma$  results in a small variance for the distribution of  $\mu$ , thus  $\mu$ 's in the bottom row are closer to the original (mean of the distribution). When  $\gamma$  is 0, posterior collapse happens. Different batch sizes do not diff a lot, so 32 is a decent choice. An intuitive improvement of our method is to automatically learn different  $\gamma$  for different latent dimensions, which we leave for future work.
A.2 Proof in CVAE
The KL can be computed as:
$$
\begin{array}{l} K L = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(\frac {\sigma_ {q i} ^ {2} + \left(\mu_ {q i} - \mu_ {p i}\right) ^ {2}}{\sigma_ {p i} ^ {2}}\right) \tag {14} \\ + \sigma_ {p i} ^ {2} + \mu_ {p i} ^ {2} - \log \sigma_ {q i} ^ {2} - 1). \\ \end{array}
$$
We need to prove that KL will not achieve the minimum number when  $\mu_{pi}$  equals to  $\mu_{qi}$  and  $\sigma_{pi}$  equals  $\sigma_{qi}$ . We take hidden size as 1 for example. The binary function about  $\mu_{pi}$  and  $\sigma_{pi}$  is:
$$
\begin{array}{l} f _ {\mu_ {p i}, \sigma_ {p i}} = \left(\frac {\sigma_ {q i} ^ {2} + \left(\mu_ {q i} - \mu_ {p i}\right) ^ {2}}{\sigma_ {p i} ^ {2}}\right) \tag {15} \\ + \sigma_ {p i} ^ {2} + \mu_ {p i} ^ {2} - l o g \sigma_ {q i} ^ {2} - 1), \\ \end{array}
$$
the maxima and minima of  $f_{\mu_{pi},\sigma_{pi}}$  must be the stationary point of  $f_{\mu_{pi},\sigma_{pi}}$  due to its continuity. The stationary point is: