SlowGuess/NAACL_2022_OCR huggingface dataset:
Finetuned Pretrained Models. To manifest the suitability of DELLA for different pretrained language models, we compare it with fine-tuned GPT2 on unconditional generation and story generation, and with fine-tuned BART-base on summarization generation and paraphrase generation.
Optimus (Li et al., 2020a): a large-scale VAE model which takes a pre-trained BERT as encoder and pretrained GPT-2 as decoder. This model is first pretrained as a VAE, which simultaneously utilizes the two paradigms, Embedding and Memory as introduced in Sec. 3.3, for injecting latent variables, with both KL annealing and KL threshold tricks, and then fine-tuned on downstream tasks.
Transformer-based VAE. Besides Optimus, we also comprise the three paradigms, namely Embedding (Li et al., 2020a), Memory (Fang et al., 2021) and Softmax (Wang and Wan, 2019), and incorporate each paradigm into the same pre-trained model as DELLA on each dataset for fair comparison.
For unconditional generation tasks, we consider three types of metrics. (a) Representation Learning Capability: we report PPL, ELBO, KL, mutual information (MI) (Alemi et al., 2016) and activate units (AU) (Burda et al., 2016). These metrics measure VAE's ability to mitigate KL vanishing and learn meaningful representations. Different from traditional language models like GPT-2, VAE-based models could not produce exact PPL due to randomness, so we use importance-weighted sam
$^{2}$ https://en.wikipedia.org/wiki/Interaction_information
<sup>3</sup>https://quoradata.quora.com/First-Quora-Dataset
Release-Question-Pairs
Model	Representation Learning	Generation Quality	Generation Diversity
PPL↓	ELBO↓	KL↑	MI↑	AU↑	BLEU↑	CND↓	MAUVE↑	SB↓	Dist↑	JS↓
Dataset: Yelp
GPT-2	22.13	-	-	-	-	56.92	0.68	0.12	65.90	17.96	0.51
Optimus	22.79	344.10	15.09	7.67	-	-	-	-	-	-	-
Embed	19.98	327.28	4.77	4.14	6	56.34	0.31	0.42	65.27	15.59	0.44
Memory	19.95	326.60	5.70	5.30	11	57.37	0.27	0.46	63.90	16.91	0.39
Softmax	20.14	328.13	7.50	6.29	13	56.83	0.30	0.45	64.26	16.51	0.40
DELLA	12.35	239.83	29.47	10.78	23	57.15	0.13	0.55	60.02	17.63	0.43
Dataset: Yahoo
GPT-2	24.17	-	-	-	-	44.25	0.55	0.15	54.06	21.07	0.28
Optimus	23.11	293.34	17.45	8.85	-	-	-	-	-	-	-
Embed	22.18	286.85	3.63	3.03	3	42.27	0.45	0.31	54.15	20.80	0.32
Memory	22.03	285.47	4.87	4.62	18	45.20	0.46	0.37	54.59	21.87	0.33
Softmax	22.35	287.44	6.35	5.52	19	44.28	0.44	0.34	54.49	21.65	0.32
DELLA	11.49	201.34	27.84	12.31	21	44.67	0.19	0.38	48.53	21.88	0.31
Table 1: Evaluation results for language modelling and unconditional generation. Results of Optimus are directly copied from the original paper with  $\lambda = {0.5}$  . SB means Self-BLEU.
ples to estimate PPL, following He et al. (2019). We set the threshold in AU to 0.2 to further distinguish different models. (b) Generation Quality: we report BLEU (Papineni et al., 2002), CND (Li et al., 2020b) and MAUVE (Pillutla et al., 2021). CND and MAUVE measure the divergence between human-authored text and the generated one. (c) Generation Diversity: we report Self-BLEU (Zhu et al., 2018), Dist (Li et al., 2016) and JS (Jaccard similarity) (Wang and Wan, 2018) to assess the diversity and novelty of generated text.
For conditional generation tasks, we report BLEU, Rouge-1, Rouge-2, Rouge-L (Lin and Hovy, 2002), and BERTScore (Zhang et al., 2020) to evaluate the quality of generated texts, as well as the same diversity metrics used in unconditional generation. We also report KL and AU value to present representation learning capability. More details of metrics are provided in Appendix A.3.
5.5.1 Unconditional Generation
We present results on Yelp and Yahoo in Table 1 and leave the those on PTB and SNLI in the Appendix A.5 due to space limitations. We also show the learning curves of ELBO and KL in Fig. 5.
As shown in Table 1, DELLA achieves notably improvement on almost all the metrics, especially superior on representation learning metrics. Much higher KL, MI and AU, and a big gap in PPL obtained by DELLA indicate the latent variables encode more meaningful text information and won't diminish when propagating through Transformer
layers, which strongly supports our motivation that fusing latent variables with hidden states more deeply could effectively alleviate the KL vanishing problem. Such results also empirically verify the theoretical advantage of our model (Theorem 1), demonstrating entangled layer-wise latent variables can preserve more encoded knowledge for decoder. We will show that  $z$  can involve more information when injected into more layers in Sec. 5.8.
Besides, DELLA also gets good performance (comparable BLEU and much better CND and MAUVE) on generation quality. With more informative latent variables, DELLA could achieve a better ELBO and hence further boost the learning of data distribution  $p(x)$  in Eq.(2), leading to satisfactory quality of generated texts.
Generally, DELLA also outperforms baseline models on generation diversity. The reason is twofold: randomly sampled latent variables  $z$  should bring diversity, while the VAE-based baselines tend to ignore  $z$  as mentioned before, losing some randomness. In contrast, latent variables are deeply fused in DELLA, maintaining enough randomness. Besides, each latent variable is sampled in corresponding layer, and thus such a sampling process accumulates and enhances randomness, further benefiting diversity while keeping good quality.
5.5.2 Conditional Generation
We report the results of WP and CNN/DM in Table 2, and leave those of Quora in Appendix A.5. As we can see, DELLA performs better on most quality metrics, but gets a little worse on diversity
Model	Quality	Diversity	KL↑	AU↑
BLEU↑	Rouge-1↑	Rouge-2↑	Rouge-L↑	BERTScore↑	SB↓	Dist↑	JS↓
Dataset: WritingPrompts
GPT-2	27.89	27.72	7.96	14.30	78.12	53.78	22.99	0.51	-	-
Embed	39.67	36.17	7.96	15.78	81.64	64.55	14.31	0.73	2.35	3
Memory	40.79	36.13	8.04	16.16	81.68	67.56	12.90	0.80	0.07	0
Softmax	41.04	36.14	8.12	16.30	81.75	67.02	13.08	0.78	0.32	0
DELLA	41.39	35.46	8.78	17.20	81.77	56.28	20.91	0.60	28.14	8
Dataset: CNN/DM
Bart-base	48.74	41.33	19.82	29.63	87.75	29.94	43.68	0.10	-	-
Embed	44.10	40.43	19.41	29.43	87.60	29.60	44.04	0.10	0.0	0
Memory	46.02	41.18	19.74	29.64	87.78	29.79	43.92	0.11	0.0	0
Softmax	44.40	40.94	19.63	29.61	87.00	29.64	44.11	0.10	0.0	0
DELLA	49.18	41.27	19.85	29.84	88.09	29.07	44.24	0.09	0.91	1
Table 2: Evaluation results for conditional generation.
Dataset: WritingPrompts
Model	Fluency	Coherence	Novelty
GPT2	1.83	2.12	2.50
Embed	2.16	2.33	2.67
Memory	2.45	2.28	2.78
Softmax	2.48	2.42	2.85
DELLA	2.51	2.38	2.89
Dataset: CNN/DM
Model	Informativeness	Coherence	Novelty
Bart-base	3.12	4.32	3.52
Embed	2.88	4.08	3.50
Memory	2.95	4.23	3.48
Softmax	2.91	4.33	3.50
DELLA	3.05	4.33	3.56
compared to GPT-2. This is because GPT-2 may produce some ill-formed contents which 'improve' diversity by cheating the metrics but also lead to much worse quality (lower BLEU and Rouge). Even so, on both WP and CNN/DM, DELLA still beats all previous VAE paradigms in diversity, manifesting the effectiveness of our DELLA.
In addition, all baselines methods suffer from severer KL vanishing problems on conditional generation tasks than on the unconditional ones. This is because the given condition text could aggravate the reliance of these models on preceding generated tokens and the condition, and therefore bypass latent variables. By contrast, DELLA could learn more informative  $z$  and hence keep a relatively higher KL value even given the condition text.
Table 3: Human evaluation results on conditional generation. The scores range from 1 (worst) to 5 (best). The p-value is 0.002 and Kappa score is 0.64 which indicates acceptable inter-annotator agreement.
Model	PPL↓	ELBO↓	KL↑	MI ↑	AU↑
DELLA	12.35	239.83	29.47	10.78	23
-LTP	12.68	249.32	28.52	9.77	21
-LW	19.88	324.45	20.12	7.23	18
Separate	14.17	286.30	28.82	9.88	16
l = 1 KL	12.55	266.97	0.15	0.15	0
l = 12 KL	12.48	263.38	0.73	0.61	0
Embed(384)	20.11	327.29	0.55	0.38	0
Memory(384)	20.09	326.24	0.46	0.25	0
Softmax(384)	20.15	330.24	5.04	7.15	0
Table 4: Ablation study on Yelp dataset. LTP: low-rank tensor product. LW: layer-wise latent variables. Separate: latent variables in each layer are independent.  $l = 1$  or  $4\mathrm{{KL}}$  means we only compute  $\mathrm{{KL}}$  loss on  ${z}_{1}$  or  ${z}_{L}$  ,respectively. 384 means the dimension of latent variable used in baseline are  ${12} \times  {32} = {384}$  .
5.6 Human Evaluation
To better verify the effectiveness of DELLA, we also conduct human evaluation on the two conditional generation tasks. For each model, we generated 30 samples on each task, and invite 5 competent annotators to score these samples in terms of three criteria, Fluency, Coherence and Novelty for story generation, and Informativeness, Coherence and Novelty for summarization generation.
As shown in Table 3, DELLA obtains satisfactory performance in quality, and is consistently superior to all baselines on diversity and novelty. See Appendix A.4 for more detailed evaluation protocols.
Table 4 shows the results of ablation study on Yelp. We can find both tensor product and the layer-wise latent variables benefit the learning of informative
Figure 2: PPL, ELBO end KL on Yelp with different numbers of latent variables. The values start layer  $i$  and end layer  $j$  means latent variables are produced and utilized only from  $i$ -th layer to the last layer, or from the first layer to  $j$ -th layer of the encoder respectively.
Model	PPL↓	ELBO↓	KL↑	MI ↑	AU↑
Embed	22.21	339.12	0.03	0.03	0
+BOW	19.98	326.51	2.75	2.48	4
+Annealing	19.98	327.28	4.77	4.14	6
+Annealing + BOW	20.59	332.44	19.51	9.12	28
+Annealing + BN	21.14	338.59	21.09	8.98	25
Memory	22.16	338.68	0.00	0.01	0
+BOW	19.87	326.00	3.89	3.59	8
+Annealing	19.95	326.60	5.70	5.30	11
+Annealing + BOW	20.41	331.09	18.76	9.14	28
+Annealing + BN	20.25	331.59	18.11	9.07	24
Softmax	22.43	333.93	0.47	0.3	0
+BOW	20.53	331.89	10.16	5.57	28
+Annealing	20.14	328.13	7.50	6.29	13
+Annealing + BOW	21.14	335.48	17.51	8.46	28
+Annealing + BN	20.95	337.10	21.25	9.15	25
DELLA	17.18	312.45	9.39	5.32	6
+BOW	13.98	289.94	11.59	9.25	8
+Annealing	12.35	239.83	29.47	10.78	23
+Annealing+BOW	12.82	249.98	32.79	11.26	26
Backbone: GPT-2 medium (24 layers)
Embed	18.33	317.44	2.13	1.44	3
Mem	18.30	317.24	4.47	4.26	10
Softmax	18.47	318.80	5.80	5.03	12
DELLA	11.01	230.96	17.09	23.69	27
Table 5: Results on Yelp for transformer-bsaed VAE with BOW loss, KL annealing and batch normalization tricks, and use 24-layer GPT2-medium as backbone. Here we fix  $\gamma$  in batch normalization as 1.
latent variables, while the latter contributes the most to DELLA. To further verify the performance gain originating from Theorem 1 instead of simply increasing the number or the dimension of latent variables, we conduct two groups of experiments.
First, we remove the conditional dependence between layer-wise latent variables by independently sampling each  $z_{l}$  in both training and testing. We can see that removing dependence causes a significant performance drop. Besides, we keep the dependence between  $z_{l}$  but optimize only one of the KL terms in Eq.(6), and find all representation
capability metrics deteriorate, especially KL, MI and AU. Such results effectively demonstrate the necessity of using and optimizing the conditional inference of layer-wise latent variables, supporting our theoretical interpretation of DELLA.
Second, we enlarge the dimension of  $z_{l}$  used in the three paradigms to 384 (12 × 32), equal to the total latent dimension used in DELLA. The results show that simply increasing the dimension of latent variables brings a more sparse latent space, even exacerbating the KL vanishing problem.
5.8 Analysis
Training Tricks To reveal the robustness of our model, we evaluate the influence of three commonly used training tricks to relieve KL vanishing, i.e., BOW (bag-of-words) loss (Wang et al., 2017), batch normalization (Zhu et al., 2020) and KL annealing (Fu et al., 2019), to the performance of DELLA and the three paradigms. As shown in Table 5, previous methods suffer KL vanishing seriously without annealing or BOW loss, getting KL, MI and AU almost 0. Though not good as using annealing, DELLA still maintains acceptable performance and mitigates KL vanishing even without any training tricks. Bow and batch normalization dramatically prevent low KL divergence, but obstruct the optimization and thus cause higher PPL.
Number of Latent Variables We observe the change of PPL, KL and ELBO with different numbers of latent variables. We conduct two groups of experiments where we produce and utilize layerwise latent variables starting from and ending at different layers. As shown in Fig. 2, incorporating more latent variables could continuously improve performance, consistent to our claim in Sec. 4. With the same number of latent variables, starting from a higher layer is better than ending at a lower layer, which indicates that latent variables
Sentence 1: A man is taking beautiful photos by the river
0.1: A man is taking photos by the beautiful river
0.3: A man is taking pictures by the river
0.5: A man is enjoying the picnic by the river
0.7: A goat is enjoying the day by the river
0.9: A goat is enjoying the day on the farm
Sentence 2: A goat is enjoying the sun on the farm
Sentence 1: Two girls are playing baseball on the beach
0.1: Two girls are playing basketball on the beach.
0.3: Two girls are playing music in the park.
0.5: Two girls are singing on the beach.
0.7: Crowds of people are dancing in the park
0.9: Lots of people are dancing in the park
Sentence 2: Many people are dancing in front of building
Prompt: After receiving perfect score on your final exam, your school is asking you to give a speech on the subject of the exam. Little does anyone know you guessed every single answer.
Memory: The room was silent except for me. I could hear some whispering behind us as we continued to pace about campus, but there was no need. We knew better than that any moment that would come, and at least if they were around, we wouldn't have to be so paranoid.
DELLA: "Hey mister!" I just got back from work and the whole class was already ready for question. So go ahead I took the stage. The teacher smiled encouragingly. Standing up in the front of desk, I started talking about all sorts of things about answering test question.
Figure 3: Interpolating latent space. The sentence in each row is generated with a latent variable interpolated from those of sentence 1 and sentence 2.
Figure 4: Generation examples of Memory and DELLA based on the prompt from test set of WritingPrompts.
Figure 5: Reconstruction loss and KL Divergence throughout training process.
generated from higher layers encode more helpful information compared to those from lower layers, manifesting disadvantages of the two previous paradigms, Softmax (starting from the last layer) and Embedding (ending at the first layer).
Model size We compare the performance of DELLA and three paradigms with 24-layer GPT2-medium as backbone. As shown in Table 5, with the increasing of model size, DELLA consistently achieves better performance than baselines.
5.9 Case Study
VAE captures text representations in a smooth latent space. We take two sentences  $x_{1}$  and  $x_{2}$  and
sample two posterior latent variables  $\pmb{z}^{(1)}$  and  $\pmb{z}^{(2)}$  from  $p(\pmb{z}^{(1)}|\pmb{x}_1)$  and  $p(\pmb{z}^{(2)}|\pmb{x}_2)$ , and get interpolated latent variables with  $\pmb{z} = \tau \pmb{z}^{(1)} + (1 - \tau)\pmb{z}^{(2)}$ . We generate multiple sentences with a continuously changed  $\tau$  from 0 to 1. As shown in Fig. 3, sentences generated from interpolated  $\pmb{z}$  mix the semantics of the two initial sentences and smoothly change from  $\pmb{x}_1$  to  $\pmb{x}_2$ , showing DELLA's ability of learning a flexible latent space.
Fig. 4 shows the generation examples of DELLA and one of baseline, Memory, given the same prompt WritingPrompts. We observe that the generated text of Memory is irrelevant to the prompt, while DELLA generates coherent and vivid text.
In this paper, we propose a novel variational Transformer framework DELLA. Our framework learns a series of layer-wise latent variables with iterative dependence. These latent variables are conditionally inferred and injected into corresponding decoder layers by low-rank tensor product for deeper fusion. The experiments on both unconditional and conditional generation tasks demonstrate DELLA's ability to significantly mitigate KL vanishing and improve generated text's quality and diversity. In the future, we plan to explore further the potential of DELLA in larger pretrained language models.
Thanks for the anonymous reviewers for their comments. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502) and International Innovation Center of Tsinghua University, Shanghai, China.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2016. Deep variational information bottleneck. In International Conference on Learning Representations.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In International Conference on Learning Representations.
Rewon Child. 2020. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. 2019. Avoiding latent variable collapse with generative skip models. In International Conference on Artificial Intelligence and Statistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. 2021. Transformer-based conditional variational autoencoder for controllable story generation. arXiv: Computation and Language.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating KL vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers), pages 240-250, Minneapolis, Minnesota. Association for Computational Linguistics.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2017. A deep generative framework for paraphrase generation. In National Conference on Artificial Intelligence.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880-5894, Florence, Italy. Association for Computational Linguistics.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lapping inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Representations.
Irina Higgins, Loic Matthew, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning, pages 1587-1596. PMLR.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In International Conference on Learning Representations.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-4751.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. 2019. Learning hierarchical priors in vaes. In Neural Information Processing Systems.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.