SlowGuess/NAACL_2022_OCR huggingface dataset:
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3603-3614, Hong Kong, China. Association for Computational Linguistics.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiu-jun Li, Yizhe Zhang, and Jianfeng Gao. 2020a. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4678-4699, Online. Association for Computational Linguistics.
Jianing Li, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2020b. On the relation between quality-diversity evaluation and distribution-fitting goal in text generation. In International Conference on Machine Learning.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Juntao Li, Yan Song, Haisong Zhang, Dongmin Chen, Shuming Shi, Dongyan Zhao, and Rui Yan. 2018. Generating classical Chinese poems via conditional variational autoencoder and adversarial training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3890-3900, Brussels, Belgium. Association for Computational Linguistics.
Chin-Yew Lin and Eduard Hovy. 2002. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages 45-51, Phildadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Zhaojiang Lin, Genta Indra Winata, Peng Xu, Zihan Liu, and Pascale Fung. 2020. Variational transformers for diverse response generation. arXiv: Computation and Language.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2247-2256, Melbourne, Australia. Association for Computational Linguistics.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated
corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In Neural Information Processing Systems.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-vaes. In International Conference on Learning Representations.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 627-637, Copenhagen, Denmark. Association for Computational Linguistics.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv: Computation and Language.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder variational autoencoders. In Neural Information Processing Systems.
Arash Vahdat and Jan Kautz. 2020. Nvae: A deep hierarchical variational autoencoder. In Neural Information Processing Systems.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Neural Information Processing Systems.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ke Wang and Xiaojun Wan. 2018. Sentigan: Generating sentimental texts via mixture adversarial networks. In International Joint Conference on Artificial Intelligence.
Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. 2017. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In Neural Information Processing Systems.
Tianming Wang and Xiaojun Wan. 2019. T-cvae: transformer-based conditioned variational autoencoder for story completion. In International Joint Conference on Artificial Intelligence.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In International conference on machine learning, pages 3881-3890. PMLR.
Xiaoyuan Yi, Ruoyu Li, Cheng Yang, Wenhao Li, and Maosong Sun. 2020. Mixpoet: Diverse poetry generation via learning controllable mixed latent space. In Proceedings of The Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, USA.
Meng-Hsuan Yu, Juntao Li, Danyang Liu, Dongyan Zhao, Rui Yan, Bo Tang, and Haisong Zhang. 2020. Draft and edit: Automatic storytelling through multi-pass hierarchical conditional variational autoencoder. In National Conference on Artificial Intelligence.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 654-664, Vancouver, Canada. Association for Computational Linguistics.
Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020. A batch normalized inference network keeps the KL vanishing away. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2636-2649, Online. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. In International ACM SIGIR Conference on Research and Development in Information Retrieval.
We load pretrained model GPT-2 (Radford et al., 2019) as initial parameters for unconditional generation and story generation, and pretrained BART-base (Lewis et al., 2020) for summarization and paraphrasing generation tasks. For the summarization and paraphrasing generation, we keep the encoder-decoder attention block. No encoder-decoder attention is used in unconditional generation and story generation tasks. The number of layers and dimensions of hidden states in DELLA is consistent with the configurations of corresponding pretrained models (GPT-2 has 12 layers and Bart-base has 6-layer encoder and 6-layer decoder. The hidden size of both is 768). We use the state of a special token to obtain the representation in the encoder. We utilize cyclical annealing tricks to train DELLA and other VAE baselines. Specifically, two epochs are one annealing period. In one period,  $\beta$  (the weight of KL term in ELBO) keeps 1e-5 in the first half, then linearly increases to 1 in the next quarter, then keeps at 1 for the last quarter. We select batch size over \{16, 32\} and learning rate over \{5e-5, 7e-5\}. We use beam search for DELLA and top-k sampling for compared baseline models for the unconditional generation and story generation. For the summarization and paraphrasing generation, we use beam search in all the models.
We implement DELLA and other VAE baselines based on Huggingface Transformers (Wolf et al., 2020) library of v4.10.0 and use NVIDIA GeForce RTX 3090 to train our model. The total number of training GPU hours on different datasets is in Table 6. The number of parameters for our model is 193,353,984 in the unconditional generation setting and 195,180,114 in the conditional generation one. All experimental results are trained and tested in a single run.
Dataset	Training Time
Yelp	20h
Yahoo	20h
PTB	6h
SNLI	12h
CNN/DM	40h
WP	170h
Quora	5h
Table 6: GPU hours of training DELLA with RTX3090
Dataset	# Train	# Dev	# Test	Avarage Length
Yelp	100k	10k	10k	96
Yahoo	100k	10k	10k	79
PTB	42k	3k	3k	21
SNLI	100k	10k	10k	10
CNN/DM	287k	13k	11k	S: 790	T: 61
WP	272k	15k	15k	S: 28	T: 674
Quora	134k	5k	10k	S: 10	T: 10
Table 7: Statistics of datasets. We present the size of train/dev/test sets and the average length for 7 datasets. S means source text and T means target text.
A.2 Datasets Details
The detailed dataset statistics are in Table 7. For the licenses of the datasets we use, CNN/DM and WritingPrompts use MIT License, while SNLI uses CC BY-SA 4.0. Meanwhile, PTB, Quora, and Yelp use their own license: LDC User Agreement, Yelp Data Agreement, and Quora's Terms of Service, respectively. All of these licenses and agreements allow their data for academic use. Unfortunately, we did not find the license for the Yahoo Dataset.
A.3 Metrics Details
Here we provide more details of the metrics used in our experiments.
Perplexity (PPL).  $\mathrm{PPL} = p(\pmb{x})^{-1 / n}$  is commonly used to evaluate the performance of language models, where  $n$  is number of tokens  $\pmb{x}$  contains. For VAE-based model, we can only obtain the lower bound of  $\log p(\pmb{x})$ . We consider  $k$  latent variables  $z_{1}, z_{2}, \ldots, z_{k}$  sampled from the posterior distribution  $q(z_{i}|\pmb{x})$ . Based on the fact that average importance weights are an unbiased estimator of  $\log p(\pmb{x})$  (Burda et al., 2016) and Jensen's Inequality, we have:
$$
\begin{array}{l} \mathcal {L} _ {k} = \mathbb {E} \left[ \log \frac {1}{k} \sum_ {i = 1} ^ {k} \frac {p \left(\boldsymbol {x} , \boldsymbol {z} _ {i}\right)}{q \left(\boldsymbol {z} _ {i} \mid \boldsymbol {x}\right)} \right] \tag {13} \\ \leq \log \mathbb {E} \left[ \frac {1}{k} \sum_ {i = 1} ^ {k} \frac {p (\boldsymbol {x} , \boldsymbol {z} _ {i})}{q (\boldsymbol {z} _ {i} | \boldsymbol {x})} \right] = \log p (\boldsymbol {x}). \\ \end{array}
$$
We use  $\mathcal{L}_k$  to estimate  $\log p(\pmb{x})$  and calculate PPL.
ELBO. The ELBO is the sum of reconstruction loss and KL divergence.
KL. The KL divergence of the posterior and prior distribution.
Mutual Information(MI) (Alemi et al., 2016).
Mutual Information  $\mathcal{I}(\pmb{x},\pmb{z})$  is defined as:
$$
\begin{array}{l} \mathcal {I} _ {q} (\boldsymbol {x}, \boldsymbol {z}) \tag {14} \\ = \mathbb {E} _ {p (\boldsymbol {x})} \mathbb {E} _ {q (\boldsymbol {z} | \boldsymbol {x})} \log q (\boldsymbol {z} | \boldsymbol {x}) - \mathbb {E} _ {q (\boldsymbol {z})} \log q (\boldsymbol {z}) \\ \end{array}
$$
where  $q(\boldsymbol {z}) = \mathbb{E}_{p(\boldsymbol {x})}q(\boldsymbol {z}|\boldsymbol {x})$  is called the aggregated posterior.
Activate Units(AU) (Burda et al., 2016). AU is the active units in latent variables, defined as  $A_{z} = \mathrm{Cov}_{\boldsymbol{x}}(\mathbb{E}_{z \sim q(z|\boldsymbol{x})}[z]) > \delta$ , where  $\delta$  is a threshold, commonly set as 0.01. However, we find that with  $\delta = 0.01$ , all VAE models in our experiments have full active unit. So we increase the threshold to 0.2 to distinguish the performance of different models on this metric. Please note that DELLA incorporates latent variables in all layers, and hence we calculate AU for the latent variable in each layer and then report the average.
BLEU (Papineni et al., 2002). BLEU measures the n-gram overlap of generated sequences and the reference ones. For unconditional setting, we regard all samples in the test set as references to each generated example.
CND (Li et al., 2020b). CND approximates the divergence of the empirical reference distribution and generated text distribution in n-gram spaces.
MAUVE (Pillutla et al., 2021). MAUVE measures the gap between reference text and generated text using divergence frontiers.
Self-BLEU (Zhu et al., 2018). Self-Bleu calculates the BLEU score on the generated samples, which averages the BLEU score of each generated sequence calculated with other generated ones as references. This metric measures the diversity of a set of generated sequences. Higher Self-BLEU means these generated sequences are more distinguishable from each other.
Dist (Li et al., 2016). Dist measures the proportion of distinct n-grams on generated samples.
Jaccard Similarity(JS) (Wang and Wan, 2018). JS calculates the average n-gram Jaccard similarity between every two generated sequences.
Rouge (Lin and Hovy, 2002). Rouge computes n-gram overlap of generated examples with given target samples. We use rouge-score v0.0.4 to evaluate the rouge score of our model and the baselines.
BERTScore (Zhang et al., 2020). BERTScore uses pre-trained BERT (Devlin et al., 2019) to obtain the vector representations of generated and reference text and calculates their cosine similarity. We use bert-score v0.3.10 to calculate the BERTScore of our model and the baselines.
A.4 Human Evaluation Details
Due to the relatively long length of generated text, we randomly sample 30 examples in the test set of WP and CNN/DM as input to DELLA and other compared baseline models to generate the target. We invite five graduate students proficient in English to score the generated text. The criteria for story generation include fluency, coherence, and novelty, and the criteria for summarization generation include informativeness, consistency, and novelty. Specifically, fluency measures whether the generated sentences are syntactically fluent; coherence measures whether the generated text is logically structured and consistent with the input text; novelty measures whether the content is novel and attractive; informativeness measures to what extent the generated summarization summarizes the general idea of the article.
When conducting the human evaluation, we informed the participants as follows:
- The following contents are generated by the automatic models. Some of them may be offensive or contain improper arguments. Please be conscious of these risks and evaluate these contents equitably and adequately.
- The evaluation you provide will be used only for academic use and will never be used commercially.
Every evaluator will sign their signature below these warnings to confirm that they have read those words. After finishing the annotation, they will receive $25. This amount is determined by the time of the whole annotation process and the estimation of average hourly income. The ethics review board for data collection protocol is not essential in our country, so we did not conduct this review for our data collection protocol.
A.5 Additional Experimental Result
Table 8 and Table 9 report the results on PTB, SNLI and Quora dataset.
A.6 Case Study Details
We take two sentences  $x_{1}$  and  $x_{2}$  and sample two groups of latent variables  $z^{(1)} = \{z_1^{(1)},z_2^{(1)},\dots ,z_L^{(1)}\}$  and  $z^{(2)} = \{z_1^{(2)},z_2^{(2)},\dots ,z_L^{(2)}\}$  from posterior distributions  $p(\pmb {z}^{(1)}|\pmb {x}_1)$  and  $p(\pmb {z}^{(2)}|\pmb {x}_2)$ . We obtain the weighted latent variables  $\hat{\pmb{z}} = \{\hat{\pmb{z}}_1,\hat{\pmb{z}}_2,\dots ,\hat{\pmb{z}}_L\}$  by
taking weighted sum at each corresponding element in two groups, i.e.  $\hat{\pmb{z}}_i = \tau *\pmb{z}_i^{(1)} + (1 - \tau)*\pmb{z}_i^{(2)}$ . The mixed sentence  $\hat{\pmb{x}}$  is generated conditioned on  $p(\hat{\pmb{x}} |\hat{\pmb{z}})$  by the decoder.
A.7 Potential Risks and Limitations of our work
Due to the unclean corpus (especially in the WP dataset) we use where slang repeatedly appears, the model training on this corpus may also output some rude expressions during generation. Also, the text generated in the unconditional generation task is not controllable, which may contain some bias or politically sensitive expression. Besides, since our model significantly improves the quality and diversity of generated, it can produce more plausible texts like news, which could be possibly utilized to create fake news or disinformation. However, on the other hand, our model could benefit fairness in language generation. Previous text generation models tend to produce biases like gender or nationality biases, which means only the majority would be appropriately described while the minority may be ignored. These biases are mainly caused by the biased training corpus. With the same data, our model can improve the diversity of generated text, which is also potential for mitigating these biased. We will try to develop debiased language generation systems in future work to avoid these risks harming society.
While DELLA shows good performance on text generation, it has one limitation: training efficiency. DELLA brings more parameters compared with three baseline methods. Training efficiency needs to be considered if we further explore the performance of DELLA on the large pretrained model.
Model	Representation Learning	Generation Quality	Generation Diversity
PPL↓	ELBO↓	KL↑	MI↑	AU↑	BLEU↑	CND↓	MAUVE↑	SB↓	Dist↑	JS↓
Dataset: PTB
GPT-2	25.80	-	-	-	-	27.91	1.12	0.73	41.55	37.79	0.30
Optimus	22.79	344.10	15.09	7.67	-	-	-	-	-	-	-
Embed	19.98	327.28	4.77	4.14	6	28.04	1.38	0.69	41.32	34.46	0.33
Memory	24.41	90.25	1.22	1.17	4	21.31	1.21	0.58	26.58	38.28	0.08
Softmax	24.04	90.63	2.13	1.89	21	28.59	1.39	0.72	42.15	33.91	0.30
DELLA	10.28	58.43	12.46	12.35	22	28.15	0.63	0.68	24.87	41.84	0.17
Dataset: SNLI
GPT-2	20.19	-	-	-	-	63.57	1.95	0.71	75.34	19.11	0.58
Optimus	16.67	38.50	16.35	8.89	-	-	-	-	-	-	-
Embed	13.79	32.97	3.24	3.16	20	59.26	0.98	0.72	65.59	20.89	0.44
Memory	13.78	32.62	2.13	2.08	10	62.80	1.24	0.67	54.59	21.87	0.33
Softmax	14.21	33.18	2.70	2.65	16	60.51	1.94	0.71	71.84	18.59	0.57
DELLA	5.13	10.23	5.86	16.58	23	62.94	0.85	0.69	36.85	32.61	0.21
Table 8: Additional results for language model and unconditional generation task. The results of Optimus are copied from original paper with  $\lambda = 0.5$ .
Model	BLEU↑	Rouge-1↑	Rouge-2↑	Rouge-L↑	Bertscore↑	KL↑
Bart-base	64.34	63.27	39.83	60.28	94.72	-
Embed	63.94	63.12	39.42	60.22	94.66	0.0
Mem	63.78	62.86	39.18	59.96	94.65	0.0
Softmax	64.30	63.25	39.92	60.39	94.71	0.0
DELLA	64.40	63.80	40.58	61.03	94.84	3.88
Table 9: Results on Quora dataset. Because the sentences in Quora are quite short and constrained, the results of the three diversity metrics on all baselines are almost the same. So we omit them here.
B Additional Proof
B.1 Derivation of KL Divergence of Layer-Wise Latent Variables
KL divergence of layer-wise latent variables
$$
\begin{array}{l} K L (q (z | x) | | p (z)) \\ = \int q (z | x) \log \frac {q (z | x)}{p (z)} d z \\ = \int \prod_ {l = 1} ^ {L} q (\boldsymbol {z} _ {l} | \boldsymbol {x}, \boldsymbol {z} _ {<   l}) \log \frac {\prod_ {l = 1} ^ {L} q (\boldsymbol {z} _ {l} | \boldsymbol {x} , \boldsymbol {z} _ {<   l})}{\prod_ {l = 1} ^ {L} p (\boldsymbol {z} _ {l} | \boldsymbol {z} _ {<   l})} d \boldsymbol {z} _ {1} d \boldsymbol {z} _ {2} \dots d \boldsymbol {z} _ {L} \\ = \sum_ {i = 1} ^ {L} \int \prod_ {l = 1} ^ {L} q \left(\boldsymbol {z} _ {l} \mid \boldsymbol {x}, \boldsymbol {z} _ {<   l}\right) \log \frac {q \left(\boldsymbol {z} _ {l} \mid \boldsymbol {x} , \boldsymbol {z} _ {<   l}\right)}{p \left(\boldsymbol {z} _ {l} \mid \boldsymbol {z} _ {<   l}\right)} d \boldsymbol {z} _ {1} d \boldsymbol {z} _ {2} \dots d \boldsymbol {z} _ {L} \tag {15} \\ = \sum_ {l = 1} ^ {L} \int q (\boldsymbol {z} _ {<   l} | \boldsymbol {x}) q (\boldsymbol {z} _ {l} | \boldsymbol {x}, \boldsymbol {z} _ {<   l}) \log \frac {q (\boldsymbol {z} _ {l} | \boldsymbol {x} , \boldsymbol {z} _ {<   l})}{p (\boldsymbol {z} _ {l} | \boldsymbol {z} _ {<   l})} d \boldsymbol {z} _ {1} d \boldsymbol {z} _ {2} \dots d \boldsymbol {z} _ {l} \\ = \sum_ {l = 1} ^ {L} \mathbb {E} _ {q (\boldsymbol {z} _ {<   l} | \boldsymbol {x})} \mathrm {K L} (q (\boldsymbol {z} _ {l} | \boldsymbol {x}, \boldsymbol {z} _ {<   l}) | | p (\boldsymbol {z} _ {l} | \boldsymbol {z} _ {<   l})) \\ \end{array}
$$
First, we consider on term in the summation and can obtain:
where  $H$  is the Shannon entropy. Then, the summation has a lower bound:
$$
\begin{array}{l} \sum_ {i = 1} ^ {L} \mathbb {E} _ {p (\boldsymbol {x})} \mathbb {E} _ {q (\boldsymbol {z} _ {<   l} | \boldsymbol {x})} [ \mathrm {K L} (q (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {z} _ {<   l}) | | p (\boldsymbol {z} _ {l} | \boldsymbol {z} _ {<   l})) ] \\ \geq \sum_ {i = 1} ^ {L} H \left(\boldsymbol {z} _ {l} \mid \boldsymbol {z} _ {<   l}\right) - H \left(\boldsymbol {z} _ {l} \mid \boldsymbol {z} _ {<   l}, \boldsymbol {x}\right) \tag {17} \\ = H \left(z _ {1}, \dots , z _ {L}\right) - H \left(z _ {1}, \dots , z _ {L} \mid x\right) \\ = I \left(\boldsymbol {x}; \boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {L}\right) \\ \end{array}
$$
where  $I$  is mutual information. Next, we prove the following inequality with induction:
$$
I (\boldsymbol {x}; \boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {L}) \geq I (\boldsymbol {x}; \boldsymbol {z} _ {1}; \dots ; \boldsymbol {z} _ {L}) \tag {18}
$$
When  $L = 2$ , we proof  $I(\pmb{x}; \pmb{z}_1, \pmb{z}_2) \geq I(\pmb{x}; \pmb{z}_1; \pmb{z}_2)$ . Actually, we have the following facts:
$$
\begin{array}{l} I (\boldsymbol {x}; \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}) \\ = H (\boldsymbol {x}) + H \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) - H (\boldsymbol {x}, \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}) (19) \\ I (\boldsymbol {x}; \boldsymbol {z} _ {1}; \boldsymbol {z} _ {2}) \\ = H (\boldsymbol {x}) + H \left(\boldsymbol {z} _ {1}\right) + H \left(\boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) (20) \\ - H \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) - H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}\right) - H \left(\boldsymbol {x}, \boldsymbol {z} _ {2}\right) \\ \end{array}
$$
Based on the facts above, we have:
$$
I \left(\boldsymbol {x}; \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) \geq I \left(\boldsymbol {x}; \boldsymbol {z} _ {1}; \boldsymbol {z} _ {2}\right) \tag {21}
$$