SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is often crucial to integrate domain knowledge that guides the model toward desirable cluster assignments. Thus, we propose an extension of TreeVAE where we integrate recent advances in contrastive learning (van den Oord et al., 2018; Chen et al., 2020; Li et al., 2021), whereby prior knowledge on data invariances can be encoded through augmentations. For a batch  $\mathbf{X}$  with  $N$  samples, we randomly augment every sample twice to obtain the augmented batch  $\tilde{\mathbf{X}}$  with  $2N$  samples. For all positive pairs  $(i,j)$  where  $\tilde{\boldsymbol{x}}_i$  and  $\tilde{\boldsymbol{x}}_j$  stem from the same original sample, we utilize the NT-Xent (Chen et al., 2020), which introduces losses  $\ell_{i,j} = -\log \frac{\exp(s_{i,j} / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]}\exp(s_{i,k} / \tau)}$ , where  $s_{i,j}$  denotes the cosine similarity between the representations of  $\tilde{\boldsymbol{x}}_i$  and  $\tilde{\boldsymbol{x}}_j$ , and  $\tau$  is a temperature parameter. We integrate  $\ell_{i,j}$  in both the bottom-up and the routers of TreeVAE. In the bottom-up, similar to Chen et al. (2020), we compute  $\ell_{i,j}$  on the projections  $g_h(\mathbf{d}_h)$ . For the routers, we directly compute the loss on the predicted probabilities  $r_{q,i}(\mathbf{d}_h)$ . Finally, we average the terms over all positive pairs and add them to the negative ELBO (12) in real-world image experiments. Implementation details can be found in Appendix E, while a loss ablation is shown in Appendix C.3.
Deep latent variable models automatically learn structure from data by combining the flexibility of deep neural networks and the statistical foundations of generative models (Mattei & Frellsen, 2018). Variational autoencoders (VAEs) (Rezende et al., 2014; Kingma & Welling, 2014) are among the most used frameworks (Nasiri & Bepler, 2022; Bae et al., 2023; Bredell et al., 2023). A variety of works has been proposed to integrate more complex empirical prior distributions, thus reducing the gap between approximate and true posterior distributions (Ranganath et al., 2015; Webb et al., 2017; Klushyn et al., 2019). Among these, the most related to our work is the VAE-nCRP (Goyal et al., 2017; Shin et al., 2019) and the TMC-VAE (Vikram et al., 2018). Both works use Bayesian nonparametric hierarchical clustering based on the nested Chinese restaurant process (nCRP) prior (Blei et al., 2003), and on the time-marginalized coalescent (TMC). However, even if they allow more flexible prior distributions these models suffer from restrictive posterior distributions (Kingma et al., 2016). To overcome the above issue, deep hierarchical VAEs (Gregor et al., 2015; Kingma et al., 2016) have been proposed to employ structured approximate posteriors, which are composed of hierarchies of conditional stochastic variables that are connected sequentially. Among a variety of proposed methods (Vahdat & Kautz, 2020b; Falck et al., 2022; T. Z. Xiao & Bamler, 2023), Ladder VAE (Sonderby et al., 2016) is most related to TreeVAE. The authors propose to model the approximate posterior by combining a "bottom-up" recognition distribution with the "top-down" prior. Further extensions include BIVA (Maaloe et al., 2019), which introduces a bidirectional inference network, and GraphVAE (He et al., 2019), that introduces gated dependencies over a fixed number of latent variables. Contrary to the previous approaches, TreeVAE models a tree-based posterior distribution of latent variable, thus allowing hierarchical clustering of samples. For further work on hierarchical clustering and its supervised counterpart, decision trees, we refer to Appendix B.
Datasets and Metrics: We evaluate the clustering and generative performance of TreeVAE on MNIST (LeCun et al., 1998), Fashion-MNIST (H. Xiao et al., 2017), 20Newsgroups (Lang, 1995), Omniglot (Lake et al., 2015), and Omniglot-5, where only 5 vocabularies (Braille, Glagolitic, Cyrillic, Odia, and Bengali) are selected and used as true labels. We assess the hierarchical clustering performance by computing dendrogram purity (DP) and leaf purity (LP), as defined by (Kobren et al., 2017a) using the datasets labels, where we assume the number of true clusters is unknown. We also report standard clustering metrics, accuracy (ACC) and normalized mutual information (NMI), by setting the number of leaves for TreeVAE and for the baselines to the true number of clusters. In terms of generative performance, we compute the approximated true log-likelihood calculated using 1000 importance-weighted samples, together with the ELBO (12) and the reconstruction loss (16). We also perform hierarchical clustering experiments on real-world imaging data, namely CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009) with 20 superclasses as labels, and CelebA (Z. Liu et al., 2015) using the contrastive extension (Sec. 2.6). We refer to Appendix D for more dataset details.
Baselines: We compare the generative performance of TreeVAE to the VAE (Rezende et al., 2014; Kingma & Welling, 2014), its non-hierarchical counterpart, and the LadderVAE (Sonderby et al., 2016), its sequential counterpart. For a fair comparison, all methods share the same architecture and
Table 1: Test set hierarchical clustering performances (%) of TreeVAE compared with baselines. Means and standard deviations are computed across 10 runs with different random model initialization. The star "***" indicates real-world image datasets on which contrastive approaches were applied.
Dataset	Method	DP	LP	ACC	NMI
MNIST	Agg	63.7 ± 0.0	78.6 ± 0.0	69.5 ± 0.0	71.1 ± 0.0
VAE + Agg	79.9 ± 2.2	90.8 ± 1.4	86.6 ± 4.9	81.6 ± 2.0
LadderVAE + Agg	81.6 ± 3.9	90.9 ± 2.5	80.3 ± 5.6	82.0 ± 2.1
DeepECT	74.6 ± 5.9	90.7 ± 3.2	74.9 ± 6.2	76.7 ± 4.2
TreeVAE (ours)	87.9 ± 4.9	96.0 ± 1.9	90.2 ± 7.5	90.0 ± 4.6
Fashion	Agg	45.0 ± 0.0	67.6 ± 0.0	51.3 ± 0.0	52.6 ± 0.0
VAE + Agg	44.3 ± 2.5	65.9 ± 2.3	54.9 ± 4.4	56.1 ± 3.2
LadderVAE + Agg	49.5 ± 2.3	67.6 ± 1.2	55.9 ± 3.0	60.7 ± 1.4
DeepECT	44.9 ± 3.3	67.8 ± 1.4	51.8 ± 5.7	57.7 ± 3.7
TreeVAE (ours)	54.4 ± 2.4	71.4 ± 2.0	63.6 ± 3.3	64.7 ± 1.4
20Newsgroups	Agg	13.1 ± 0.0	30.8 ± 0.0	26.1 ± 0.0	27.5 ± 0.0
VAE + Agg	7.1 ± 0.3	18.1 ± 0.5	15.2 ± 0.4	11.6 ± 0.3
LadderVAE + Agg	9.0 ± 0.2	20.0 ± 0.7	17.4 ± 0.9	17.8 ± 0.6
DeepECT	9.3 ± 1.8	17.2 ± 3.8	15.6 ± 3.0	18.1 ± 4.1
TreeVAE (ours)	17.5 ± 1.5	38.4 ± 1.6	32.8 ± 2.3	34.4 ± 1.5
Omniglot-5	Agg	41.4 ± 0.0	63.7 ± 0.0	53.2 ± 0.0	33.3 ± 0.0
VAE + Agg	46.3 ± 2.3	68.1 ± 1.6	52.9 ± 4.2	34.4 ± 2.9
LadderVAE + Agg	49.8 ± 3.9	71.3 ± 2.0	59.6 ± 4.9	44.2 ± 4.7
DeepECT	33.3 ± 2.5	55.1 ± 2.8	41.1 ± 4.2	23.5 ± 4.3
TreeVAE (ours)	58.8 ± 4.0	77.7 ± 3.9	63.9 ± 7.0	50.0 ± 5.9
CIFAR-10*	VAE + Agg	10.54 ± 0.12	16.33 ± 0.15	14.43 ± 0.19	1.86 ± 1.66
LadderVAE + Agg	12.81 ± 0.20	25.37 ± 0.62	19.29 ± 0.60	7.41 ± 0.42
DeepECT	10.01 ± 0.02	10.30 ± 0.40	10.31 ± 0.39	0.18 ± 0.10
TreeVAE (ours)	35.30 ± 1.15	53.85 ± 1.23	52.98 ± 1.34	41.44 ± 1.13
CIFAR-100*	VAE + Agg	5.27 ± 0.02	9.86 ± 0.19	8.82 ± 0.11	2.46 ± 0.10
LadderVAE + Agg	6.36 ± 0.07	16.08 ± 0.28	14.01 ± 0.41	8.99 ± 0.41
DeepECT	5.28 ± 0.18	6.97 ± 0.69	6.97 ± 0.69	1.71 ± 0.86
TreeVAE (ours)	10.44 ± 0.38	24.16 ± 0.65	21.82 ± 0.77	17.80 ± 0.42
hyperparameters whenever possible. We compare TreeVAE to non-generative hierarchical clustering baselines for which the code was publicly available: Ward's minimum variance agglomerative clustering (Agg) (Ward, 1963; Murtagh & Legendre, 2014), and the DeepECT (Mautz et al., 2020). We propose two additional baselines, where we perform Ward's agglomerative clustering on the latent space of the VAE  $(\mathrm{VAE} + \mathrm{Agg})$  and of the last layer of the LadderVAE  $(\mathrm{LadderVAE} + \mathrm{Agg})$ . For the contrastive clustering experiments, we apply a contrastive loss similar to TreeVAE to the VAE and the LadderVAE, while for DeepECT we use the contrastive loss proposed by the authors.
Implementation Details: While we believe that more complex architectures could have a substantial impact on the performance of TreeVAE, we choose to employ rather simple settings to validate the proposed approach. We set the dimension of all latent embeddings  $\mathbf{z} = \{\mathbf{z}_0,\dots ,\mathbf{z}_V\}$  to 8 for MNIST, Fashion, and Omniglot, to 4 for 20Newsgroups, and to 64 for CIFAR-10, CIFAR-100, and CelebA. The maximum depth of the tree is set to 6 for all datasets, except 20Newsgroups where we increased the depth to 7 to capture more clusters. To compute DP and LP, we allow the tree to grow to a maximum of 30 leaves for 20Newsgroups and CIFAR-100, and 20 for the rest, while for ACC and NMI we fix the number of leaves to the number of true classes. The transformations consist of one-layer MLPs of size 128 and the routers of two-layers of size 128 for all datasets except for the real-world imaging data where we slightly increase the MLP complexity to 512. Finally, the encoder and decoders consist of simple CNNs and MLPs. The trees are trained for  $N_{t} = 150$  epochs at each growth step, and the final tree is finetuned for  $N_{f} = 200$  epochs. For the real-world imaging experiments, we set the weight of the contrastive loss to 100. See Appendix E for additional details.
Table 2: Test set generative performances of TreeVAE with 10 leaves compared with baselines. Means and standard deviations are computed across 10 runs with different random model initialization.
Dataset	Method	LL	RL	ELBO
MNIST	VAE	-101.9 ± 0.2	87.2 ± 0.3	-104.6 ± 0.3
LadderVAE	-99.9 ± 0.5	87.8 ± 0.7	-103.2 ± 0.7
TreeVAE (ours)	-92.9 ± 0.2	80.3 ± 0.2	-96.8 ± 0.2
Fashion	VAE	-242.2 ± 0.2	231.7 ± 0.5	-245.4 ± 0.5
LadderVAE	-239.4 ± 0.5	231.5 ± 0.6	-243.0 ± 0.6
TreeVAE (Ours)	-234.7 ± 0.1	226.5 ± 0.3	-239.2 ± 0.4
20Newsgroups	VAE	-44.26 ± 0.01	45.52 ± 0.03	-44.61 ± 0.01
LadderVAE	-44.30 ± 0.03	43.52 ± 0.03	-44.62 ± 0.02
TreeVAE (Ours)	-51.67 ± 0.59	45.83 ± 0.36	-52.79 ± 0.66
Omniglot	VAE	-115.3 ± 0.3	101.6 ± 0.3	-118.2 ± 0.3
LadderVAE	-113.1 ± 0.5	100.7 ± 0.7	-117.5 ± 0.6
TreeVAE (Ours)	-110.4 ± 0.5	96.9 ± 0.5	-114.6 ± 0.4
Hierarchical Clustering Results Table 1 shows the quantitative hierarchical clustering results averaged across 10 seeds. First, we assume the true number of clusters is unknown and report DP and LP. Second, we assume we have access to the true number of clusters  $K$  and compute ACC and NMI. As can be seen, TreeVAE outperforms the baselines in both experiments. This suggests that the proposed approach successfully builds an optimal tree based on the data's intrinsic characteristics. Among the different baselines, agglomerative clustering using Ward's method (Agg) trained on the last layer of LadderVAE shows competitive performances. To the best of our knowledge, we are the first to report these results. It is noteworthy to observe that it consistently improves over VAE + Agg, indicating that the last layer of LadderVAE captures more cluster information than the VAE.
**Generative Results** In Table 2, we evaluate the generative performance of the proposed approach, TreeVAE, compared to the VAE, its non-hierarchical counterpart, and LadderVAE, its sequential counterpart. TreeVAE outperforms the baselines on the majority of datasets, indicating that the proposed ELBO (12) can achieve a tighter lower bound of the log-likelihood. The most notable improvement appears to be reflected in the reconstruction loss, showing the advantage of using cluster-specialized decoders. However, this improvement comes at the expense of a larger neural network architecture and an increase in the number of parameters (as TreeVAE has  $L$  decoders). While this requires more computational resources at training time, during deployment the tree structure of TreeVAE permits lightweight inference through conditional sampling, thus matching the inference time of LadderVAE. It is also worth mentioning that results differ from (Sønderby et al., 2016) as we adapt their architecture to match our experimental setting and consequently use smaller latent dimensionality. Finally, we notice that more complex methods are prone to overfitting on the 20Newsgroups dataset, hence the best performances are achieved by the VAE.
Real-world Imaging Data & Contrastive Learning Clustering real-world imaging data is extremely difficult as there are endless possibilities of how the data can be partitioned (such as the colors, the landscape, etc). We therefore inject prior information through augmentations to guide TreeVAE and the baselines to semantically meaningful splits. Table 1 (bottom) shows the hierarchical clustering performance of TreeVAE and its baselines, all employing contrastive learning, on CIFAR-10 and CIFAR-100. We observe that DeepECT struggles in separating the data as their contrastive approach leads to all samples falling into the same leaf. In Table 3, we present the leaf-frequencies of various face attributes using the tree learned by TreeVAE. For all datasets, TreeVAE is able to group the data into contextually meaningful hierarchies and groups, evident from its superior performance compared to the baselines and from the distinct attribute frequencies in the leaves and subtrees.
Discovery of Hierarchies In addition to solely clustering data, TreeVAE is able to discover meaningful hierarchical relations between the clusters, thus allowing for more insights into the
Figure 4: Hierarchical structures learned by TreeVAE on Fashion. Subtree (a) encodes tops, while (b) encodes shoes, purses, and pants.
Figure 5: Hierarchical structure learned by TreeVAE on 20News-groups.
Figure 6: Hierarchical structures learned by TreeVAE on Omniglot-5. Subtree (a) learns a hierarchy over Braille and the Indian alphabets, while (b) groups Slavic alphabets.
Attribute	1	2	3	4	5	6	7	8
Female	97.2	55.0	97.7	86.6	23.1	30.7	46.6	43.7
Bangs	1.6	1.2	24.1	61.7	3.4	11.1	9.1	11.4
Blonde	1.1	3.7	66.7	2.2	5.8	2.6	26.1	7.1
Makeup	75.7	43.4	76.6	59.7	15.0	12.4	16.3	12.8
Smiling	54.3	66.6	66.4	51.2	54.7	42.4	37.3	22.4
Hair Loss	3.6	17.8	3.0	0.2	18.9	6.9	21.2	10.6
Beard	1.1	20.6	0.4	3.7	39.5	36.5	21.3	21.4
Table 3: We present the frequency (in %) of selected attributes for each leaf of TreeVAE with eight leaves in CelebA.
Figure 7: Hierarchical structure learned by TreeVAE with eight leaves on the CelebA dataset with generated images through conditional sampling. Generally, most females are in the left subtree, while most males are in the right subtree. We observe that leaf 1 is associated with dark-haired females, leaf 2 with smiling, dark-haired individuals, leaf 3 with blonde females, leaf 4 with bangs, leaf 7 with a receding hairline, and leaf 8 with non-smiling people. See Table 3 for further details.
dataset. In the introductory Fig. 1, 5, and 6, we present the hierarchical structures learned by TreeVAE, while in Fig. 4 and 7, we additionally display conditional cluster generations from the leaf-specific decoders. In Fig. 4, TreeVAE separates the fashion items into two subtrees, one containing shoes and bags, and the other containing the tops, which are further refined into long and short sleeves. In Fig. 5, we depict the most prevalent ground-truth topic label in each leaf. TreeVAE learns to
separate technological and societal subjects and discovers semantically meaningful subtrees. In Fig. 6, TreeVAE learns to split alphabets into Indian (Odia and Bengali) and Slavic (Glagolitic and Cyrillic) subtrees, while Braille is grouped with the Indian languages due to similar circle-like structures. For CelebA, Fig. 7 and Table 3, the resulting tree separates genders in the root split. Females (left) are further divided by hair color and hairstyle (bangs). Males (right) are further divided by smile intensity, beard, hair loss, and age. In Fig. 8 and Appendix C we show how TreeVAE can additionally be used to sample unconditional generations for all clusters simultaneously, by sampling from the root and propagating through the entire tree. The generations differ across the leaves by their cluster-specific features, whereas cluster-independent properties are retained across all generations.
Figure 8: Selected unconditional generations of CelebA. One row corresponds to one sample from the root, for which we depict the visualizations obtained from the 8 leaf-decoders. The overall face shape, skin color, and face orientation are retained among leaves from the same row, while several properties (such as make-up, beard, mustache, glasses, and hair) vary across the different leaves.
In this paper, we introduced TreeVAE, a new generative method that leverages a tree-based posterior distribution of latent variables to capture the hierarchical structures present in the data. TreeVAE optimizes the balance between shared and specialized architecture, enhancing the learning and adaptation capabilities of generative models. Empirically, we showed that our model offers a substantial improvement in hierarchical clustering performance compared to the related work, while also providing a tighter lower bound to the log-likelihood of the data. We presented qualitatively how the hierarchical structures learned by TreeVAE enable a more comprehensive understanding of the data, thereby facilitating enhanced analysis, interpretation, and decision-making. Our findings highlight the versatility of the proposed approach, which we believe to hold significant potential for unsupervised representation learning, paving the way for exciting advancements in the field.
Limitations & Future Work: Currently, TreeVAE uses a simple heuristic on which node to split that might not work on datasets with unbalanced clusters. Additionally, the contrastive losses on the routers encourage balanced clusters. Thus, more research is necessary to convert the heuristics to data-driven approaches. While deep latent variable models, such as VAEs, provide a framework for modeling explicit relationships through graphical structures, they often exhibit poor performance on synthetic image generation. However, more complex architectural design (Vahdat & Kautz, 2020a) or recent advancement in diffusion latent models (Rombach et al., 2021) present potential solutions to enhance image quality generation, thus striking an optimal balance between generating high-quality images and capturing meaningful representations.
We thank Thomas M. Sutter for the insightful discussions throughout the project, Jorge da Silva Gonçalves for providing interpretable visualizations of the TreeVAE model, and Gabriele Manduchi for the valuable feedback on the notation of the ELBO. LM is supported by the SDSC PhD Fellowship #1-001568-037. MV is supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00047. AR is supported by the StimuLoop grant #1-007811-002 and the Vontobel Foundation.
Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Petryk, S., ... Gonzalez, J. E. (2021). NBDT: neural-backed decision tree. In 9th international conference on learning representations, ICLR 2021, virtual event, austria, may 3-7, 2021. OpenReview.net. Retrieved from https://openreview.net/forum?id=mCLVeEpp1NE
Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236-244.
Webb, S., Golinski, A., Zinkov, R., Narayanaswamy, S., Rainforth, T., Teh, Y. W., & Wood, F. (2017). Faithful inversion of generative models for effective amortized inference. In Neural information processing systems.
Williams, C. (1999). A mcmc approach to hierarchical mixture modelling. Advances in Neural Information Processing Systems, 12.
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018). Unsupervised feature learning via non-parametric instance discrimination. In 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, salt lake city, ut, usa, june 18-22, 2018 (pp. 3733-3742). Computer Vision Foundation / IEEE Computer Society. Retrieved from http://openaccess.thecvf.com/content_cvpr_2018/html/Wu_Unsupervised_Feature_Learning_CVPR_2018_paper.html doi: 10.1109/CVPR.2018.00393
Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747. Retrieved from http://arxiv.org/abs/1708.07747
Xiao, T. Z., & Bamler, R. (2023). Trading information between latents in hierarchical variational autoencoders. In The eleventh international conference on learning representations. Retrieved from https://openreview.net/forum?id=eWtMdr6yCmL
You, C., Robinson, D. P., & Vidal, R. (2015). Scalable sparse subspace clustering by orthogonal matching pursuit. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3918-3927.
Zharmagambetov, A., & Carreira-Perpiñán, M. Á. (2022). Semi-supervised learning with decision trees: Graph laplacian tree alternating optimization. In Neurips. Retrieved from http://papers.nips.cc/paper_files/paper/2022/bitnet/104f7b25495a0e40e65fb7c7eee37ed9-Abscract-Conference.html
Zhou, Z.-H., & Feng, J. (2017). Deep forest: Towards an alternative to deep neural networks. In Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI-17 (pp. 3553-3559). Retrieved from https://doi.org/10.24963/ijcai.2017/497 doi: 10.24963/ijcai.2017/497
A Evidence Lower Bound
In this section, we provide a closer look at the loss function of TreeVAE. We focus on the derivations of the Kullback-Leibler divergence term of the Evidence Lower Bound and provide an interpretable factorization. Furthermore, we address the computational complexity, thus offering an in-depth understanding of the loss function, its practical implications, and the trade-offs involved in its computation.
A.1 ELBO Derivations
In this section, we derive the KL loss (17) of the ELBO (12), which is the Kullback-Leibler divergence (KL) between the prior and the variational posterior of TreeVAE. Additionally, we give details about the underlying distributional assumptions for computing the reconstruction loss.
Let us define  $\mathcal{P}_l$  the decision path from root 0 to leaf  $l$ ,  $L$  is the number of leaves, which is equal to the number of paths in  $\mathcal{T}$ ,  $\mathbf{z}_{\mathcal{P}_l} = \{\mathbf{z}_i \mid i \in \mathcal{P}_l\}$  the set of latent variables selected by the path  $\mathcal{P}_l$ , the parent node of the node  $i$  as  $pa(i)$ ,  $p(c_{pa(i) \rightarrow i} \mid z_{pa(i)})$  the probability of going from  $pa(i)$  to  $i$ . For example, if we consider the path in Fig. 2 (right) we will observe  $c_0 = 0$ ,  $c_1 = 1$ , and  $c_4 = 0$ , where  $c_i = 0$  means the model selects the left child of node  $i$ . The KL loss can be expanded using Eq. 1/3:
In the following, we will simplify each of the three terms 24, 25, and 26 separately.
A.1.1 KL Root
The term (24) corresponds to the KL of the root node. We can integrate out all the latent variables  $z_{i}$  for  $i \neq 0$  and all decisions  $c_{i}$ . The first term can be then written as follows:
$$
\begin{array}{l} \mathrm {K L} _ {\text {r o o t}} = \sum_ {i \in \{1, 2 \}} \int_ {\boldsymbol {z} _ {0}} q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right) q \left(c _ {0 \rightarrow i} \mid \boldsymbol {z} _ {0}\right) \log \left(\frac {q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right)}{p \left(\boldsymbol {z} _ {0}\right)}\right) (27) \\ = \int_ {\boldsymbol {z} _ {0}} q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right)\left[ \sum_ {i \in \{1, 2 \}} q \left(c _ {0 \rightarrow i} \mid \boldsymbol {z} _ {0}\right)\right] \log \left(\frac {q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right)}{p \left(\boldsymbol {z} _ {0}\right)}\right) (28) \\ = \int_ {\mathbf {z} _ {0}} q \left(\mathbf {z} _ {0} \mid \mathbf {x}\right) \left[ q \left(\mathrm {c} _ {0} = 0 \mid \mathbf {z} _ {0}\right) + q \left(\mathrm {c} _ {0} = 1 \mid \mathbf {z} _ {0}\right) \right] \log \left(\frac {q \left(\mathbf {z} _ {0} \mid \mathbf {x}\right)}{p \left(\mathbf {z} _ {0}\right)}\right) (29) \\ = \int_ {\boldsymbol {z} _ {0}} q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right) \log \left(\frac {q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right)}{p \left(\boldsymbol {z} _ {0}\right)}\right) = \mathrm {K L} \left(q \left(\boldsymbol {z} _ {0} \mid \boldsymbol {x}\right) \| p \left(\boldsymbol {z} _ {0}\right)\right), (30) \\ \end{array}
$$