SlowGuess/ACL_2020_OCR huggingface dataset:
Qile Zhu $^{1}$ , Wei Bi $^{2}$ , Xiaojiang Liu $^{2}$ , Xiyao Ma $^{1}$ , Xiaolin Li $^{3}$  and Dapeng Wu $^{1}$
<sup>1</sup>University of Florida, <sup>2</sup>Tencent AI Lab, <sup>3</sup>AI Institute, Tongdun Technology  
{valder,maxiy,dpwu}@ufl.edu  
{victoriabi,kieranliu}@tencent.com  
xiaolin.li@tongdun.net
Variational Autoencoder (VAE) is widely used as a generative model to approximate a model's posterior on latent variables by combining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE often converges to a degenerated local optimum known as "posterior collapse". Previous approaches consider the Kullback-Leibler divergence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL's distribution positive. Then we propose Batch Normalized-VAE (BN-VAE), a simple but effective approach to set a lower bound of the expectation by regularizing the distribution of the approximate posterior's parameters. Without introducing any new model component or modifying the objective, our approach can avoid the posterior collapse effectively and efficiently. We further show that the proposed BN-VAE can be extended to conditional VAE (CVAE). Empirically, our approach surpasses strong autoregressive baselines on language modeling, text classification and dialogue generation, and rivals more complex approaches while keeping almost the same training time as VAE.
Variational Autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) is one of the most popular generative framework to model complex distributions. Different from the Autoencoder (AE), VAE provides a distribution-based latent representation for the data, which encodes the input  $\mathbf{x}$  into a probability distribution  $\mathbf{z}$  and reconstructs the original input using samples from  $\mathbf{z}$ . When
inference, VAE first samples the latent variable from the prior distribution and then feeds it into the decoder to generate an instance. VAE has been successfully applied in many NLP tasks, including topic modeling (Srivastava and Sutton, 2017; Miao et al., 2016; Zhu et al., 2018), language modeling (Bowman et al., 2016), text generation (Zhao et al., 2017b) and text classification (Xu et al., 2017).
An autoregressive decoder (e.g., a recurrent neural network) is a common choice to model the text data. However, when paired with strong autoregressive decoders such as LSTMs (Hochreiter and Schmidhuber, 1997) and trained under conventional training strategy, VAE suffers from a well-known problem named the posterior collapse or the KL vanishing problem. The decoder in VAE learns to reconstruct the data independent of the latent variable  $\mathbf{z}$ , and the KL vanishes to 0.
Many convincing solutions have been proposed to prevent posterior collapse. Among them, fixing the KL as a positive constant is an important direction (Davidson et al., 2018; Guu et al., 2018; van den Oord et al., 2017; Xu and Durrett, 2018; Tomczak and Welling, 2018; Kingma et al., 2016; Razavi et al., 2019). Some change the Gaussian prior with other distributions, e.g., a uniform prior (van den Oord et al., 2017; Zhao et al., 2018) or a von Mises-Fisher (vMf) distribution (Davidson et al., 2018; Guu et al., 2018; Xu and Durrett, 2018). However, these approaches force the same constant KL and lose the flexibility to allow various KLs for different data points (Razavi et al., 2019). Without changing the Gaussian prior, free-bits (Kingma et al., 2016) adds a threshold (free-bits) of the KL term in the ELBO object and stops the optimization of the KL part when its value is smaller than the threshold. Chen et al. (2017) point out that the objective of free-bits is non-smooth and suffers from the optimization challenges.  $\delta$ -VAE (Razavi et al., 2019) sets the parameters in a specific range
*This work was done when Qile Zhu was an intern at Tencent AI Lab. Wei Bi is the corresponding author.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2636-2649 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
to achieve a positive KL value for every latent dimension, which may limit the model performance.
Other work analyzes this problem from a view of optimization (Bowman et al., 2016; Zhao et al., 2017a; Chen et al., 2017; Alemi et al., 2018). Recently, He et al. (2019) observe that the inference network is lagging far behind the decoder during training. They propose to add additional training loops for the inference network only. Li et al. (2019) further propose to initialize the inference network with an encoder pretrained from an AE objective, then trains the VAE with the free-bits. However, these two methods are much slower than the original VAE.
The limitation of the constant KL and the high cost of additional training motivate us to seek an approach that allows flexible modeling for different data points while keeping as fast as the original VAE. In this paper, instead of considering the KL individually for each data point, we let it follow a distribution across the whole dataset. We demonstrate that keeping a positive expectation of the KL's distribution is sufficient to prevent posterior collapse in practice. By regularizing the distribution of the approximate posterior's parameters, a positive lower bound of this expectation could be ensured. Then we propose Batch Normalized-VAE (BN-VAE), a simple yet effective approach to achieving this goal, and discuss the connections between BN-VAE and previous enhanced VAE variants. We further extend BN-VAE to the conditional VAE (CVAE). Last, experimental results demonstrate the effectiveness of our approach on real applications, including language modeling, text classification and dialogue generation. Empirically, our approach surpasses strong autoregressive baselines and is competitive with more sophisticated approaches while keeping extremely higher efficiency. Code and data are available at https://github.com/valdersoul/bn-vae.
In this section, we first introduce the basic background of VAE, then we discuss the lagging problem (He et al., 2019). At last, we present more related work.
2.1 VAE Background
VAE (Kingma and Welling, 2014; Rezende et al., 2014) aims to learn a generative model  $p(\mathbf{x}, \mathbf{z})$  to maximize the marginal likelihood  $\log p(\mathbf{x})$  on a
dataset. The marginal likelihood cannot be calculated directly due to an intractable integral over the latent variable  $\mathbf{z}$ . To solve this, VAE introduces a variational distribution  $q_{\phi}(\mathbf{z}|\mathbf{x})$  which is parameterized by a complex neural network to approximate the true posterior. Then it turns out to optimize the ELBO of  $\log p(\mathbf{x})$ :
$$
\mathcal {L} = \mathrm {E} _ {q _ {\phi} (\mathbf {z} | \mathbf {x})} [ \log p _ {\theta} (\mathbf {x} | \mathbf {z}) ] - K L (q _ {\phi} (\mathbf {z} | \mathbf {x}) | | p (\mathbf {z})) \tag {1}
$$
where  $\phi$  represents the inference network and  $\theta$  denotes the decoder. The above first term is the reconstruction loss, while the second one is the KL between the approximate posterior and the prior. The Gaussian distribution  $\mathcal{N} \sim (0, I)$  is a usual choice for the prior, and the KL between the approximate posterior  $q_{\phi}(\mathbf{z}|\mathbf{x})$  and the prior  $p(\mathbf{z})$  can be computed as:
$$
K L = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(\mu_ {i} ^ {2} + \sigma_ {i} ^ {2} - \log \sigma_ {i} ^ {2} - 1\right), \tag {2}
$$
where  $\mu_{i}$  and  $\sigma_{i}$  is the mean and standard deviation of approximate posterior for the  $i_{th}$  latent dimension, respectively. When the decoder is autoregressive, it can recover the data independent of the latent  $\mathbf{z}$  (Bowman et al., 2016). The optimization will encourage the approximate posterior to approach the prior which results in the zero value of the KL.
2.2 The Lagging Problem
Recently, He et al. (2019) analyze posterior collapse with the Gaussian prior from a view of training dynamics. The collapse is a local optimum of VAE when  $q_{\phi}(\mathbf{z}|\mathbf{x}) = p_{\theta}(\mathbf{z}|\mathbf{x}) = p(\mathbf{z})$  for all inputs. They further define two partial collapse states: model collapse, when  $p_{\theta}(\mathbf{z}|\mathbf{x}) = p(\mathbf{z})$ , and inference collapse, when  $q_{\phi}(\mathbf{z}|\mathbf{x}) = p(\mathbf{z})$ . They observe that the inference collapse always happens far before the model collapse due to the existence of autoregressive decoders. Different from the model posterior, the inference network lacks guidance and easily collapses to the prior at the initial stage of training, and thus posterior collapse happens. Based on this understanding, they propose to aggressively optimize the inference network. However, this approach cost too much time compared with the original VAE. In our work, we also employ the Gaussian prior and thus suffer from the same lagging problem. Yet, our proposed approach does
not involve additional training efforts, which can effectively avoid the lagging problem (Section 3.3) and keep almost the same training efficiency as the original VAE (Section 5.1). More details can be found in Section 3.3.
To prevent posterior collapse, we have mentioned many work about changing the prior in the introduction. Besides these approaches, some work modifies the original training objective directly. For example, Bowman et al. (2016) introduce an annealing strategy, where they slightly increase the weight of KL from 0 to 1 during the warm-up period.  $\beta$ -VAE (Higgins et al., 2017) treats the KL weight as a hyperparameter to constrain the minimum value of the KL. Alemi et al. (2017), on the other hand, set a fixed KL weight to control the mutual information between  $\mathbf{z}$  and  $\mathbf{x}$ . Tolstikhin et al. (2018) leverage the Wasserstein distance to replace the KL. Zhao et al. (2017a) replace the KL with maximum mean discrepancy. Fang et al. (2019) introduce sample-based representations which lead to implicit latent features with an auxiliary network.
Some change the training strategy. Kim et al. (2018) address the amortization gap (Cremer et al., 2018) in VAE and propose Semi-Amortized VAE to compose the inference network with additional mean-field updates. Fu et al. (2019) propose a cyclical annealing schedule, which repeats the process of increasing  $\beta$  multiple times.
There are various other approaches to solve the posterior collapse. For example, some researchers choose to weaken the decoder by replacing the LSTM decoder with convolution neural networks without autoregressive modeling (Semeniuta et al., 2017; Yang et al., 2017). Chen et al. (2017) input a lossy representation of data to the autoregressive decoder and enforce  $\mathbf{z}$  to capture the information about the original input. Inheriting this idea, some following work add direct connections between  $\mathbf{z}$  and  $\mathbf{x}$  (Zhao et al., 2017b; Dieng et al., 2019). Ma et al. (2019) introduce an additional regularization to learn diverse latent representation.  $\delta$ -VAE (Razavi et al., 2019) and free-bits (Kingma et al., 2016) set a minimum number of KL for each latent dimension to prevent the posterior collapse.
Srivastava and Sutton (2017, 2018) find that using ADAM (Kingma and Ba, 2014) with a high learning rate to train VAE may cause the gradients to diverge early. Their explanation for the diverg
ing behavior lies in the exponential curvature of the gradient from the inference network which produces the variance part of the approximate posterior. Then they apply batch normalization to the variance part to solve this problem. We use the simple SGD without momentum to train our model. Moreover, we apply batch normalization to the mean part of the inference network to keep the expectation of the KL's distribution positive, which is different from their work. We also find that Sønderby et al. (2016) utilize batch normalization in all fully connected layers with nonlinear activation functions to improve the model performance. Different from it, our approach directly applies batch normalization to the parameters of the approximate posterior, which is the output of the inference network.
3 Batch-Normalized VAE
In this section, we first derive the expectation of the KL's distribution and show that it is enough to avoid posterior collapse by keeping the expectation of the KL's distribution positive. Then we propose our regularization method on the parameters of the approximate posterior to ensure a positive lower bound of this expectation. We further discuss the difference between our approach and previous work.
3.1 Expectation of the KL's Distribution
Given an  $\pmb{x} \in \mathcal{X}$ , the inference network parametrizes a  $n$ -dimensional Gaussian distribution with its mean  $\mu = f_{\mu}(\pmb{x})$  and diagonal covariance  $\pmb{\Sigma} = \text{diag}(f_{\Sigma}(\pmb{x}))$ , where  $f_{\mu}$  and  $f_{\Sigma}$  are two neural networks. In practice, the ELBO is computed through a Monte Carlo estimation from  $b$  samples. The KL in Eq. 2 is then computed over  $b$  samples from  $\mathcal{X}$ :
$$
\begin{array}{l} K L = \frac {1}{2 b} \sum_ {j = 1} ^ {b} \sum_ {i = 1} ^ {n} \left(\mu_ {i j} ^ {2} + \sigma_ {i j} ^ {2} - \log \sigma_ {i j} ^ {2} - 1\right) \\ = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(\frac {\sum_ {j = 1} ^ {b} \mu_ {i j} ^ {2}}{b} + \frac {\sum_ {j = 1} ^ {b} \sigma_ {i j} ^ {2}}{b} \right. \\ - \frac {\sum_ {j = 1} ^ {b} \log \sigma_ {i j} ^ {2}}{b} - 1). \tag {3} \\ \end{array}
$$
When  $b$  gets larger, the above empirical value will approach the mean of the KL across the whole dataset.
To make use of this observation, we assume that  $\mu_{i}$  and  $\log \sigma_i^2$  for each latent dimension  $i$  follow
a certain distribution with a fixed mean and variance across the dataset respectively. The distribution may vary between different latent dimensions. In this way, the KL turns to a distribution of  $\mu_{i}$ 's and  $\log \sigma_{i}^{2}$ 's. From Eq. 3, we can see that  $\sum_{j=1}^{b} \mu_{ij}^{2} / b$  is the sample mean of  $\mu_{i}^{2}$ , which converges to  $\mathrm{E}[\mu_{i}^{2}] = \mathrm{Var}[\mu_{i}] + \mathrm{E}^{2}[\mu_{i}]$ . Similarly,  $\sum_{j=1}^{b} \sigma_{ij}^{2} / b$  converges to  $\mathrm{E}[\sigma_{i}^{2}]$ , and  $\sum_{j=1}^{b} \log \sigma_{ij}^{2} / b$  to  $\mathrm{E}[\log \sigma_{i}^{2}]$ . Thus, we can derive the expectation of the KL's distribution as:
$$
\begin{array}{l} \operatorname {E} [ K L ] = \frac {1}{2} \sum_ {i = 1} ^ {n} (\operatorname {V a r} [ \mu_ {i} ] + \operatorname {E} ^ {2} [ \mu_ {i} ] \\ + \mathrm {E} \left[ \sigma_ {i} ^ {2} \right] - \mathrm {E} \left[ \log \sigma_ {i} ^ {2} \right] - 1) \\ \geq \frac {1}{2} \sum_ {i = 1} ^ {n} \left(\operatorname {V a r} \left[ \mu_ {i} \right] + \mathrm {E} ^ {2} \left[ \mu_ {i} \right]\right), \tag {4} \\ \end{array}
$$
where  $\operatorname{E}[\sigma_i^2 - \log \sigma_i^2] \geq 1$  since the minimum of  $e^x - x$  is 1. If we can guarantee a positive lower bound of  $\operatorname{E}[KL]$ , we can then effectively prevent the posterior collapse.
Based on Eq. 4, the lower bound is only dependent on the number of latent dimensions  $n$  and  $\mu_{i}$ 's mean and variance. This motivates our idea that with proper regularization on the distributions of  $\mu_{i}$ 's to ensure a positive lower bound of  $\operatorname{E}[KL]$ .
3.2 Normalizing Parameters of the Posterior
The remaining key problem is to construct proper distributions of  $\mu_{i}$ 's that can result in a positive lower bound of  $\operatorname{E}[KL]$  in Eq. 4. Here, we propose a simple and efficient approach to accomplish this by applying a fixed batch normalization on the output of the inference network  $(\mu_{i})$ . Batch Normalization (BN) (Ioffe and Szegedy, 2015) is a widely used regularization technique in deep learning. It normalizes the output of neurons and makes the optimization landscape significantly smoother (Santurkar et al., 2018). Different from other tasks that apply BN in the hidden layers and seek fast and stable training, here we leverage BN as a tool to transform  $\mu_{i}$  into a distribution with a fixed mean and variance. Mathematically, the regularized  $\mu_{i}$  is written by:
$$
\hat {\mu} _ {i} = \gamma \frac {\mu_ {i} - \mu_ {\mathcal {B} i}}{\sigma_ {\mathcal {B} i}} + \beta , \tag {5}
$$
where  $\mu_{i}$  and  $\hat{\mu_i}$  are means of the approximate posterior before and after BN.  $\mu_{\mathcal{B}i}$  and  $\sigma_{\mathcal{B}i}$  denote the mean and standard deviations of  $\mu_{i}$ . They are biased estimated within a batch of samples for each
dimension indecently.  $\gamma$  and  $\beta$  are the scale and shift parameter. Instead of using a learnable  $\gamma$  in Eq. 5, we use a fixed BN which freezes the scale  $\gamma$ . In this way, the distribution of  $\mu_{i}$  has the mean of  $\beta$  and the variance of  $\gamma^2$ .  $\beta$  is a learnable parameter that makes the distribution more flexible.
Now, we derive the lower bound of  $\operatorname{E}[KL]$  by using the fixed BN. With the fixed mean  $\beta$  and variance  $\gamma^2$  for  $\mu_i$  in hand, we get a new lower bound as below:
$$
\begin{array}{l} \operatorname {E} [ K L ] \geq \frac {1}{2} \sum_ {i} ^ {n} (\operatorname {V a r} [ \mu_ {i} ] + \operatorname {E} ^ {2} [ \mu_ {i} ]) \\ = \frac {n \cdot \left(\gamma^ {2} + \beta^ {2}\right)}{2}. \tag {6} \\ \end{array}
$$
To this end, we can easily control the lower bound of  $\operatorname{E}[KL]$  by setting  $\gamma$ . Algorithm 1 shows the training process.
Algorithm 1 BN-VAE training.
1: Initialize  $\phi$  and  $\theta$ .
2: for  $i = 1,2,\dots$  Until Convergence do
3: Sample a mini-batch  $\mathbf{x}$
4:  $\mu, \log \sigma^2 = f_{\phi}(\mathbf{x})$
5:  $\mu^{\prime} = BN_{\gamma ,\beta}(\mu)$
6: Sample  $\mathbf{z} \sim \mathcal{N}(\mu', \sigma^2)$  and reconstruct  $\mathbf{x}$  from  $f_{\theta}(\mathbf{z})$ .
7: Compute gradients  $\mathbf{g}_{\phi ,\theta}\gets \nabla_{\phi ,\theta}\mathcal{L}(\mathbf{x};\phi ,\theta)$
8: Update  $\phi, \theta$  using  $\mathbf{g}_{\phi, \theta}$ .
9: end for
3.3 Connections with Previous Approaches
Constructing a positive KL: Both free-bits (Kingma et al., 2016) and  $\delta$ -VAE (Razavi et al., 2019) set a threshold on the KL value. Free-bits changes the KL term in the ELBO to a hinge loss term:  $\sum_{i}^{n}\max (\lambda ,KL(q_{\phi}(z_{i}|x)||p(z_{i}))$ . Another version of free-bits is to apply the threshold to the entire sum directly instead of the individual value. Training with the free-bits objective, the model will stop to drive down the KL value when it is already below  $\lambda$ . However, Chen et al. (2017) point out that the objective of free-bits is non-smooth and suffers from the optimization challenges. Our approach does not face the optimization problem since we use the original ELBO objective.
$\delta$ -VAE sets a target rate of  $\delta$  for each latent dimension by constraining the mean and variance of
2639
the approximate posterior:
$$
\begin{array}{l} \sigma_ {q} = \sigma_ {q} ^ {l} + \left(\sigma_ {q} ^ {u} - \sigma_ {q} ^ {l}\right) \frac {1}{1 + e ^ {- q _ {\phi} (x)}}, (7) \\ \mu = 2 \delta + 1 + \ln \left(\sigma_ {q} ^ {2}\right) - \sigma_ {q} ^ {2} + \max  (0, \mu_ {\phi} (\mathbf {x})), (8) \\ \end{array}
$$
where  $[\sigma^l, \sigma^u]$  are the feasible interval for  $\sigma_q$  by solving  $\ln (\sigma_q^2) - \sigma_q^2 + 2\delta + 1 \geq 0$ . Although  $\delta$ -VAE can ensure a minimum value for the KL, it limits the model performance due to that the parameters are constrained in the interval. Our approach only constrains the distributions of  $\mu$ , which is more flexible than  $\delta$ -VAE. Experiments further show that our approach surpass both free-bits and  $\delta$ -VAE.
Reducing inference lag: As we focus on the setting of the conventional Gaussian prior, the lagging problem mentioned in Section 2.2 is crucial. To this point, it is beneficial to analyze an alternate form of the ELBO:
$$
\mathcal {L} = \log p _ {\theta} (\mathbf {x}) - K L \left(q _ {\phi} (\mathbf {z} | \mathbf {x}) \mid \mid p _ {\theta} (\mathbf {z} | \mathbf {x})\right). \tag {9}
$$
With this view, the only goal of the approximate posterior  $q_{\phi}(\mathbf{z}|\mathbf{x})$  is to match the model posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$ . We examine the performance of our approach to reduce inference lag using the same synthetic experiment in He et al. (2019). Details can be found in Section 1 of the Appendix. The synthetic experiment indicates that our approach with the regularization is beneficial to rebalance the optimization between inference and generation, and finally overcomes posterior collapse. We also prefer a large  $\gamma$  due to that a small  $\gamma$  will push the approximate posterior to the prior. More details on the synthetic experiment can be found in the Appendix.
4 Extension to CVAE
Given an observation  $\mathbf{x}$  and its output  $\mathbf{y}$ , CVAE (Sohn et al., 2015; Zhao et al., 2017b) models the conditional distribution  $p(\mathbf{y}|\mathbf{x})$ . The variational lower bound of the conditional log-likelihood is:
$$
\begin{array}{l} \mathcal {L} = \operatorname {E} _ {q _ {\phi} (\mathbf {z} | \mathbf {x}, \mathbf {y})} [ \log p _ {\kappa} (\mathbf {y} | \mathbf {x}, \mathbf {z}) ] \\ - K L \left(q _ {\phi} (\mathbf {z} | \mathbf {x}, \mathbf {y}) \mid \mid p _ {\theta} (\mathbf {z} | \mathbf {x})\right) \\ \leq \log p (\mathbf {y} | \mathbf {x}). \tag {10} \\ \end{array}
$$
Different from VAE, the prior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  in CVAE is not fixed, which is also parametrized by a neural network. It is possible to apply another BN on the
mean of the prior with a different  $\gamma$  so that the expectation of the KL becomes a constant. However, this lower bound is uncontrollable due to the density of  $\mu_{1} + \mu_{2}$  is the convolution of their densities, which is intractable.
To overcome this issue, we propose to constrain the prior with a fixed distribution. We achieve it by adding another KL between the prior and a known Gaussian distribution  $r(\mathbf{z})$ , i.e.  $KL(p_{\theta}(\mathbf{z}|\mathbf{x})||r(\mathbf{z}))$ . Instead of optimizing the ELBO in Eq. 10, we optimize a lower bound of the ELBO for CVAE:
$$
\mathcal {L} ^ {\prime} = \mathcal {L} - K L (p _ {\theta} (\mathbf {z} | \mathbf {x}) | | r (\mathbf {z})) \leq \mathcal {L}. \tag {11}
$$
The KL term in the new bound is the sum of  $KL(q_{\phi}(\mathbf{z}|\mathbf{x},\mathbf{y})||p_{\theta}(\mathbf{z}|\mathbf{x}))$  and  $KL(p_{\theta}(z|x)||r(z))$  which can be computed as:
$$
\begin{array}{l} K L = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(\frac {\sigma_ {q i} ^ {2} + (\mu_ {q i} - \mu_ {p i}) ^ {2}}{\sigma_ {p i} ^ {2}}\right) \\ + \sigma_ {p i} ^ {2} + \mu_ {p i} ^ {2} - \log \sigma_ {q i} ^ {2} - 1), \tag {12} \\ \end{array}
$$
where  $\sigma_q$ ,  $\mu_q$  and  $\sigma_p$ ,  $\mu_p$  are the parameters of  $q_{\phi}$  and  $p_{\theta}$  respectively.  $n$  denotes the hidden size. The KL term vanishes to 0 when and only when  $q_{\phi}$  and  $p_{\theta}$  collapse to  $r(\mathbf{z})$ , which is the normal distribution. As we explained in Section 3.2, KL won't be 0 when we apply BN in  $q_{\phi}$ . We then prove that when  $q_{\phi}$  collapses to  $p_{\theta}$ , the KL term is not the minima (details in Section 2 of the Appendix) so that  $KL(q_{\phi}(\mathbf{z}|\mathbf{x},\mathbf{y})||p_{\theta}(\mathbf{z}|\mathbf{x}))$  won't be 0. In this way, we can avoid the posterior collapse in CVAE. Algorithm 2 shows the training details.
Algorithm 2 BN-CVAE training.