SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhekai Du, Jingjing Li*
School of Computer Science and Engineering
zhekaid@std.uestc.edu.cn, lijin117@yeah.net
Active Domain Adaptation (ADA) has emerged as an attractive technique for assisting domain adaptation by actively annotating a small subset of target samples. Most ADA methods focus on measuring the target representativeness beyond traditional active learning criteria to handle the domain shift problem, while leaving the uncertainty estimation to be performed by an uncalibrated deterministic model. In this work, we introduce a probabilistic framework that captures both data-level and prediction-level uncertainties beyond a point estimate. Specifically, we use variational inference to approximate the joint posterior distribution of latent representation and model prediction. The variational objective of labeled data can be formulated by a variational autoencoder and a latent diffusion classifier, and the objective of unlabeled data can be implemented in a knowledge distillation framework. We utilize adversarial learning to ensure an invariant latent space. The resulting diffusion classifier enables efficient sampling of all possible predictions for each individual to recover the predictive distribution. We then leverage a t-test-based criterion upon the sampling and select informative unlabeled target samples based on the p-value, which encodes both prediction variability and cross-category ambiguity. Experiments on both ADA and Source-Free ADA settings show that our method provides more calibrated predictions than previous ADA methods and achieves favorable performance on three domain adaptation datasets.
Machine learning algorithms heavily depend on large amounts of labeled data. However, in many real-world scenarios, the distribution of data can change over time or across different domains, which challenges the training of models to perform consistently in all situations. Unsupervised Domain Adaptation (UDA) [1, 2, 3] addresses this problem by transferring knowledge learned from a labeled source domain to an unlabeled target domain. Despite impressive achievements, UDA struggles to bridge the performance gap with fully supervised methods due to the lack of target-supervised information. Recognizing this limitation, a promising alternative is Active Domain Adaptation (ADA) [4, 5, 6], which actively annotates a small subset of target data to greatly benefit the UDA model.
In ADA, the primary concern revolves around designing a criterion for selecting the most informative target samples that can have the maximum impact on performance once annotated. Existing ADA methods have been mainly enlightened by traditional Active Learning (AL) methods [7, 8, 9, 10] and rely on criteria that consider both predictive uncertainty and representativeness of the entire distribution. While they have shown empirical success, they usually handle the distribution shift by further measuring the representativeness of the target domain through various criteria (e.g., the output of a domain discriminator [11], free energy score [5], weighted clustering [4]). However, these
methods still estimate uncertainty based on metrics like confidence [12], distinctive margin [6], and entropy [4], obtained from a deterministic model. It is worth noting that distribution shift can easily lead to overconfident predictions for deep models [13], especially deterministic ones [14, 15], which results in poorly calibrated point estimates and unreliable uncertainty estimation in ADA.
In this work, we aim to accurately estimate the posterior distribution of the target variable  $\mathbf{y}$  given the input  $\mathbf{x}$  for target domain samples (i.e.,  $p(\mathbf{y}|\mathbf{x})$ ), by modeling the uncertainty in both data generation and model prediction processes. To achieve this, we utilize diffusion-based generative models [16, 17, 18] to explore the implicit predictive distribution beyond a point estimate for each target domain sample. Specifically, we harness the capabilities of diffusion models for classification [19], which involve forward and reverse diffusion chains to generate predictions from the conditional predictive distribution, without any assumptions on the parametric form of  $p(\mathbf{y}|\mathbf{x})$ . Notably, another work [20] also tackles the miscalibration issue in ADA by considering the model prediction as a distribution on the probability simplex and introducing a Dirichlet prior over predictive distributions. However, such a constrained parametric form might not be effective if the prior distribution fails to accurately capture the predictive uncertainty [19]. Additionally, their approach only focuses on modeling the probability distribution in the model prediction space, neglecting the inherent uncertainty present in the data representation from a generative perspective.
To approximate the posterior distribution of both latent data representation and model prediction, we use variational inference by optimizing the evidence lower bound (ELBO) corresponding to the log-likelihood of all data points. Concretely, we show that for labeled data, the training objective can be formulated by Variational Autoencoder (VAE) [21] and a diffusion-based classifier [19] in the latent space. For unlabeled data, they can also be effective utilized by optimizing the ELBO, which can be implemented in a knowledge distillation framework. With the formulation, we establish a two-stage training procedure where we first learn a guided classifier for mean estimate and then train a diffusion probabilistic model guided by that for predictive distribution recovery. To ensure a shared latent embedding that can be used to solve tasks for both labeled and unlabeled data, we incorporate adversarial learning [1, 22], resulting in a Diffusion-based Adversarial Probabilistic Model (DAPM). With a collection of predictions generated by DAPM for each unlabeled target sample, we conduct t-test [23] between scores of the two most predicted classes and select the most informative samples based on the p-value, which generally takes into account the sampling scale, prediction variability and the cross-category ambiguity to estimate the prediction uncertainty for active annotation.
Our contributions: 1) We formulate ADA in a probabilistic framework for uncertainty estimate, which leverages the abilities of VAE and diffusion-based classification models to capture distributions of both data and prediction. 2) We conduct a two-stage training procedure and use adversarial learning to ensure an invariant latent space. A t-test-based criterion is utilized to estimate informativeness from multiple aspects. 3) We show that our method can naturally handle active learning for both UDA and Source-Free DA (SFDA). Experiments on three datasets show the effectiveness of our method.
Active Learning [24, 25] aims to tap the fully supervised performance by only annotating a small subset of training data. The selection of informative samples can be made using a variety of criteria. Committee-based methods [26, 27, 28, 29] leverage diverse classifiers and evaluate the data informativeness based on their disagreement. Representativeness-based methods [30, 31] aim to select samples that are diverse or representative enough of the entire training distribution, typically through clustering [32] or core-set selection [33]. Uncertainty-based methods resort to various uncertainty heuristics such as entropy [34], confidence [35], and best-vs-second-best score [36], etc, to select the most uncertain instances. However, these AL methods will fail when there is a distribution shift between labeled and unlabeled data, limiting their applications in domain adaptation.
Active Domain Adaptation. A series of ADA works have been conducted to select the most informative target samples under domain shift. As an early work, AADA [37] applies a domain discriminator to evaluate the domainness of target samples and weights it with entropy-based uncertainty. TQS [11] designs a transferrable query criterion based on a classifier committee and a domain discriminator to handle domain shift. CLUE [4] performs an entropy-weighted clustering to select both uncertain and diverse samples. SDM [6] uses a distinctive margin loss for training and selects data lying in the margin. EADA [5] leverages the free-energy bias and uses an energy-based criterion to evaluate
the uncertainty and domainness of target samples. Despite the empirical success, these methods mainly focus on compensating for the domain gap by incorporating target-representativeness into the query function. However, the uncertainty criteria they used are still based on a point estimate from a deterministic model, which can be easily miscalibrated for out-of-distribution data [38]. To remedy this issue, DUC [20] places a Dirichlet prior on the class probability and evaluates the predictive uncertainty with a probabilistic view. However, they only consider the uncertainty in model outputs. Besides, the true predictive distribution may not be captured by such a restricted distribution form.
Diffusion-based generative models have been widely appreciated for their impressive capabilities of generating high quality and diverse samples [17, 16, 39, 18]. While most works focus on high-dimensional generative tasks such as (conditional) image generation [39, 16], super-resolution [40, 41], image inpainting [42, 43], etc., some attempts have been made to apply diffusion models to categorical data [44] and discrete data [45], indicating their potential in classification tasks. Recently, Han et al. [19] utilize diffusion models to recover predictive distributions in regression and classification tasks, verifying their effectiveness for uncertainty estimation. Different from [19] that focuses on in-distribution scenarios, we consider a more challenging and realistic scenario of capturing predictive distribution in a cross-domain fashion. In addition, we also model the underlying variation of the data in low-dimensional latent space, which can be viewed as a data-level uncertainty and more favorable for downstream tasks compared to original image space [46].
In ADA, we have a source domain  $S = \{(\pmb{x}_i^s,y_i^s)\}_{i = 1}^{n_s}$  with  $n_{s}$  labeled samples, and a target domain  $\mathcal{T} = \{(x_i^t)\}_{i = 1}^{n_t}$  with  $n_t$  unlabeled samples, where  $\pmb{x}_i^s (\pmb{x}_i^t)$  is the input instance, and  $y_{i}^{s}\in \mathcal{V}$  is the corresponding label. It is assumed that both domains share a common label space  $\mathcal{V}$  but conform to different data distributions. Domain adaptation aims to train a model  $f_{\Omega}:\mathcal{X}\to \mathcal{Y}$  parameterized by  $\Omega$  that achieves good predictive performance on the target domain. To achieve this goal, the problem of ADA involves selecting a small subset of informative samples  $\mathcal{T}_l\subset \mathcal{T}$  for annotation with a budget  $\mathcal{B}$ , where  $\mathcal{B}\ll n_t$ , such that the model performance on the target domain can be maximally improved. The selection process is typically completed in multiple rounds, where in each round, the model queries  $b$  samples from  $\mathcal{T}_u$  (the remaining unlabeled samples in  $\mathcal{T}$ ) and adds them to  $\mathcal{T}_l$ . This process is repeated until the budget is exhausted. In this work, we examine active learning for both UDA and source-free UDA [47], where, besides using unlabeled target data, we can utilize the raw source data and a source-trained model, respectively, for the purposes of selection and adaptation.
3.1 Preliminary of Classification Diffusion Models for Uncertainty Estimation
In AL, a major concern in sample selection is predictive uncertainty. For a  $K$ -way classification problem, given an input variable  $\pmb{x}$ , the predictive uncertainty can be expressed as the posterior distribution over the predicted variable  $\pmb{y}^2$  after observing the training set  $\mathcal{D}$ , i.e.,  $p(\pmb{y} \mid \pmb{x}, \mathcal{D})$ . Most AL methods merely pay attention to accuracy and adopt a non-Bayesian approach to train the model, e.g., Maximum Likelihood Estimation (MLE) or Maximum A Posteriori (MAP). As frequentist techniques, they are not capable of inferring the variance of predictive distribution  $p(\pmb{y} \mid \pmb{x})$ , but merely provide a mean estimate  $\mathbb{E}[y \mid x]$  under the assumption of additive noise [19]. Taking a Bayesian view, the predictive uncertainty can be modeled by assuming distributions over network parameters [48, 49], while it involves expensive computation and intractable posterior inference. Some alternative methods include modeling the output of a neural network as a probability distribution over possible outcomes [50, 5] or adding the noise term in the model outputs [51]. As explicit modeling methods, they all assume a specific form in  $p(\pmb{y} \mid \pmb{x})$  (e.g., Gaussian or Dirichlet distribution).
In this work, we leverage the deep generative models to model the implicit predictive distribution. Specifically, we start with the classification diffusion models [19]. Re-denoted by  $\mathbf{y}_0 \in \mathbb{R}^K$  the one-hot label, the diffusion model first applies a forward diffusion process  $q(\mathbf{y}_{1:T} \mid \mathbf{y}_0, \mathbf{x})$  to original data by iteratively perturbing it to latent representations  $\mathbf{y}_{1:T}$  with Gaussian noises. For each step  $t$ ,
$$
q \left(\boldsymbol {y} _ {t} \mid \boldsymbol {y} _ {t - 1}, \boldsymbol {x}\right) = \mathcal {N} \left(\boldsymbol {y} _ {t}; \sqrt {1 - \beta_ {t}} \boldsymbol {y} _ {t - 1} + \left(1 - \sqrt {1 - \beta_ {t}}\right) f _ {\Omega} (\boldsymbol {x}), \beta_ {t} \boldsymbol {I}\right), \tag {1}
$$
where  $\beta_{t} \in (0,1)$  is the noise variance and fixed as hyperparameter. Here the model output  $f_{\Omega}(\pmb{x})$  (mean estimate) is injected in the forward process to act as the prior knowledge about the relationship
2We use bold font to denote a probability vector:  $\pmb {y} = [y_{1},\dots ,y_{K}]$
between  $\pmb{x}$  and  $\pmb{y}_0$ , so that with a sufficiently large  $T$  and proper schedule  $\{\beta_t\}_{t=1}^T$ , we have
$$
p \left(\boldsymbol {y} _ {T} \mid \boldsymbol {x}\right) = \mathcal {N} \left(f _ {\Omega} (\boldsymbol {x}), \boldsymbol {I}\right). \tag {2}
$$
Note that the specific form of the forward diffusion process enables an efficient sampling for arbitrary steps in a closed form:
$$
q \left(\boldsymbol {y} _ {t} \mid \boldsymbol {y} _ {0}, \boldsymbol {x}\right) = \mathcal {N} \left(\boldsymbol {y} _ {t}; \sqrt {\bar {\alpha} _ {t}} \boldsymbol {y} _ {0} + \left(1 - \sqrt {\bar {\alpha} _ {t}}\right) f _ {\Omega} (\boldsymbol {x}), (1 - \bar {\alpha} _ {t}) \boldsymbol {I}\right), \tag {3}
$$
with  $\alpha_{t} := 1 - \beta_{t}$  and  $\bar{\alpha}_{t} := \prod_{s=0}^{t} \alpha_{s}$ . An appreciable feature of diffusion models is that if we know the posterior distribution  $q(\pmb{y}_{t-1} \mid \pmb{y}_t, \pmb{x})$ , we can sample  $y_{T} \sim \mathcal{N}(f_{\Omega}(\pmb{x}), \pmb{I})$  and run the reverse diffusion process to gradually recover the original data from  $p(\pmb{y}_0 \mid \pmb{x})$  under the guidance of  $f_{\Omega}(x)$ . Unfortunately,  $q(\pmb{y}_{t-1} \mid \pmb{y}_t, \pmb{x})$  is actually intractable, and a feasible alternative is to use a model  $p_{\theta}(\pmb{y}_{t-1} \mid \pmb{y}_t, \pmb{x})$  for approximation, which can be trained with the following ELBO:
$$
\log p _ {\theta} \left(\boldsymbol {y} _ {0} \mid \boldsymbol {x}\right) \geq \mathbb {E} _ {q \left(\boldsymbol {y} _ {1: T} \mid \boldsymbol {y} _ {0}, \boldsymbol {x}\right)} \left[ \log \frac {p _ {\theta} \left(\boldsymbol {y} _ {0 : T} \mid \boldsymbol {x}\right)}{q \left(\boldsymbol {y} _ {1 : T} \mid \boldsymbol {y} _ {0} , \boldsymbol {x}\right)} \right] := \mathcal {L} _ {E L B O} ^ {d} (\boldsymbol {x}, \boldsymbol {y} _ {0}) := \mathcal {L} _ {0} + \sum_ {t = 2} ^ {T} \mathcal {L} _ {t - 1} + \mathcal {L} _ {T}, \tag {4}
$$
$$
\mathcal {L} _ {0} := \mathbb {E} _ {q} \left[ - \log p _ {\theta} \left(\boldsymbol {y} _ {0} \mid \boldsymbol {y} _ {1}, \boldsymbol {x}\right) \right], \tag {5}
$$
$$
\mathcal {L} _ {t - 1} := \mathbb {E} _ {q} \left[ D _ {\mathrm {K L}} \left(q \left(\boldsymbol {y} _ {t - 1} \mid \boldsymbol {y} _ {t}, \boldsymbol {y} _ {0}, \boldsymbol {x}\right) \| p _ {\theta} \left(\boldsymbol {y} _ {t - 1} \mid \boldsymbol {y} _ {t}, \boldsymbol {x}\right)\right) \right], \tag {6}
$$
$$
\mathcal {L} _ {T} := \mathbb {E} _ {q} \left[ D _ {\mathrm {K L}} \left(q \left(\boldsymbol {y} _ {T} \mid \boldsymbol {y} _ {0}, \boldsymbol {x}\right) \| p \left(\boldsymbol {y} _ {T} \mid \boldsymbol {x}\right)\right) \right], \tag {7}
$$
where  $D_{KL}$  is the Kullback-Leibler (KL) divergence. It is worth noting that scheduling small  $\beta_{t}$  will make  $q\left(\boldsymbol{y}_{t-1} \mid \boldsymbol{y}_{t}, \boldsymbol{y}_{0}, \boldsymbol{x}\right)$  still a Gaussian [52], which can be explicitly formulated using Bayes theorem. Therefore  $\mathcal{L}_{t-1}$  can be evaluated in a closed form.  $\mathcal{L}_T$  contains no estimable parameter and is assumed to be sufficiently small and thus can be ignored. By adopting the appropriate form of  $p_{\theta}(\boldsymbol{y}_{t-1} \mid \boldsymbol{y}_{t}, \boldsymbol{x})$  and reparameterization like DDPM [17], the model can be trained effectively by being tasked at predicting the forward noise  $\epsilon$  for sampling  $y_{t}$ , with  $\epsilon_{\theta}(\boldsymbol{x}, \boldsymbol{y}_{t}, f_{\Omega}(\boldsymbol{x}), t)$ .
3.2 Diffusion-Based Adversarial Probabilistic Model for ADA
The above conditional diffusion model admits the sampling of multiple predictions from  $p(\boldsymbol{y} \mid \boldsymbol{x}, \mathcal{D})$  for statistical analysis. However, it assumes that the evaluated sample  $\boldsymbol{x}$  and the training set  $\mathcal{D}$  comes from the same distribution, which is not the case in UDA. Aware of this, we aim to build a model that induces a common latent space  $\mathcal{Z}$ , where the source and target samples share the same representations. We first consider the following generative process defined with the joint probability distributions (We omit the subscript  $i$  to represent arbitrary data point in the dataset):
$$
p \left(\boldsymbol {y} _ {0: T} ^ {s}, \boldsymbol {x} ^ {s}, \boldsymbol {z} ^ {s}\right) = p \left(\boldsymbol {z} ^ {s}\right) p \left(\boldsymbol {x} ^ {s} \mid \boldsymbol {z} ^ {s}\right) p \left(\boldsymbol {y} _ {0: T} ^ {s} \mid \boldsymbol {z} ^ {s}, \boldsymbol {x} ^ {s}\right), \tag {8}
$$
$$
p \left(\boldsymbol {y} _ {0: T} ^ {t}, \boldsymbol {x} ^ {t}, \boldsymbol {z} ^ {t}\right) = p \left(\boldsymbol {z} ^ {t}\right) p \left(\boldsymbol {x} ^ {t} \mid \boldsymbol {z} ^ {t}\right) p \left(\boldsymbol {y} _ {0: T} ^ {t} \mid \boldsymbol {z} ^ {t}, \boldsymbol {x} ^ {t}\right), \tag {9}
$$
where  $z^s (z^t)\in \mathcal{Z}$  is the latent representation. To model the inference process of  $z^{s}(z^{t})$ , we employ a shared encoder parameterized by  $\rho$  for both the source and target domains, supposing a domain-shared embedding. And we use two separate decoders with parameters  $\phi$  and  $\psi$  for the source and target reconstructions, respectively, since the generative process requires domain-specific information encoded in the parameter. Due to intractable posterior distributions, we then aim to learn latent variables for both domains using variational inference, with the following variational objectives:
Objective for Labeled Source Samples. For a labeled source sample  $(\boldsymbol{x}^s, \boldsymbol{y}_0^s)$ , our goal is to learn both the low-dimensional embedding  $z^s$  and latent class variable  $y_{1:T}^s$ , which can be achieved by maximizing the ELBO of the marginal log-likelihood as follows:
$$
\begin{array}{l} \log p \left(\boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0} ^ {s}\right) = \log \int p \left(\boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0: T} ^ {s}, \boldsymbol {z} ^ {s}\right) d \boldsymbol {y} _ {1: T} ^ {s} d \boldsymbol {z} ^ {s} \\ \geq \mathbb {E} _ {q \left(\boldsymbol {y} _ {1: T} ^ {s}, \boldsymbol {z} ^ {s} | \boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0} ^ {s}\right)} \left[ \log \frac {p \left(\boldsymbol {y} _ {0 : T} ^ {s} , \boldsymbol {x} ^ {s} , \boldsymbol {z} ^ {s}\right)}{q \left(\boldsymbol {y} _ {1: T} ^ {s} , \boldsymbol {z} ^ {s} \mid \boldsymbol {x} ^ {s} , \boldsymbol {y} _ {0} ^ {s}\right)} \right] \\ = \underbrace {\mathbb {E} _ {q _ {\rho} \left(\boldsymbol {z} ^ {s} \mid \boldsymbol {x} ^ {s} , \boldsymbol {y} _ {0} ^ {s}\right)} \left[ \log p _ {\phi} \left(\boldsymbol {x} ^ {s} \mid \boldsymbol {z} ^ {s}\right) \right] - D _ {K L} \left(q _ {\rho} \left(\boldsymbol {z} ^ {s} \mid \boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0} ^ {s}\right) \| p \left(\boldsymbol {z} ^ {s}\right)\right)} _ {\text {V A E}} + \tag {10} \\ \end{array}
$$
$$
\mathbb {E} _ {\boldsymbol {z} ^ {s} \sim q _ {\rho} (\boldsymbol {z} ^ {s} | \boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0} ^ {s})} \underbrace {\mathbb {E} _ {q (\boldsymbol {y} _ {1 : T} ^ {s} | \boldsymbol {z} ^ {s} , \boldsymbol {y} _ {0} ^ {s})} \left[ \frac {p _ {\theta} (\boldsymbol {y} _ {0 : T} ^ {s} | \boldsymbol {z} ^ {s})}{q (\boldsymbol {y} _ {1 : T} ^ {s} | \boldsymbol {y} _ {0} ^ {s} , \boldsymbol {z} ^ {s})} \right]} _ {\mathcal {L} _ {E L B O} ^ {d} (\boldsymbol {z} ^ {s}, \boldsymbol {y} _ {0} ^ {s})} := \mathcal {L} _ {E L B O} ^ {s}.
$$
Eq. (10) implies a standard VAE and a classification diffusion model in the latent space. We provide the detailed derivation in Appendix A.1. To ensure discriminative classification boundaries in  $\mathcal{Z}$ , we additionally incorporate a deterministic classifier with parameter  $\omega$  in the training objective. Then the overall objective for the labeled source data can be expressed as:
$$
\mathcal {L} _ {s} = - \mathcal {L} _ {E L B O} ^ {s} + \mathbb {E} _ {\left(\boldsymbol {x} ^ {s}, \boldsymbol {y} _ {0} ^ {s}\right) \sim \mathcal {S}} \mathbb {E} _ {q _ {\rho} \left(\boldsymbol {z} ^ {s} \mid \boldsymbol {x} ^ {s}\right)} \left[ - \log p _ {\omega} \left(\boldsymbol {y} _ {0} ^ {s} \mid \boldsymbol {z} ^ {s}\right) \right]. \tag {11}
$$
It is worth noting that the parameters of  $f_{\Omega}$  have been further decomposed into  $\{\rho, \omega\}$  in our case.
Objective for Labeled Target Samples. For target samples with annotation, we treat them the same as source-labeled data in training. The only difference is that we use parameter  $\psi$  for target-specific reconstruction. Limited by space, we give the full expression of the training loss  $\mathcal{L}_t^l$  in Appendix A.2.
Objective for Unlabeled Target Samples. The training objective for unlabeled target samples can not be carried out in a supervised fashion due to the absence of ground-truth labels. Therefore, we design the following variational lower bound for them by treating  $y_0^t$  and  $z^t$  as latent variables:
$$
\begin{array}{l} \log p \left(\boldsymbol {x} ^ {t}\right) = \log \int p \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} _ {0} ^ {t}, \boldsymbol {z} ^ {t}\right) d \boldsymbol {y} _ {0} ^ {t} d \boldsymbol {z} ^ {t} \\ \geq \mathbb {E} _ {q (\boldsymbol {y} _ {0} ^ {t}, \boldsymbol {z} ^ {t} | \boldsymbol {x} ^ {t})} \left[ \log \frac {p (\boldsymbol {x} ^ {t} , \boldsymbol {y} _ {0} ^ {t} , \boldsymbol {z} ^ {t})}{q (\boldsymbol {y} _ {0} ^ {t} , \boldsymbol {z} ^ {t} | \boldsymbol {x} ^ {t})} \right] \\ = \underbrace {\mathbb {E} _ {q _ {\rho} \left(\boldsymbol {z} ^ {t} \mid \boldsymbol {x} ^ {t}\right)} \left[ \log p _ {\psi} \left(\boldsymbol {x} ^ {t} \mid \boldsymbol {z} ^ {t}\right) \right] - D _ {K L} \left(q _ {\rho} \left(\boldsymbol {z} ^ {t} \mid \boldsymbol {x} ^ {t}\right) \| p \left(\boldsymbol {z} ^ {t}\right)\right)} _ {\text {V A E}} - \tag {12} \\ \underbrace {\mathbb {E} _ {\boldsymbol {z} ^ {t} \sim q _ {\rho} (\boldsymbol {z} ^ {t} | \boldsymbol {x} ^ {t})} \left[ D _ {K L} (q (\boldsymbol {y} _ {0} ^ {t} \mid \boldsymbol {x} ^ {t}) \| p _ {\omega} (\boldsymbol {y} _ {0} ^ {t} \mid \boldsymbol {z} ^ {t}) \right]} _ {\mathcal {L} _ {K D}} := \mathcal {L} _ {E L B O} ^ {u}. \\ \end{array}
$$