SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tao Yang $^{1*}$ , Yuwang Wang $^{2\dagger}$ , Yan Lu $^{3}$ , Nanning Zheng $^{1\dagger}$
yt14212@stu.xjtu.edu.cn,
$^{1}$ National Key Laboratory of Human-Machine Hybrid Augmented Intelligence,
National Engineering Research Center for Visual Information and Applications,
and Institute of Artificial Intelligence and Robotics,
Xi'an Jiaotong University,
$^{2}$ Tsinghua University, Shanghai AI Laboratory,
<sup>3</sup>Microsoft Research Asia
https://github.com/thomasmry/DisDiff
Targeting to understand the underlying explainable factors behind observations and modeling the conditional generation process on these factors, we connect disentangled representation learning to Diffusion Probabilistic Models (DPMs) to take advantage of the remarkable modeling ability of DPMs. We propose a new task, disentanglement of (DPMs): given a pre-trained DPM, without any annotations of the factors, the task is to automatically discover the inherent factors behind the observations and disentangle the gradient fields of DPM into sub-gradient fields, each conditioned on the representation of each discovered factor. With disentangled DPMs, those inherent factors can be automatically discovered, explicitly represented, and clearly injected into the diffusion process via the sub-gradient fields. To tackle this task, we devise an unsupervised approach named DisDiff, achieving disentangled representation learning in the framework of DPMs. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness of DisDiff.
As one of the most successful generative models, diffusion probabilistic models (DPMs) achieve remarkable performance in image synthesis. They use a series of probabilistic distributions to corrupt images in the forward process and train a sequence of probabilistic models converging to image distribution to reverse the forward process. Despite the remarkable success of DPM in tasks such as image generation [41], text-to-images [37], and image editing [32], little attention has been paid to representation learning [49] based on DPM. Diff-AE [33] and PDAE [49] are recently proposed methods for representation learning by reconstructing the images in the DPM framework. However, the learned latent representation can only be interpreted relying on an extra pre-trained with predefined semantics linear classifier. In this paper, we will refer DPMs exclusively to the Denoising Diffusion Probabilistic Models (DDPMs) [19].
On the other hand, disentangled representation learning [17] aims to learn the representation of the underlying explainable factors behind the observed data and is thought to be one of the possible ways for AI to understand the world fundamentally. Different factors correspond to different kinds of image variations, respectively and independently. Most of the methods learn the disentangled representation
*Work done during internships at Microsoft Research Asia.
†Corresponding authors: wang-yuwang@mail.tsinghua.edu.cn; nnzheng@mail.xjtu.edu.cn.
based on generative models, such as VAE [16, 5, 23] and GAN [27]. The VAE-based methods have an inherent trade-off between the disentangling ability and generating quality [16, 5, 23]. The GAN-based methods suffer from the problem of reconstruction due to the difficulty of gan-inversion [44].
In this paper, we connect DPM to disentangled representation learning and propose a new task: the disentanglement of DPM. Given a pre-trained DPM model, the goal of disentanglement of DPM is to learn disentangled representations for the underlying factors in an unsupervised manner, and learn the corresponding disentangled conditional sub-gradient fields, with each conditioned on the representation of each discovered factor, as shown in Figure 1.
The benefits of the disentanglement of DPM are two-fold: (i) It enables totally unsupervised controlling of images by automatically discovering the inherent semantic factors behind the image data. These factors help to extend the DPM conditions information from human-defined ones such as annotations [49]/image-text pairs [21], or supervised pre-trained models [22] such as CLIP [34]. One can also flexibly sample partial conditions on the part of the information introduced by the superposition of the sub-gradient field, which is novel in existing DPM works. (ii) DPM has remarkable performance on image generation quality and is naturally friendly for the inverse problem, e.g., the inversion of DDIM [39], PDAE. Compared to VAE (trade-off between the disentangling ability and generating quality) or GAN (problem of gan-inversion), DPM is a better framework for disentangled representation learning. Besides, as Locatello et al. [30] points out, other inductive biases should be proposed except for total correlation. DPM makes adopting constraints from all different timesteps possible as a new type of inductive bias. Further, as Srivastava et al. [42] points out, and the data information includes: factorized and non-factorized. DPM has the ability to sample non-factorized (non-conditioned) information [18], which is naturally fitting for disentanglement.
To address the task of disentangling the DPM, we further propose an unsupervised solution for the disentanglement of a pre-trained DPM named DisDiff. DisDiff adopts an encoder to learn the disentangled presentation for each factor and a decoder to learn the corresponding disentangled conditional sub-gradient fields. We further propose a novel Disentangling Loss to make the encoded representation satisfy the disentanglement requirement and reconstruct the input image.
- We present a new task: disentanglement of DPM, disentangling a DPM into several disentangled sub-gradient fields, which can improve the interpretability of DPM.
- We build an unsupervised framework for the disentanglement of DPM, DisDiff, which learns a disentangled representation and a disentangled gradient field for each factor.
- We propose enforcing constraints on representation through the diffusion model's classifier guidance and score-based conditioning trick.
Diffusion Probabilistic Models DPMs have achieved comparable or superior image generation quality [38, 40, 19, 41, 20] than GAN [13]. Diffusion-based image editing has drawn much attention, and there are mainly two categories of works. Firstly, image-guided works edit an image by mixing the latent variables of DPM and the input image [7, 31, 32]. However, using images to specify the attributes for editing may cause ambiguity, as pointed out by Kwon et al. [25]. Secondly, the classifier-guided works [9, 1, 28] edit images by utilizing the gradient of an extra classifier. These methods require calculating the gradient, which is costly. Meanwhile, these methods require annotations or models pre-trained with labeled data. In this paper, we propose DisDiff to edit the image in an unsupervised way. On the other hand, little attention has been paid to representation learning in the literature on the diffusion model. Two related works are Diff-ae [33] and PDAE [49]. Diff-ae [33] proposes a diffusion-based auto-encoder for image reconstruction. PDAE [49] uses a pre-trained DPM to build an auto-encoder for image reconstruction. However, the latent representation learned by these two works does not respond explicitly to the underlying factors of the dataset.
Disentangled Representation Learning Bengio et al. [2] introduced disentangled representation learning. The target of disentangled representation learning is to discover the underlying explanatory factors of the observed data. The disentangled representation is defined as each dimension of the representation corresponding to an independent factor. Based on such a definition, some VAE-based
works achieve disentanglement [5, 23, 16, 3] only by the constraints on probabilistic distributions of representations. Locatello et al. [30] point out the identifiable problem by proving that only these constraints are insufficient for disentanglement and that extra inductive bias is required. For example, Yang et al. [47] propose to use symmetry properties modeled by group theory as inductive bias. Most of the methods of disentanglement are based on VAE. Some works based on GAN, including leveraging a pre-trained generative model [36]. Our DisDiff introduces the constraint of all time steps during the diffusion process as a new type of inductive bias. Furthermore, DPM can sample non-factorized (non-conditioned) information [18], naturally fitting for disentanglement. In this way, we achieve disentanglement for DPMs.
3.1 Diffusion Probabilistic Models (DPM)
We take DDPM [19] as an example. DDPM adopts a sequence of fixed variance distributions  $q(x_{t}|x_{t - 1})$  as the forward process to collapse the image distribution  $p(x_0)$  to  $\mathcal{N}(0,I)$ . These distributions are
$$
q \left(x _ {t} \mid x _ {t - 1}\right) = \mathcal {N} \left(x _ {t}; \sqrt {1 - \beta_ {t}} x _ {t - 1}, \beta_ {t} I\right). \tag {1}
$$
We can then sample  $x_{t}$  using the following formula  $x_{t} \sim \mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}} x_{0},(1 - \bar{\alpha}_{t}))$ , where  $\alpha_{t} = 1 - \beta_{t}$  and  $\bar{\alpha}_t = \Pi_{t = 1}^t\alpha_t$ , i.e.,  $x_{t} = \sqrt{\bar{\alpha}_{t}} x_{0} + \sqrt{1 - \bar{\alpha}_{t}\epsilon}$ . The reverse process is fitted by using a sequence of Gaussian distributions parameterized by  $\theta$ :
$$
p _ {\theta} \left(x _ {t - 1} \mid x _ {t}\right) = \mathcal {N} \left(x _ {t}; \mu_ {\theta} \left(x _ {t}, t\right), \sigma_ {t} I\right), \tag {2}
$$
where  $\mu_{\theta}(x_t,t)$  is parameterized by an Unet  $\epsilon_{\theta}(x_t,t)$ , it is trained by minimizing the variational upper bound of negative log-likelihood through:
$$
\mathcal {L} _ {\theta} = \underset {x _ {0}, t, \epsilon} {\mathbb {E}} \| \epsilon - \epsilon_ {\theta} (x _ {t}, t) \|. \tag {3}
$$
3.2 Representation learning from DPMs
The classifier-guided method [9] uses the gradient of a pre-trained classifier,  $\nabla_{x_t}\log p(y|x_t)$ , to impose a condition on a pre-trained DPM and obtain a new conditional DPM:  $\mathcal{N}(x_t;\mu_\theta (x_t,t) + \sigma_t\nabla_{x_t}\log p(y|x_t),\sigma_t)$ . Based on the classifier-guided sampling method, PDAE [49] proposes an approach for pre-trained DPM by incorporating an auto-encoder. Specifically, given a pre-trained DPM, PDAE introduces an encoder  $E_{\phi}$  to derive the representation by equation  $z = E_{\phi}(x_0)$ . They use a decoder estimator  $G_{\psi}(x_t,z,t)$  to simulate the gradient field  $\nabla_{x_t}\log p(z|x_t)$  for reconstructing the input image.
By this means, they create a new conditional DPM by assembling the unconditional DPM as the decoder estimator. Similar to regular DPM  $\mathcal{N}(x_t;\mu_\theta (x_t,t) + \sigma_tG_\psi (x_t,z,t),\sigma_t)$ , we can use the following objective to train encoder  $E_{\phi}$  and the decoder estimator  $G_{\psi}$ :
$$
\mathcal {L} _ {\psi} = \underset {x _ {0}, t, \epsilon} {\mathbb {E}} \| \epsilon - \epsilon_ {\theta} (x _ {t}, t) + \frac {\sqrt {\alpha_ {t}} \sqrt {1 - \bar {\alpha} _ {t}}}{\beta_ {t}} \sigma_ {t} G _ {\psi} (x _ {t}, z, t) \|. \tag {4}
$$
In this section, we initially present the formulation of the proposed task, Disentangling DPMs, in Section 4.1. Subsequently, an overview of DisDiff is provided in Section 4.2. We then elaborate on the detailed implementation of the proposed Disentangling Loss in Section 4.3, followed by a discussion on balancing it with the reconstruction loss in Section 4.4.
4.1 Disentanglement of DPM
We assume that dataset  $\mathcal{D} = \{x_0 | x_0 \sim p(x_0)\}$  is generated by  $N$  underlying ground truth factors  $f^c$ , where each  $c \in \mathcal{C} = \{1, \dots, N\}$ , with the data distribution  $p(x_0)$ . This implies that the underlying factors condition each sample. Moreover, each factor  $f^c$  follows the distribution  $p(f^c)$ , where  $p(f^c)$  denotes the distribution of factor  $c$ . Using the Shapes3D dataset as an example, the underlying
(a) PGM of disentanglement
(b) Image Space
(c) Sampled Images
Figure 1: Illustration of disentanglement of DPMs. (a) is the diagram of Probabilistic Graphical Models (PGM). (b) is the diagram of image space. (c) is the demonstration of sampled images in (b). Surface indicates the conditional data distribution of a single factor  $p(x|f^c)$ . Different colors correspond to various factors. Here, we show three factors: object color, background color, and floor color. Arrows are gradient fields  $\nabla_{x_t} \log p(f^c | x_t)$  modeled by using a decoder  $G_{\phi}^{c}(x_t, t, z^c)$ . The black points are the sampled images shown in (b).
concept factors include background color, floor color, object color, object shape, object scale, and pose. We illustrate three factors (background color, floor color, and object color) in Figure 1 (c). The conditional distributions of the factors are  $\{p(x_0|f^c)|c\in \mathcal{C}\}$ , each of which can be shown as a curved surface in Figure 1 (b). The relation between the conditional distribution and data distribution can be formulated as: The conditional distributions of the factors, denoted as  $\{p(x_0|f^c)|c\in \mathcal{C}\}$ , can each be represented as a curved surface in Figure 1 (b). The relationship between the conditional distribution and the data distribution can be expressed as follows:
$$
p \left(x _ {0}\right) = \int p \left(x _ {0} \mid f ^ {c}\right) p \left(f ^ {c}\right) d f ^ {c} = \int \dots \int p \left(x _ {0} \mid f ^ {1}, \dots , f ^ {N}\right) p \left(f ^ {1}\right) \dots p \left(f ^ {N}\right) d f ^ {1} \dots d f ^ {N}, \tag {5}
$$
this relationship holds true because  $f^1, \ldots, f^N$  and  $x_0$  form a v-structure in Probabilistic Graphical Models (PGM), as Figure 1 (a) depicts. A DPM learns a model  $\epsilon_{\theta}(x_t, t)$ , parameterized by  $\theta$ , to predict the noise added to a sample  $x_t$ . This can then be utilized to derive a score function:  $\nabla_{x_t} \log p(x_t) = 1 / \sqrt{\alpha_t - 1} \cdot \epsilon_{\theta}(x_t, t)$ . Following [9] and using Bayes' Rule, we can express the score function of the conditional distribution as:
$$
\nabla_ {x _ {t}} \log p \left(x _ {t} \mid f ^ {c}\right) = \nabla_ {x _ {t}} \log p \left(f ^ {c} \mid x _ {t}\right) + \nabla_ {x _ {t}} \log p (x _ {t}), \tag {6}
$$
By employing the score function of  $p(x_{t}|f^{c})$ , we can sample data conditioned on the factor  $f^{c}$ , as demonstrated in Figure 1 (b). Furthermore, the equation above can be extended to one conditioned on a subset  $S \subseteq C$  by substituting  $f^{c}$  with  $S$ . The goal of disentangling a DPM is to model  $\nabla_{x_{t}}\log p(x_{t}|f^{c})$  for each factor  $c$ . Based on the equation above, we can learn  $\nabla_{x_t}\log p(f^c |x_t)$  for a pre-trained DPM instead, corresponding to the arrows pointing towards the curve surface in Figure 1 (b). However, in an unsupervised setting,  $f^{c}$  is unknown. Fortunately, we can employ disentangled representation learning to obtain a set of representations  $\{z^c |c = 1,\dots ,N\}$ . There are two requirements for these representations: (i) they must encompass all information of  $x_0$ , which we refer to as the completeness requirement, and (ii) there is a bijection,  $z^c\mapsto f^c$ , must exist for each factor  $c$ , which we designate as the disentanglement requirement. Utilizing these representations, the gradient  $\nabla_{x_t}\log p(z^c |x_t)$  can serve as an approximation for  $\nabla_{x_t}\log p(f^c |x_t)$ . In this paper, we propose a method named DisDiff as a solution for the disentanglement of a DPM.
4.2 Overview of DisDiff
The overview framework of DisDiff is illustrated in Figure 2. Given a pre-trained unconditional DPM on dataset  $\mathcal{D}$  with factors  $\mathcal{C}$ , e.g., a DDPM model with parameters  $\theta$ ,  $p_{\theta}(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_{\theta}(x_t, t), \sigma_t)$ , our target is to disentangle the DPM in an unsupervised manner. Given an input  $x_0 \in \mathcal{D}$ , for each factor  $c \in \mathcal{C}$ , our goal is to learn the disentangled representation  $z^c$  simultaneously and its corresponding disentangled gradient field  $\nabla_{x_t} \log p(z^c|x_t)$ . Specifically, for each representation  $z^c$ , we employ an encoder  $E_{\phi}$  with learnable parameters  $\phi$  to obtain the
(a) The networks architecture of DisDiff
(b) The demonstration of disentangling loss
Figure 2: Illustration of DisDiff. (a) Grey networks indicate the pre-trained Unet of DPM  $\epsilon_{\theta}(x_t,t)$ . Image  $x_0$  is first encoded to representations  $\{z^1,z^2,\dots z^N\}$  of different factors by encoder  $E_{\phi}$ $(N = 3$  in the figure), which are then decoded by decoder  $G_{\psi}$  to obtain the gradient field of the corresponding factor. We can sample the image under the corresponding condition with the obtained gradient field. (b) We first sample a factor  $c$  and decode the representation  $z^{c}$  to obtain the gradient field of the corresponding factor, which allows us to obtain the predicted  $x_0$  of that factor. At the same time, we can obtain the predicted  $\hat{x}_0$  of the original pre-trained DPM. We then encode the images into two different representations and calculate the disentangling loss based on them.
representations as follows:  $E_{\phi}(x_0) = \{E_{\phi}^1 (x_0),E_{\phi}^2 (x_0),\ldots ,E_{\phi}^N (x_0)\} = \{z^1,z^2,\dots ,z^N\}$ . As the random variables  $z^{1},\ldots ,z^{N}$  and  $x_0$  form a common cause structure, it is evident that  $z^{1},\ldots ,z^{N}$  are independent when conditioned on  $x_0$ . Additionally, this property also holds when conditioning on  $\epsilon$ . We subsequently formulate the gradient field, denoted as  $\nabla_{x_t}\log p(z^{\mathcal{S}}|x_t)$ , with  $z^{\mathcal{S}} = \{z^{c}|c\in \mathcal{S}\}$ , conditioned on a factor subset  $S\subseteq \mathcal{C}$ , as follows:
$$
\nabla_ {x _ {t}} \log p \left(z ^ {S} \mid x _ {t}\right) = \sum_ {c \in S} \nabla_ {x _ {t}} \log p \left(z ^ {c} \mid x _ {t}\right). \tag {7}
$$
Adopting the approach from [9, 49], we formulate the conditional reverse process as a Gaussian distribution, represented as  $p_{\theta}(x_{t - 1}|x_t,z^S)$ . In conjunction with Eq. 7, this distribution exhibits a shifted mean, which can be described as follows:
$$
\mathcal {N} \left(x _ {t - 1}; \mu_ {\theta} \left(x _ {t}, t\right) + \sigma_ {t} \sum_ {c \in S} \nabla_ {x _ {t}} \log p \left(z ^ {c} \mid x _ {t}\right), \sigma_ {t}\right). \tag {8}
$$
Directly employing  $\nabla_{x_t}\log p(z^c |x_t)$  introduces computational complexity and complicates the diffusion model training pipeline, as discussed in [18]. Therefore, we utilize a network  $G_{\psi}(x_t,z^c,t)$ , with  $c\in S$  and parameterized by  $\psi$ , to estimate the gradient fields  $\nabla_{x_t}\log p(z^S |x_t)$ . To achieve this goal, we first use  $\sum_{c\in \mathcal{C}}G_{\psi}(x_t,z^c,t)$  to estimate  $\nabla_{x_t}\log p(z^{\mathcal{C}}|x_t)$  based on Eq. 7. Therefore, we adopt the same loss of PDAE [49] but replace the gradient field with the summation of  $G_{\psi}(x_t,z^c,t)$  as
$$
\mathcal {L} _ {r} = \underset {x _ {0}, t, \epsilon} {\mathbb {E}} \| \epsilon - \epsilon_ {\theta} (x _ {t}, t) + \frac {\sqrt {\alpha_ {t}} \sqrt {1 - \bar {\alpha} _ {t}}}{\beta_ {t}} \sigma_ {t} \sum_ {c \in \mathcal {C}} G _ {\psi} \left(x _ {t}, z ^ {c}, t\right) \| \tag {9}
$$
The aforementioned equation implies that  $x_0$  can be reconstructed utilizing all disentangled representations when  $S = \mathcal{C}$ , thereby satisfying the completeness requirement of disentanglement. Secondly, in the following, we fulfill the disentanglement requirement and the approximation of gradient field  $\nabla_{x_t}\log p(z^c |x_t)$  with decoder  $G_{\psi}(x_t,z^c,t)$ . In the next section, we introduce a disentangling loss to address the remaining two conditions.
4.3 Disentangling Loss
In this section, we present the Disentangling Loss to address the disentanglement requirement and ensure that  $G_{\psi}(x_t,z^c,t)$  serves as an approximation of  $\nabla_{x_t}\log p(z^c|x_t)$ . Given the difficulty in identifying sufficient conditions for disentanglement in an unsupervised setting, we follow the previous works in disentangled representation literature, which propose necessary conditions that are effective in practice for achieving disentanglement. For instance, the group constraint in [47] and maximizing mutual information in [27, 45] serve as examples. We propose minimizing the mutual information between  $z^{c}$  and  $z^{k}$ , where  $k\neq c$ , as a necessary condition for disentangling a DPM.
Subsequently, we convert the objective of minimizing mutual information into a set of constraints applied to the representation space. We denote  $\hat{x}_0$  is sampled from the pre-trained unconditioned
DPM using  $\epsilon_{\theta}(x_t,t)$ ,  $\hat{x}_0^c$  is conditioned on  $z^c$  and sampled using  $p_{\theta}(x_{t - 1}|x_t,z^c)$ . Then we can extract the representations from the samples  $\hat{x}_0$  and  $\hat{x}_0^c$  with  $E_{\phi}$  as  $\hat{z}^{k} = E_{\phi}^{k}(\hat{x}_{0})$  and  $\hat{z}^{k|c} = E_{\phi}^{k}(\hat{x}_{0}^{c})$  respectively. According to Proposition 1, we can minimize the proposed upper bound (estimator) to minimize the mutual information.
Proposition 1. For a PGM in Figure 1 (a), we can minimize the upper bound (estimator) for the mutual information between  $z^c$  and  $z^k$  (where  $k \neq c$ ) by the following objective:
$$
\min  _ {k, c, x _ {0}, \hat {x} _ {0} ^ {c}} \| \hat {z} ^ {k | c} - \hat {z} ^ {k} \| - \| \hat {z} ^ {k | c} - z ^ {k} \| + \| \hat {z} ^ {c | c} - z ^ {c} \| \tag {10}
$$
The proof is presented in Appendix K. We have primarily drawn upon the proofs in CLUB [6] and VAE [24] as key literature sources to prove our claims. As outlined below, we can partition the previously mentioned objectives into two distinct loss function components.
Invariant Loss. In order to minimize the first part,  $\| \hat{z}^{k|c} - \hat{z}^k\|$ , where  $k \neq c$ , we need to minimize  $\| \hat{z}^{k|c} - \hat{z}^k\|$ , requiring that the  $k$ -th ( $k \neq c, k \in \mathcal{C}$ ) representation remains unchanged. We calculate the distance scalar between their  $k$ -th representation as follows:
$$
d _ {k} = \left\| \hat {z} ^ {k | c} - \hat {z} ^ {k} \right\|. \tag {11}
$$
We can represent the distance between each representation using a distance vector  $d = [d_1, d_2, \ldots, d_C]$ . As the distances are unbounded and their direct optimization is unstable, we employ the CrossEntropy loss to identify the index  $c$ , which minimizes the distances at other indexes. The invariant loss  $\mathcal{L}_{in}$  is formulated as follows:
$$
\mathcal {L} _ {i n} = \underset {c, \hat {x} _ {0}, \hat {x} _ {0} ^ {c}} {\mathbb {E}} [ \text {C r o s s E n t r o p y} (d, c) ]. \tag {12}
$$
Variant Loss. In order to minimize the second part,  $\| \hat{z}^{c|c} - z^c \| - \| \hat{z}^{k|c} - z^k \|$ , in Eq. 10. Similarly, we implement the second part in CrossEntropy loss to maximize the distances at indexes  $k$  but minimize the distance at index  $c$ , similar to Eq. 12. Specifically, we calculate the distance scalar of the representations as shown below:
$$
d _ {k} ^ {n} = \| \hat {z} ^ {k} - z ^ {k} \|,
$$
$$
d _ {k} ^ {p} = \left\| \hat {z} ^ {k | c} - z ^ {k} \right\|, \tag {13}
$$
We adopt a cross-entropy loss to achieve the subjective by minimizing the variant loss  $\mathcal{L}_{va}$ , we denote  $[d_1^n, d_2^n, \ldots, d_N^n]$  as  $d^n$  and  $[d_1^p, d_2^p, \ldots, d_N^p]$  as  $d^p$ .
$$
\mathcal {L} _ {v a} = \underset {c, x _ {0}, \hat {x} _ {0}, \hat {x} _ {0} ^ {c}} {\mathbb {E}} [ \text {C r o s s E n t r o p y} (d ^ {n} - d ^ {p}, c) ], \tag {14}
$$
However, sampling  $\hat{x}_0^c$  from  $p_{\theta}(x_{t - 1}|x_t,z^c)$  is not efficient in every training step. Therefore, we follow [9, 49] to use a score-based conditioning trick [41, 40] for fast and approximated sampling. We obtain an approximator  $\epsilon_{\psi}$  of conditional sampling for different options of  $\mathcal{S}$  by substituting the gradient field with decoder  $G_{\psi}$ :