SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To tackle the disentanglement in real-world settings, we propose replacing the encoder in DisDiff with a powerful encoder, which lets us leverage the knowledge of the pre-trained model to facilitate the disentanglement. Specifically, we adopt CLIP as the encoder. Since the CLIP is an image-text pair pre-trained model, the multi-modal information is the pre-trained information we can leverage. Therefore, we use the CLIP text encoder to provide the condition but use the CLIP image encoder to calculate the disentangling loss. Figure 8 shows the obtained new architecture.
H Qualitative Results on FFHQ
We train the CLIP-guided DisDiff on FFHQ to verify its effectiveness. Since the attribute words used in the training of the CLIP-guided DisDiff are unpaired and provided in ASYRP, the learning of these factors is still unsupervised. As discussed in Section 4.2, DisDiff can do partial sampling that is conditioned on the part of the factors. Similarly, as shown in Figure 13, CLIP-guided DisDiff also has such an ability. In addition, as shown in Figure 12, we can also use the trained decoder of CLIP-guided DisDiff to edit the images by using the inverse ODE of DDIM to infer the  $x_{T}$ .
I The Latent Number  $<$  The Number of GT Factors
When the latent number is smaller than the number of factors, the model's performance will significantly decrease, which is in accordance with our intuition. However, the performance of other methods will also decrease under this condition. We conducted experiments using FactorVAE and  $\beta$ -TCVAE to demonstrate it. We trained our model, FactorVAE and  $\beta$ -TCVAE on Shapes3D (The number of factors is 6) with a latent number of 3, and the hyperparameters and settings in their original paper were adopted for the latter two models accordingly. As shown in Table 6, our model showed a lower degree of disentanglement reduction than these two models and was significantly lower than  $\beta$ -TCVAE.
Table 6: The disentanglement performance when the latent number is 3 (mean  $\pm$  std,higher is better).
Model	DCI score	FactorVAE score
β-TCVAE	0.098 ± 0.043	0.383 ± 0.0444
FactorVAE	0.3113 ± 0.014	0.4982 ± 0.0049
DisDiff (ours)	0.3279 ± 0.023	0.4986 ± 0.0054
J Quantitative Results on More Metrics
We experiment with MIG and MED metrics on the Shapes3D dataset, two classifier-free metrics. For MIG, we utilized the implementation provided in disentanglement_lib<sup>4</sup>, while for MED, we used the official implementation available at Github<sup>5</sup>. The experimental results in Table 7 indicate that our proposed method still has significant advantages, especially in MED-topk and MIG.
<sup>4</sup>https://github.com/google-research/disentanglement_lib
<sup>5</sup>https://github.com/noahcao/disentanglement_lib_med
Table 7: Comparisons of disentanglement on the additional metrics: MED, MED-topk, and MIG (mean ± std, higher is better). Except for DisCo and DisDiff, other results (MED and MED-topk) are taken from the MED [4]. The missing value is due to the need for publicly released pre-trained models.
Model	MED	MED-topk	MIG
β-VAE	52.5 ± 9.4	19.2 ± 1.4	-
β-TCVAE	53.2 ± 4.9	25.0 ± 0.5	0.406 ± 0.175
FactorVAE	55.9 ± 8.0	8.2 ± 4.0	0.434 ± 0.143
DIP-VAE-I	43.5 ± 3.7	16.2 ± 0.9	-
DIP-VAE-II	52.6 ± 5.2	-	-
AnnealedVAE	56.1 ± 1.5	-	-
MoCo	-	18.1 ± 0.6	-
MoCov2	-	13.6 ± 1.7	-
BarlowTwins	-	20.0 ± 0.3	-
SimSiam	-	30.0 ± 2.0	-
BYOL	-	19.7 ± 1.3	-
InfoGAN-CR	-	-	0.297 ± 0.124
LD	-	-	0.168 ± 0.056
DS	-	-	0.356 ± 0.090
DisCo	38.0 ± 2.7	54.8 ± 2.1	0.512 ± 0.068
DisDiff (Ours)	59.5 ± 1.1	64.5 ± 1.6	0.5671 ± 0.033
K Proof of Proposition 1
**Background:** Our encoder transforms the data  $x_0$  (hereafter denoted as  $x$ , the encoder keeps fixed for disentangling loss) into two distinct representations  $z^c$  and  $z^k$ . We assume that the encoding distribution follows a Gaussian distribution  $p(z^c|x)$ , with a mean of  $E^c(x)$ . However, the distribution  $p(x|z^c)$  is intractable. We can employ conditional sampling of our DPM  $q_\theta(x|z^c)$  to approximate the sampling from the distribution  $p(x|z^c)$ . Given that we optimize the disentangling loss at each training step, we implement Eq. 15 and Eq. 16 in the main paper as an efficient alternative to sample from  $q_\theta(x|z^c)$ . We further assume that the distribution used to approximate  $p(z^k|z^c)$  follows a Gaussian distribution  $q_\theta(z^k|z^c)$ , with a mean of  $z^{k|c} = E^k(D(z^c))$ , where  $D$  represents the operations outlined in Eq. 15 and Eq. 16. We represent the pretrained unconditional DPM using the notation  $q_\theta(x) \approx p(x)$ .
Proposition 2. For a PGM:  $z^c \to x \to z^k$ , we can minimize the upper bound (estimator) for the mutual information between  $z^c$  and  $z^k$  (where  $k \neq c$ ) by the following objective:
$$
\min  _ {k, c, x _ {0}, \hat {x} _ {0} ^ {c}} \| \hat {z} ^ {k | c} - \hat {z} ^ {k} \| - \| \hat {z} ^ {k | c} - z ^ {k} \| + \| \hat {z} ^ {c | c} - z ^ {c} \| \tag {20}
$$
Considering the given context, we can present the PGM related to  $q$  as follows:  $z^{c} \to x \to z^{k}$ . The first transition is implemented by DPM represented by the distribution  $q_{\theta}$ . Subsequently, the second transition is implemented by the encoder, denoted by the distribution  $p$ .
Proposition 1.1 The estimator below is an upper bound of the mutual information  $I(z^{c},z^{k})$ . For clarity, we use expectation notation with explicit distribution.
$$
I _ {e s t} = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right]. \tag {21}
$$
Equality is attained if and only if  $z^k$  and  $z^c$  are independent.
Proof:  $I_{est} - I(z^{c},z^{k})$  can be formulated as follows:
$$
\begin{array}{l} \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) - \log p \left(z ^ {k}\right) \right] \\ = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log p \left(z ^ {k}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right] \tag {22} \\ = \mathbb {E} _ {p (z ^ {k})} [ \log p (z ^ {k}) - \mathbb {E} _ {p (z ^ {c})} [ \log p (z ^ {k} | z ^ {c}) ] ]. \\ \end{array}
$$
Using Jensen's Inequality, we obtain  $\log p(z^k) = \log (\mathbb{E}_{p(z^c)}[p(z^k |z^c)])\geqslant \mathbb{E}_{p(z^c)}[\log p(z^k |z^c)]$  Therefore, the following holds:
$$
I _ {e s t} - I \left(z ^ {c}, z ^ {k}\right) = \mathbb {E} _ {p \left(z ^ {k}\right)} \left[ \log p \left(z ^ {k}\right) - \mathbb {E} _ {p \left(z ^ {c}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) \right] \right] \geqslant 0. \tag {23}
$$
Since the distribution  $p(z^k | z^c)$  is intractable, we employ  $q_{\theta}(z^{k}|z^{c})$  to approximate  $p(z^{k}|z^{c})$ .
$$
I _ {\theta} = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) \right]. \tag {24}
$$
Proposition 1.2: The new estimator can be expressed as follows:
$$
I _ {\theta} = - \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \| z ^ {k} - \hat {z} ^ {k | c} \| + \mathbb {E} _ {p \left(z ^ {c}\right)} \mathbb {E} _ {q _ {\theta} (\hat {x})} \mathbb {E} _ {p \left(\hat {z} ^ {k} | \hat {x}\right)} \| \hat {z} ^ {k} - \hat {z} ^ {k | c} \| + C _ {1}. \tag {25}
$$
The first term corresponds to  $d_k^p$  in Eq. 13 of the main paper, while the second term corresponds to Eq. 11 in the main paper.
Considering the first term, we obtain:
$$
\log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) = \log \exp \left(- \| z ^ {k} - E ^ {k} \left(D \left(z ^ {c}\right)\right) \|\right) + C _ {2} = - \| z ^ {k} - \hat {z} ^ {k | c} \| + C _ {2}. \tag {26}
$$
For the second term, we obtain:
$$
\begin{array}{l} \mathbb {E} _ {p \left(z ^ {k}\right)} \left[ \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) \right] = \int \int p (x, z ^ {k}) \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) d z ^ {k} d x \\ = \int p (x) \int p \left(z ^ {k} | x\right) \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) d z ^ {k} d x \\ \approx \int q _ {\theta} (\hat {x}) \int p \left(\hat {z} ^ {k} | \hat {x}\right) \log q _ {\theta} \left(\hat {z} ^ {k} \mid z ^ {c}\right) d \hat {z} ^ {k} d \hat {x} \tag {27} \\ = \mathbb {E} _ {q _ {\theta} (\hat {x})} \mathbb {E} _ {p (\hat {z} ^ {k} | \hat {x})} [ \log q _ {\theta} (\hat {z} ^ {k} | z ^ {c}) ] \\ = \mathbb {E} _ {q _ {\theta} (\hat {x})} \mathbb {E} _ {p (\hat {z} ^ {k} | \hat {x})} \| \hat {z} ^ {k} - E ^ {k} (D (z ^ {c})) \| + C _ {3} \\ = \mathbb {E} _ {q _ {\theta} (\hat {x})} \mathbb {E} _ {p (\hat {z} ^ {k} | \hat {x})} \| \hat {z} ^ {k} - \hat {z} ^ {k | c} \| + C _ {3}. \\ \end{array}
$$
Proposition 1.3: If the following condition is fulfilled, the new estimator  $I_{\theta}$  can serve an upper bound for the mutual information  $I(z^{c},z^{k})$ , i.e.,  $I_{\theta}\geqslant I(z^{c},z^{k})$ . The equality holds when  $z^k$  and  $z^c$  are independent. We denote  $q_{\theta}(z^{k}|z^{c})p(z^{c})$  as  $q_{\theta}(z^{k},z^{c})$ .
$$
D _ {K L} \left(p \left(z ^ {k}, z ^ {c}\right) \| q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right) \leqslant D _ {K L} \left(p \left(z ^ {k}\right) p \left(z ^ {c}\right) \| q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right). \tag {28}
$$
$I_{\theta} - I(z^{c},z^{k})$  can be formulated as follows:
$$
\begin{array}{l} \mathbb {E} _ {p (z ^ {c}, z ^ {k})} \bigl [ \log q _ {\theta} (z ^ {k} | z ^ {c}) \bigr ] - \mathbb {E} _ {p (z ^ {c}) p (z ^ {k})} \bigl [ \log q _ {\theta} (z ^ {k} | z ^ {c}) \bigr ] - \mathbb {E} _ {p (z ^ {c}, z ^ {k})} \bigl [ \log p (z ^ {k} | z ^ {c}) - \log p (z ^ {k}) \bigr ] \\ = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) - \log p \left(z ^ {k} \mid z ^ {c}\right) \right] + \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ - \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) + \log p \left(z ^ {k}\right) \right] \\ = \mathbb {E} _ {p \left(z ^ {c}\right) p \left(z ^ {k}\right)} \left[ - \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) + \log p \left(z ^ {k}\right) \right] - \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log p \left(z ^ {k} \mid z ^ {c}\right) - \log q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right) \right] \tag {29} \\ = \mathbb {E} _ {p (z ^ {c}) p (z ^ {k})} \left[ \log \frac {p (z ^ {k}) p (z ^ {c})}{q _ {\theta} (z ^ {k} | z ^ {c}) p (z ^ {c})} \right] - \mathbb {E} _ {p (z ^ {c}, z ^ {k})} \left[ \log \frac {p (z ^ {k} | z ^ {c}) p (z ^ {c})}{q _ {\theta} (z ^ {k} | z ^ {c}) p (z ^ {c})} \right] \\ = D _ {K L} \left(p \left(z ^ {c}\right) p \left(z ^ {k}\right) \mid \mid q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right) - D _ {K L} \left(p \left(z ^ {c}, z ^ {k}\right) \mid \mid q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right). \\ \end{array}
$$
If  $D_{KL}(p(z^k,z^c)||q_\theta (z^k,z^c))\leqslant D_{KL}(p(z^k)p(z^c)||q_\theta (z^k,z^c))$  , we obtain:
$$
I _ {\theta} \geqslant I \left(z ^ {c}, z ^ {k}\right). \tag {30}
$$
Since the condition is not yet applicable, we will continue exploring alternative conditions.
Proposition 1.4: If  $D_{KL}(p(z^k |z^c)||q_\theta (z^k |z^c))\leqslant \epsilon$  , then, either of the following conditions holds:
$$
I _ {\theta} \geqslant I \left(z ^ {c}, z ^ {k}\right) \text {o r} \left| I _ {\theta} - I \left(z ^ {c}, z ^ {k}\right) \right| \leqslant \epsilon .
$$
Proof: If  $D_{KL}(p(z^k, z^c)) \| q_\theta(z^k, z^c)) \leqslant D_{KL}(p(z^k)p(z^c)) \| q_\theta(z^k, z^c))$ , then
$$
I _ {\theta} \geqslant I \left(z ^ {c}, z ^ {k}\right), \tag {31}
$$
$$
\begin{array}{l} D _ {K L} (p (z ^ {k}) p (z ^ {c}) | | q _ {\theta} (z ^ {k}, z ^ {c})) <   D _ {K L} (p (z ^ {k}, z ^ {c}) | | q _ {\theta} (z ^ {k}, z ^ {c})) \\ = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log \frac {p \left(z ^ {k} \mid z ^ {c}\right)}{q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right)} \right] \tag {32} \\ = D _ {K L} \left(p \left(z ^ {k} \mid z ^ {c}\right) | | q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right)\right) \\ \leqslant \epsilon . \\ \end{array}
$$
$$
\left| I _ {\theta} - I \left(z ^ {c}, z ^ {k}\right) \right| <   \max  \left(D _ {K L} \left(p \left(z ^ {k}, z ^ {c}\right) \mid q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right), D _ {K L} \left(p \left(z ^ {k}\right) p \left(z ^ {c}\right) \mid q _ {\theta} \left(z ^ {k}, z ^ {c}\right)\right)\right) \leqslant \epsilon . \tag {33}
$$
Proposition 1.5: Consider a PGM  $z^c \to x \to z^k$ ; if  $D_{KL}(q_{\theta}(x|z^c)|p(x|z^c)) = 0$ , it follows that  $D_{KL}(p(z^k |z^c)||q_\theta (z^k |z^c)) = 0$ .
Proof: if  $D_{KL}(q_{\theta}(x|z^c)|p(x|z^c)) = 0$ , we have  $q_{\theta}(x|z^{c}) = p(x|z^{c})$
$$
\begin{array}{l} D _ {K L} \left(p \left(z ^ {k} \mid z ^ {c}\right) \mid \mid q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right)\right) = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log \frac {p \left(z ^ {k} \mid z ^ {c}\right)}{q _ {\theta} \left(z ^ {k} \mid z ^ {c}\right)} \right] \\ = \mathbb {E} _ {p \left(z ^ {c}, z ^ {k}\right)} \left[ \log \frac {\int p \left(x \mid z ^ {c}\right) p \left(z ^ {k} \mid x\right) d x}{\int q _ {\theta} \left(x \mid z ^ {c}\right) p \left(z ^ {k} \mid x\right) d x} \right] \tag {34} \\ = \mathbb {E} _ {p (z ^ {c}, z ^ {k})} [ \log 1 ] = 0. \\ \end{array}
$$
Ideally, by minimizing  $D_{KL}(q_{\theta}(x|z^{c})|p(x|z^{c}))$  to 0, we can establish  $I_{\theta}$  either as an upper bound or as an estimator of  $I(z^{c},z^{k})$ .
Proposition 1.6: we can minimize  $D_{KL}(q_{\theta}(x|z^{c})|p(x|z^{c}))$  by minimizing the following objective:
$$
\mathbb {E} _ {p \left(z ^ {c}\right)} \mathbb {E} _ {q _ {\theta} \left(x \mid z ^ {c}\right)} \| z ^ {c} - \hat {z} ^ {c \mid c} \| + C _ {4}. \tag {35}
$$
Note that the equation above exactly corresponds to  $d_c^p$  in Eq. 13, and  $d_k^n$  is a constant.
$$
D _ {K L} \left(q _ {\theta} \left(x \mid z ^ {c}\right) \mid p \left(x \mid z ^ {c}\right)\right) = \int p \left(z ^ {c}\right) \int q _ {\theta} \left(x \mid z ^ {c}\right) \log \frac {q _ {\theta} \left(x \mid z ^ {c}\right)}{p \left(x \mid z ^ {c}\right)} d x d z ^ {c}. \tag {36}
$$
We focus on the following term:
$$
\int q _ {\theta} (x \mid z ^ {c}) \log \frac {q _ {\theta} (x \mid z ^ {c})}{p (x \mid z ^ {c})} d x = \int q _ {\theta} (x \mid z ^ {c}) [ \log q _ {\theta} (x \mid z ^ {c}) - \log p (x \mid z ^ {c}) ] d x. \tag {37}
$$
Applying the Bayes rule, we obtain:
$$
\begin{array}{l} = \int q _ {\theta} (x | z ^ {c}) [ \log q _ {\theta} (x | z ^ {c}) - \log \frac {p (z ^ {c} | x) p (x)}{p (z ^ {c})} ] d x \\ = \int q _ {\theta} (x | z ^ {c}) \left[ \log \frac {q _ {\theta} (x | z ^ {c})}{p (x)} - \log p (z ^ {c} | x) + \log p (z ^ {c}) \right] d x \\ = D _ {K L} \left(q _ {\theta} \left(x \mid z ^ {c}\right) \mid p (x)\right) - \mathbb {E} _ {q _ {\theta} \left(x \mid z ^ {c}\right)} \left[ \log p \left(z ^ {c} \mid x\right) \right] + \log p \left(z ^ {c}\right) \tag {38} \\ \leqslant - \mathbb {E} _ {q _ {\theta} (x | z ^ {c})} [ \log p (z ^ {c} | x) ] + \log p (z ^ {c}) \\ = - \mathbb {E} _ {q _ {\theta} (x | z ^ {c})} [ \log \exp (- | z ^ {c} - E ^ {c} (x) |) ] + \log p (z ^ {c}) + C _ {5} \\ = \mathbb {E} _ {q _ {\theta} (x | z ^ {c})} \| z ^ {c} - \hat {z} ^ {c | c} \| + \log p (z ^ {c}) + C _ {5}. \\ \end{array}
$$
We can minimize  $\mathbb{E}_{p(z^c)}\mathbb{E}_{q_\theta (x|z^c)}\| z^c -\hat{z}^{c|c}\|$  instead.
Note that the loss usually cannot be optimized to a global minimum in a practical setting, but how the gap influences  $I_{\theta}$  is beyond the scope of this paper. We leave it as future work. Figure 4 in the main paper shows that our model has successfully fitted the distribution of  $p(x|z^c)$  using  $q_{\theta}(x|z^{c})$ . Therefore, we can understand that the model's loss can be regarded as  $I_{est}$ .
Table 8: Ablation study of DisDiff on image encoder, components, batchsize and token numbers on Shapes3D.
Method	SSIM(↑)	LPIPS(↓)	MSE(↓)	DCI(↑)	Factor VAE(↑)
PDAE	0.9830	0.0033	0.0005	0.3702	0.6514
DiffAE	0.9898	0.0015	8.1983e-05	0.0653	0.1744
DisDiff (ours)	0.9484	0.0006	9.9293e-05	0.723	0.902
L Comparison with Diffusion-based Autoencoders
We include a comparison with more recent diffusion baselines, such as PADE [49] and DiffAE [33]. The results are shown in Table 8.
The relatively poor disentanglement performance of the two baselines can be attributed to the fact that neither of these models was specifically designed for disentangled representation learning, and thus, they generally do not possess such capabilities. DiffAE primarily aims to transform the diffusion model into an autoencoder structure, while PDAE seeks to learn an autoencoder for a pre-trained diffusion model. Unsupervised disentangled representation learning is beyond their scope.
Traditional disentangled representation learning faces a trade-off between generation quality and disentanglement ability (e.g., betaVAE [16], betaTCVAE [5], FactorVAE [23]). However, fortunately, disentanglement in our model does not lead to a significant drop in generation quality; in fact, some metrics even show improvement. The reason for this, as described in DisCo [36], is that using a well-pre-trained generative model helps avoid this trade-off.
Figure 9: The qualitative results of orthogonal constraint on CelebA-HQ (seed 0). The source (SRC) row provides the images of the generated image. The remaining rows of images are generated by shifting the representation along the learned direction of the corresponding factor on CelebA-HQ. DisDiff learns the direction of pure factors unsupervised. The learned factors are listed in the leftmost column. The images are sampled by using the inferred  $x_{T}$ .
Figure 10: The qualitative results of orthogonal constraint on CelebA-HQ (seed 1). The source (SRC) row provides the images of the generated image. The remaining rows of images are generated by shifting the representation along the learned direction of the corresponding factor on CelebA-HQ. DisDiff learns the direction of pure factors unsupervised. The learned factors are listed in the leftmost column. The images are sampled by using the inferred  $x_{T}$ .
Figure 11: The qualitative results of orthogonal constraint on CelebA-HQ (latent number  $= 30$ ). The source (SRC) row provides the images of the generated image. The remaining rows of images are generated by shifting the representation along the learned direction of the corresponding factor on CelebA-HQ. DisDiff learns the direction of pure factors unsupervised. The learned factors are listed in the leftmost column. The images are sampled by using the inferred  $x_{T}$ .
Figure 12: The qualitative results of CLIP-guided DisDiff on FFHQ. The source (SRC) row provides the images of the generated image. The remaining rows of images are generated by shifting the representation along the learned direction of the corresponding factor on FFHQ. DisDiff learns the direction of pure factors unsupervised. The learned factors are listed in the leftmost column. The images are sampled by using the inferred  $x_{T}$ .
Figure 13: The partial condition sampling results of CLIP-guided DisDiff on FFHQ. The source (SRC) row provides the images of the generated image. The remaining rows of images are generated by shifting the representation along the learned direction of the corresponding factor on FFHQ. DisDiff learns the direction of pure factors unsupervised. The learned factors are listed in the leftmost column. The images are sampled by using the inferred  $x_{T}$ .