SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\epsilon_ {\psi} \left(x _ {t}, z ^ {\mathcal {S}}, t\right) = \epsilon_ {\theta} \left(x _ {t}, t\right) - \sum_ {c \in \mathcal {S}} \sqrt {1 - \bar {\alpha} _ {t}} G _ {\psi} \left(x _ {t}, z ^ {c}, t\right). \tag {15}
$$
To improve the training speed, one can achieve the denoised result by computing the posterior expectation following [8] using Tweedie's Formula:
$$
\hat {x} _ {0} ^ {\mathcal {S}} = \mathbb {E} \left[ x _ {0} \mid x _ {t}, S \right] = \frac {x _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \epsilon_ {\psi} \left(x _ {t} , z ^ {\mathcal {S}} , t\right)}{\sqrt {\bar {\alpha} _ {t}}}. \tag {16}
$$
4.4 Total Loss
The degree of condition dependence for generated data varies among different time steps in the diffusion model [46]. For different time steps, we should use different weights for Disentangling Loss. Considering that the difference between the inputs of the encoder can reflect such changes in condition. We thus propose using the MSE distance between the inputs of the encoder as the weight coefficient:
$$
\gamma_ {d} = \lambda \| \hat {x} _ {0} - \hat {x} _ {0} ^ {c} \| ^ {2}, \mathcal {L} _ {a} = \mathcal {L} _ {r} + \gamma_ {d} \left(\mathcal {L} _ {i n} + \mathcal {L} _ {v a}\right). \tag {17}
$$
where  $\lambda$  is a hyper-parameter. We stop the gradient of  $\hat{x}_0$  and  $\hat{x}_0^c$  for the weight coefficient  $\gamma_d$ .
<sup>3</sup>We denote the function torch(nn.CrossEntropyLoss as CrossEntropy.
Table 1: Comparisons of disentanglement on the FactorVAE score and DCI disentanglement metrics (mean ± std, higher is better). DisDiff achieves state-of-the-art performance with a large margin in almost all the cases compared to all baselines, especially on the MPI3D dataset.
Method	Cars3D	Shapes3D	MPI3D
FactorVAE score	DCI	FactorVAE score	DCI	FactorVAE score	DCI
VAE-based:
FactorVAE	0.906 ± 0.052	0.161 ± 0.019	0.840 ± 0.066	0.611 ± 0.082	0.152 ± 0.025	0.240 ± 0.051
β-TCVAE	0.855 ± 0.082	0.140 ± 0.019	0.873 ± 0.074	0.613 ± 0.114	0.179 ± 0.017	0.237 ± 0.056
GAN-based:
InfoGAN-CR	0.411 ± 0.013	0.020 ± 0.011	0.587 ± 0.058	0.478 ± 0.055	0.439 ± 0.061	0.241 ± 0.075
Pre-trained GAN-based:
LD	0.852 ± 0.039	0.216 ± 0.072	0.805 ± 0.064	0.380 ± 0.062	0.391 ± 0.039	0.196 ± 0.038
GS	0.932 ± 0.018	0.209 ± 0.031	0.788 ± 0.091	0.284 ± 0.034	0.465 ± 0.036	0.229 ± 0.042
DisCo	0.855 ± 0.074	0.271 ± 0.037	0.877 ± 0.031	0.708 ± 0.048	0.371 ± 0.030	0.292 ± 0.024
Diffusion-based:
DisDiff-VQ (Ours)	0.976 ± 0.018	0.232 ± 0.019	0.902 ± 0.043	0.723 ± 0.013	0.617 ± 0.070	0.337 ± 0.057
Implementation Details.  $x_0$  can be a sample in an image space or a latent space of images. We take pre-trained DDIM as the DPM (DisDiff-IM) for image diffusion. For latent diffusion, we can take the pre-trained KL-version latent diffusion model (LDM) or VQ-version LDM as DPM (DisDiff-KL and DisDiff-VQ). For details of network  $G_{\theta}$ , we follow Zhang et al. [49] to use the extended Group Normalization [9] by applying the scaling & shifting operation twice. The difference is that we use learn-able position embedding to indicate  $c$ :
$$
A d a G N (h, t, z ^ {c}) = z _ {s} ^ {c} \left(t _ {s} ^ {c} G N (h) + t _ {b} ^ {c}\right) + z _ {b} ^ {c}, \tag {18}
$$
where  $GN$  denotes group normalization, and  $[t_s^c, t_b^c], [z_s^c, z_b^c]$  are obtained from a linear projection:  $z_s^c, z_b^c = \text{linearProj}(z^c), t_s^c, t_b^c = \text{linearProj}([t, p^c])$ .  $p^c$  is the learnable positional embedding.  $h$  is the feature map of Unet. For more details, please refer to Appendix A and B.
Datasets To evaluate disentanglement, we follow Ren et al. [36] to use popular public datasets: Shapes3D [23], a dataset of 3D shapes. MPI3D [12], a 3D dataset recorded in a controlled environment, and Cars3D [35], a dataset of CAD models generated by color renderings. All experiments are conducted on  $64 \times 64$  image resolution, the same as the literature. For real-world datasets, we conduct our experiments on CelebA [29].
Baselines & Metrics We compare the performance with VAE-based and GAN-based baselines. Our experiment is conducted following DisCo. All of the baselines and our method use the same dimension of the representation vectors, which is 32, as referred to in DisCo. Specifically, the VAE-based models include FactorVAE [23] and  $\beta$ -TCVAE [5]. The GAN-based baselines include InfoGANCR [27], GANspace (GS) [14], LatentDiscovery (LD) [43] and DisCo [36]. Considering the
Table 2: Ablation study of DisDiff on image encoder, components, batchsize and token numbers on Shapes3D.
Method	FactorVAE score	DCI
DisDiff-IM	0.783	0.655
DisDiff-KL	0.837	0.660
DisDiff-VQ	0.902	0.723
DisDiff-VQ woLin	0.782	0.538
DisDiff-VQ woLva	0.810	0.620
DisDiff-VQ woLdis	0.653	0.414
wo detach	0.324	0.026
constant weighting	0.679	0.426
loss weighting	0.678	0.465
attention condition	0.824	0.591
wo pos embedding	0.854	0.678
wo orth embedding	0.807	0.610
latent number N=6	0.865	0.654
latent number N=10	0.902	0.723
influence of performance on the random seed. We have ten runs for each method. We use two representative metrics: the FactorVAE score [23] and the DCI [11]. However, since  $\{z^c | c \in \mathcal{C}\}$  are vector-wise representations, we follow Du et al. [10] to perform PCA as post-processing on the representation before evaluation.
Figure 3: The qualitative results on Shapes3D and CelebA. The source (SRC) images provide the representations of the generated image. The target (TRT) image provides the representation for swapping. Other images are generated by swapping the representation of the corresponding factor. The learned factors on Shapes3D are orientation (Oren), wall color (Wall), floor color (Floor), object color (Color), and object shape (Shape). For more visualizations, please refer to Appendix C-H.
We conduct the following experiments to verify the disentanglement ability of the proposed DisDiff model. We regard the learned representation  $z^{c}$  as the disentangled one and use the popular metrics in disentangled representation literature for evaluation. The quantitative comparison results of disentanglement under different metrics are shown in Table 1. The table shows that DisDiff outperforms the baselines, demonstrating the model's superior disentanglement ability. Compared with the VAE-based methods, since these methods suffer from the trade-off between generation and disentanglement [26], DisDiff does not. As for the GAN-based methods, the disentanglement is learned by exploring the latent space of GAN. Therefore, the performance is limited by the latent space of GAN. DisDiff leverages the gradient field of data space to learn disentanglement and does not have such limitations. In addition, DisDiff resolves the disentanglement problem into 1000 sub-problems under different time steps, which reduces the difficulty.
We also incorporate the metric for real-world data into this paper. We have adopted the TAD and FID metrics from [45] to measure the disentanglement and generation capabilities of DisDiff, respectively. As shown in Table 3, DisDiff still outperforms new baselines in comparison. It is worth mentioning that InfoDiffusion [45] has made significant improvements in both disentanglement and generation capabilities on CelebA compared to previous baselines. This has resulted in DisDiff's advantage over the baselines on CelebA being less pronounced than on Shapes3D.
Table 3: Comparisons of disentanglement on Realworld dataset CelebA with TAD and FID metrics (mean ± std). DisDiff still achieves state-of-the-art performance compared to all baselines.
Model	TAD	FID
β-VAE	0.088 ± 0.043	99.8 ± 2.4
InfoVAE	0.000 ± 0.000	77.8 ± 1.6
Diffae	0.155 ± 0.010	22.7 ± 2.1
InfoDiffusion	0.299 ± 0.006	23.6 ± 1.3
DisDiff (ours)	0.3054 ± 0.010	18.249 ± 2.1
The results are taken from Table 2 in [45], except for the results of DisDiff. For TAD, we follow the method used in [45] to compute the metric. We also evaluate the metric for 5-folds. For FID, we follow Wang et al. [45] to compute the metric with five random sample sets of 10,000 images. Specifically, we use the official implementation of [48] to compute TAD, and the official GitHub repository of [15] to compute FID.
Figure 4: The partial condition sampling on Shapes3D and CelebA. The target (TRT) image provides the representation of partial sampling images. Images in each row are generated by imposing a single gradient field of the corresponding factor on the pre-trained DPM. DisDiff samples image condition on only a single factor. The sampled image has a fixed factor, e.g., the images in the third row have the same background color as the target one. The conditioned factors on Shapes3D are orientation (Oren), wall color (Wall), floor color (Floor), object color (Color), and object shape (Shape).
5.3 Qualitative Results
In order to analyze the disentanglement of DisDiff qualitatively, we swap the representation  $z^c$  of two images one by one and sample the image conditioned on the swapped representation. We follow LDM-VQ to sample images in 200 steps. For the popular dataset of disentanglement literature, we take Shapes3D as an example. As Figure 3 shows, DisDiff successfully learned pure factors. Compared with the VAE-based methods, DisDiff has better image quality. Since there are no ground truth factors for the real-world dataset, we take CelebA as an example. As demonstrated in Figure 3, DisDiff also achieves good disentanglement on real-world datasets. Please note that, unlike DisCo [36], DisDiff can reconstruct the input image, which is unavailable for DisCo. Inspired by Kwon et al. [25], we generalize DisDiff to complex large-natural datasets FFHQ and CelebA-HQ. The experiment results are given in Appendix E-H.
In order to analyze the effectiveness of the proposed parts of DisDiff, we design an ablation study from the following five aspects: DPM type, Disentangling Loss, loss weighting, condition type, and latent number. We take Shapes3D as the dataset to conduct these ablation studies.
DPM type The decomposition of the gradient field of the diffusion model derives the disentanglement of DisDiff. Therefore, the diffusion space influences the performance of DisDiff. We take Shapes3D as an example. It is hard for the model to learn shape and scale in image space, but it is much easier in the latent space of the auto-encoder. Therefore, we compare the performance of DisDiff with different diffusion types: image diffusion model, e.g., DDIM (DisDiff-IM), KL-version latent diffusion model (DisDiff-KL), and VQ-version latent diffusion model, e.g., VQ-LDM (DisDiff-VQ). As shown in Table 2, the LDM-version DisDiff outperforms the image-version DisDiff as expected.
Disentangling Loss Disentangling loss comprises two parts: Invariant loss  $\mathcal{L}_{in}$  and Variant loss  $\mathcal{L}_{va}$ . To verify the effectiveness of each part, we train DisDiff-VQ without it.  $\mathcal{L}_{in}$  encourages in-variance of representation not being sampled, which means that the sampled factor will not affect the representation of other factors ( $z^k$ ,  $k \neq c$  of generated  $\hat{x}_0^c$ ). On the other hand,  $\mathcal{L}_{va}$  encourages the representation of the sampled factor ( $z^c$  of generated  $\hat{x}_0^c$ ) to be close to the corresponding one of  $x_0$ . As shown in Table 2, mainly  $\mathcal{L}_{in}$  promotes the disentanglement, and  $\mathcal{L}_{va}$  further constrains the model and improves the performance. Note that the disentangling loss is optimized w.r.t.  $G_\theta$  but not  $E_\theta^c$ . If the loss is optimized on both modules, as shown in Table 2, DisDiff fails to achieve disentanglement. The reason could be that the disentangling loss influenced the training of the encoder and failed to do reconstruction.
Loss weighting As introduced, considering that the condition varies among time steps, we adopt the difference of the encoder as the weight coefficient. In this section, we explore other options to verify its effectiveness. We offer two different weighting types: constant weighting and loss weighting. The
first type is the transitional way of weighting. The second one is to balance the Distangling Loss and diffusion loss. From Table 2, these two types of weighting hurt the performance to a different extent.
Condition type & Latent number DisDiff follows PDAE [49] and Dhariwal and Nichol [9] to adopt AdaGN for injecting the condition. However, there is another option in the literature: cross-attention. As shown in Table 2, cross-attention hurt the performance but not much. The reason may be that the condition is only a single token, which limits the ability of attention layers. We use learnable orthogonal positional embedding to indicate different factors. As shown in Table 2, no matter whether no positional embedding (wo pos embedding) or traditional learnable positional embedding (wo orth embedding) hurt the performance. The reason is that the orthogonal embedding is always different from each other in all training steps. The number of latent is an important hyper-parameter set in advance. As shown in Table 2, if the latent number is greater than the number of factors in the dataset, the latent number only has limited influence on the performance.
5.5 Partial Condition Sampling
As discussed in Section 4.2, DisDiff can partially sample conditions on the part of the factors. Specifically, we can use Eq. 15 to sample image condition on factors set  $S$ . We take Shapes3D as an example when DisDiff sampling images conditioned on the background color is red. We obtain images of the background color red and other factors randomly sampled. Figure 4 shows that DisDiff can condition individual factors on Shapes3D. In addition, DisDiff also has such ability on the real-world dataset (CelebA) in Figure 4. DisDiff is capable of sampling information exclusively to conditions.
Our method is completely unsupervised, and without any guidance, the learned disentangled representations on natural image sets may not be easily interpretable by humans. We can leverage models like CLIP as guidance to improve interpretability. Since DisDiff is a diffusion-based method, compared to the VAE-based and GAN-based, the generation speed is slow, which is also a common problem for DPM-based methods. The potential negative societal impacts are malicious uses.
In this paper, we demonstrate a new task: disentanglement of DPM. By disentangling a DPM into several disentangled gradient fields, we can improve the interpretability of DPM. To solve the task, we build an unsupervised diffusion-based disentanglement framework named DisDiff. DisDiff learns a disentangled representation of the input image in the diffusion process. In addition, DisDiff learns a disentangled gradient field for each factor, which brings the following new properties for disentanglement literature. DisDiff adopted disentangling constraints on all different timesteps, a new inductive bias. Except for image editing, with the disentangled DPM, we can also sample partial conditions on the part of the information by superpositioning the sub-gradient field. Applying DisDiff to more general conditioned DPM is a direction worth exploring for future work. Besides, utilizing the proposed disentangling method to pre-trained conditional DPM makes it more flexible.
We thank all the anonymous reviewers for their constructive and helpful comments, which have significantly improved the quality of the paper. The work was partly supported by the National Natural Science Foundation of China (Grant No. 62088102).
[1] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208-18218, 2022.
[2] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. PAMI, 2013.
[47] Tao Yang, Xuanchi Ren, Yuwang Wang, Wenjun Zeng, and Nanning Zheng. Towards building a group-based unsupervised representation disentanglement framework. In International Conference on Learning Representations, 2021.
[48] Eric Yeats, Frank Liu, David Womble, and Hai Li. Nashae: Disentangling representations through adversarial covariance minimization. In European Conference on Computer Vision, pages 36-51. Springer, 2022.
[49] Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre-trained diffusion probabilistic models. NeurIPS, 2022.
A The Training and Sampling Algorithms
As shown in Algorithm 1, we also present the training procedure of DisDiff. In addition, Algorithms 2 and 3 show the sampling process of DisDiff in the manner of DDPM and DDIM under the condition of  $S$ .
Algorithm 1 Training procedure
Prepare: dataset distribution  $p(x_0)$ , pre-trained DPM  $\epsilon_{\theta}$ , pre-trained VQ-VAE,  $E_{VQ},D_{VQ}$  Initialize: encoder  $E_{\phi}$ , decoder  $G_{\psi}$ . repeat  
 $x_0\sim p(x_0)$ $z_{x0} = E_{VQ}(x_0)$ $t\sim$  Uniform  $(0,1,\dots,T)$ $\epsilon \sim \mathcal{N}(0,I)$ $z_{xt} = \sqrt{\bar{\alpha}_t} z_{x0} + \sqrt{1 - \bar{\alpha}_t}\epsilon$ $c\sim$  Uniform  $(1,2,\dots,N)$ $z = E_{\phi}(x_0)$ $\mathcal{L}_r = \| \epsilon -\epsilon_\theta (z_{xt},t) + \frac{\sqrt{\alpha_t}\sqrt{1 - \bar{\alpha}_t}}{\beta_t}\sigma_t\sum_{i\in \mathcal{C}}G_{\psi}(z_{xt},z^i,t,i)\|$   
Use  $z_{xt},t,E_{\phi},E_{VQ},D_{VQ}$  and  $c$  to calculate  $\mathcal{L}_{in}$  and  $\mathcal{L}_{va}$  by Eq. 12 and Eq. 14. Calculate weight  $\gamma_d$  by Eq. 17  
Update by taking gradient descent step on  $\nabla_{\phi ,\psi}(\mathcal{L}_r + \gamma_d(\mathcal{L}_{in} + \mathcal{L}_{va}))$   
until converged;
Algorithm 2 DDPM Sampling
Prepare: pre-trained DPM  $\epsilon_{\theta}$  , pre-trained VQ-VAE,  $E_{VQ},D_{VQ}$  , the sampling factor set  $S\subseteq C$    
Input: image  $x$ $z = E_{\phi}(x)$ $z_{xT}\sim \mathcal{N}(0,I)$    
for  $t = T$  to 1 do   
if  $t\neq 1$  then   
 $\epsilon \sim \mathcal{N}(0,I)$    
else   
 $\epsilon = 0$    
end if   
 $z_{x(t - 1)} = \frac{1}{\sqrt{\alpha_t}}\left[z_{xt} - \frac{\beta_t}{\sqrt{1 - \alpha_t}}\epsilon_\theta (z_{xt},t)\right] + \sigma_t^2\sum_{i\in S}G_\phi (z_{xt},z^i,t,i) + \sigma_t\epsilon$    
end for   
return  $D_{VQ}(z_{x0})$
Encoder We use the encoder architecture following DisCo [36] for a fair comparison. The details of the encoder are presented in Table 3, which is identical to the encoder of  $\beta$ -TCVAE and FactorVAE in Table 1.
Decoder We use the decoder architecture following PADE [49]. The details of the decoder are presented in Table 4, which is identical to the Unet decoder in the pre-trained DPM. DPM We adopted the VQ-version latent diffusion for image resolution of 64 and 256 as our pre-trained DPM. The details of the DPM are presented in Table 4. During the training of DisDiff, we set the batch size as 64 for all datasets. We always set the learning rate as  $1e - 4$ . We use EMA on all model
Algorithm 3 DDIM Sampling
Prepare: pre-trained DPM  $\epsilon_{\theta}$ , pre-trained VQ-VAE  $E_{VQ}$ ,  $D_{VQ}$ , the sampling factor set  $S \subseteq C$
Input: sample  $x$ , sampling sequence  $\{t_i\}_{i=1}^K$ , and  $t_1 = 0, t_K = T$
$$
z = E _ {\phi} (x)
$$
$z_{xT} \sim \mathcal{N}(0, I)$  or inferred  $z_{xT}$  of inverse DDIM
for  $j = K$  to 2 do
$$
\epsilon_ {\theta} \left(z _ {x t _ {j}}, z ^ {S}, t _ {j}\right) = \epsilon_ {\theta} \left(z _ {x t _ {j}}, t\right) - \sum_ {i \in S} \sqrt {1 - \bar {\alpha} _ {t _ {j}}} G _ {\psi} \left(z _ {x t _ {j}}, z ^ {i}, t _ {j}, i\right)
$$
$$
z _ {x t _ {j - 1}} = \sqrt {\bar {\alpha} _ {t _ {j - 1}}} \left(\frac {z _ {x t _ {j}} - \sqrt {1 - \bar {\alpha} _ {t _ {j}}} \epsilon_ {\theta} (z _ {x t _ {j}} , z ^ {S} , t _ {j})}{\sqrt {\bar {\alpha} _ {t _ {j}}}}\right) + \sqrt {1 - \bar {\alpha} _ {t _ {j - 1}}} \epsilon_ {\theta} (z _ {x t _ {j}}, z ^ {S}, t _ {j})
$$
return  $D_{VQ}(z_{x0})$
parameters with a decay factor of 0.9999. Following DisCo [36], we set the representation vector dimension to 32.
Table 4: Encoder E architecture used in DisDiff.
Conv 7 × 7 × 3 × 64, stride = 1
ReLU
Conv 4 × 4 × 64 × 128, stride = 2
ReLU
Conv 4 × 4 × 128 × 256, stride = 2
ReLU
Conv 4 × 4 × 256 × 256, stride = 2
ReLU
Conv 4 × 4 × 256 × 256, stride = 2
ReLU
FC 4096 × 256
ReLU
FC 256 × 256
ReLU
FC 256 × K
Table 5: Decoder (pre-trained DPM) architecture used in DisDiff.
Parameters	Shapes3D / Cars3D/ MPI3D	FFHQ	CelebA-HQ
Base channels	16	64	64
Channel multipliers	[1,2,4,4]	[1,2,3,4]	[1,2,3,4]
Attention resolutions	[1,2,4]	[2,4,8]	[2,4,8]
Attention heads num	8	8	8
Model channels	64	224	224
Dropout	0.1
Images trained	0.48M/0.28M/1.03M	130M	52M
β scheduler	Linear
Training T	1000
Diffusion loss	MSE with noise prediction ε
C Visualizations on Cars3D.
We also provide the qualitative results on Cars3D in Figure 5. We can observe that on the artificial dataset Cars3D, DisDiff also learns disentangled representation. In addition, we also provide the results of swapping representations that contain no information.
Figure 5: The qualitative results on Cars3D. The source (SRC) images provide the representations of the generated image. The target (TRT) image provides the representations for swapping. The remaining rows of images are generated by swapping the representation of the corresponding factor on Shapes3D. DisDiff learns pure factors by each representation. The learned factors are None, None, azimuth, grey, and car color.
D Visualizations on MPI3D.
We provide the qualitative results on MPI3D in Figure 6. We can observe that even on such a challenging dataset (MPI3D) of disentanglement, DisDiff also learns disentangled representation. In addition, we also provide three rows of images swapping representations that contain no information.
E Orthogonal-constrained DisDiff
We follow DisCo [36] and LD [43] to construct learnable orthogonal directions of the bottleneck layer of Unet. The difference is that we use QR decomposition rather than the orthonormal matrix proposed by LD. Specifically, for a learnable matrix  $A \in \mathbb{R}^{C \times d}$ , where  $d$  is the dimension of direction, we have the following equation:
$$
A = Q R \tag {19}
$$
where  $Q \in \mathbb{R}^{C \times d}$  is an orthogonal matrix, and  $R \in \mathbb{R}^{d \times d}$  is an upper triangular matrix, which is discarded in our method. We use  $Q$  to provide directions in Figure 7. Since the directions are sampled, and we follow DisCo and LD to discard the image reconstruction loss, we use the DDIM inverse ODE to obtain the inferred  $x_{T}$  for image editing. The sampling process is shown in Algorithm 3.
F Qualitative Results on CelebA-HQ
In order to demonstrate the effectiveness of Orthogonal-constrained DisDiff, we train it on dataset CelebA-HQ. We also conduct our experiments with the latent unit number of 10. The results are shown in Figure 9. Since the real-world datasets have many factors that are more than 10, we also conduct our experiment with the latent unit number of 30. As shown in Figure 11, Orthogonal-constrained DisDiff learns consistent factors for different images. There are more factors than Figure 9 shows. Therefore, we conduct another experiment using another random seed with a latent unit number of 10. The results are shown in Figure 10.
Figure 6: The qualitative results on MPI3D. The source (SRC) images provide the representations of the generated image. The target image provides the representations for swapping. The remaining rows of images are generated by swapping the representations of the corresponding factor on MPI3D. DisDiff learns pure factors by each representation. The learned factors are object color (OB color), object shape (OB shape), None, None, None, and background color (BG color).
Figure 7: Illustration of Orthogonal-constrained DisDiff. In this architecture, we use a powerful decoder to replace the original decoder in DisDiff. Since we train the decoder from scratch, the ability to disentangle on large natural datasets is limited. However, we can use the pre-trained Unet decoder as our powerful decoder by feeding a set of learnable orthogonal directions to make it output the corresponding gradient field. These directions and the decoder compose a powerful decoder that takes directions as input and generates the corresponding gradient fields as output.
G CLIP-guided DisDiff