Exploring Diffusion Time-steps for Unsupervised Representation Learning
Authors: Zhongqi Yue
Jiankun Wang
Qianru Sun
Lei Ji
Eric I-Chao Chang
Hanwang Zhang
Venue: ICLR 2024 poster (2024)
Keywords: unsupervised representation learning
diffusion model
representation disentanglement
counterfactual generation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Leverage the connection between diffusion time-steps and hidden generative attributes to learn a disentangled representation.

Abstract: Representation learning is all about discovering the hidden modular attributes that generate the data faithfully. We explore the potential of Denoising Diffusion Probabilistic Model (DM) in unsupervised learning of the modular attributes. We build a theoretical framework that connects the diffusion time-steps and the hidden attributes, which serves as an effective inductive bias for unsupervised learning. Specifically, the forward diffusion process incrementally adds Gaussian noise to samples at each time-step, which essentially collapses different samples into similar ones by losing attributes, e.g., fine-grained attributes such as texture are lost with less noise added (i.e., early time-steps), while coarse-grained ones such as shape are lost by adding more noise (i.e., late time-steps). To disentangle the modular attributes, at each time-step t, we learn a t-specific feature to compensate for the newly lost attribute, and the set of all {1,...,t}-specific features, corresponding to the cumulative set of lost attributes, are trained to make up for the reconstruction error of a pre-trained DM at time-step t. On CelebA, FFHQ, and Bedroom datasets, the learned feature significantly improves attribute classification and enables faithful counterfactual generation, e.g., interpolating only one specified attribute between two images, validating the disentanglement quality. Codes are in https://github.com/yue-zhongqi/diti.

Published as a conference paper at ICLR 2024
EXPLORING DIFFUSION TIME-STEPS FOR UNSUPER- VISED REPRESENTATION LEARNING
Zhongqi Yue', Jiankun Wang', Qianru Sun', Lei Ji', Eric I-Chao Chang', Hanwang Zhang "Nayyang Technological University, "Singapore Management University, "Microsoft Research Asia, "Sky.okk Al zhongqi.. wyweuuuuteeeeeee eed..eeduuss... sg, jiankun001 @e atu.edu..s... qi gianrusun@smu. edu. sg,
lei ji @mi cwiicrooo..comm eric. fric.i..o wwa.ggoutttgttttcacccccomoooo loc. cok,com, hanwangzhangllntu.ed edu.e.
ABSTRACT
Representation learning is all about discovering the hidden modular attributes that generate the data faithfully. We explore the potential of Denoising Diffusion Prob- abilistic Model (DM) in unsupervised learning of the modular attributes. We build a theoretical framework that connects the diffusion time-steps and the hidden at- tributes, which serves Specifically, the forward diffusion process incrementally adds Gaussian noise to samples each time-step, which essentially collapses different samples into sim- ilar ones by losing attributes, e.g. fine-grained attributes such as texture are lost with less noise added (i.e., early time-steps), while coarse-grained ones such shape are lost by adding more noise (i.e.. late time-steps). To disentangle the modular attributes, at each time-step we learn t-specific feature compensate for the newly lost attribute, and the set of all (1. ....... ?)-specific features, corre- sponding to the cumulative set of lost attributes, are trained to make up for the reconstruction error of a pre-trained DM at time-step t. On CelebA, FFHQ, and Bedroom datasets, the learned feature significantly improves attribute classifica- tion and enables faithful counterfactual generation, e.g. interpolating only one specified attribute between two images, validating the disentanglement quality. Codes are in httpthtpp///gieee wwt.//e-ehhoogggg//eeee wtt.s/ihoonggiii.
effective inductive bias for unsupervised learning.
INTRODUCTION
A good feature representation should faithfully capture the underlying generative attributes in compact and modular vector space Bengio et al. (2013), enabling not only sample inference (e.g., image classification) but also counterfactual generation Besserve al. (2020) (e.g., image synthesis of unseen attribute combinations). Over the past decade, discriminative training has been the feature learning mainstream with exceptional performance in inference tasks He al. (2016) (2020). How- ever, it hardly achieves faithful generation due to the deliberate discard of certain attributes, e.g., class-irrelevant ones in supervised learning or augmentation-reelted ones in contrastive learning.
On the other hand, generative Denoising Diffusion Probabilistic Model (DM) Sohi-Dickstein et al. Song al. (202I) can retain all the underlying attributes for faithful generation Dhariwal & Nichol 2021). or even extrapolate novel attribute combinations by textual control Rombach et al. 2022 (e.g. "Teddy bear skating in Time Square"), outperforming other generative models like Kingma & Welling (2014) and G Creswell et al. (2018). This implies that DM effectively captures the modularity hidden attributes Yue al. (2021). However, as DM's formulation has no explicit encoders that transform samples into feature vectors, the conventional encoder-decoder feature learning paradigm via reconstruction Higgins al. (017)) is no longer applicable.
Can we extract DM's knowledge about the modular attributes as a compact feature vector? To an- swer the question, we start by revealing a natural connection between the diffusion time-steps and the hidden attributes. In the forward diffusion process, given an initial sample Xo. small amount of Gaussian noise (0, I) is incrementally added to the sample across T time-steps, resulting in a sequence of noisy samples X In particular, each x, adheres its noisy sample distribution q(x, xx)), whose mean and covariance matrix o²I is denoted as a dot and a circle with radius o in Figure Ta respectively. As f increases, q(x:(xo) progressively collapses towards pure noise by

---

Published as a conference paper at ICLR 2024
12
Xy
X
X
X
others lost
gender lost
expression lost
initial samples
from
newly
cumulatively lost
expression
gender
others
(a)
Figure I: (a) Illustration of attribute loss as time-step increases in the forward diffusion process. The two axes depict a two-dimensional sample space. (b) DM reconstructed Xo. denoted as Xo. from randomly sampled x, various DM pre-trained on CelebA, from where Xg drawn.
reducing mean (i.e., dots moving closer to the origin), and increasing (i.e., larger radii), which en- larges the overlapping area (OVL) of the probability density function of noisy sample distributions. In particular, theoretically show in Section 4.1 that OVL is correlated with DM's ambiguous reconstruction of different samples, e.g., due to the large overlap between the red circles at t1. noisy samples drawn from the two distributions are reconstructed ambiguously the attribute "expression" [ distinguishes them. As f increases, more noisy sample distributions exhibit significant overlaps, inducing additional attribute loss, egg. further losing "gender" at t2 that confuses the identities of A/B and C/D. Eventually, with a large enough T, XT becomes pure noise and attributes are lost- the reconstruction can be any sample.
either or B, losing
This intrinsic connection between diffusion time-steps and modular attributes can act an effective inductive bias for unsupervised learning. Specifically, DM training can viewed as learning to reconstruct Xo given at each (Section 3.2). However, due to the aforementioned attribute loss, perfect DM reconstruction impossible, e.g.. in Figure 1b reconstructions exhibit variations in the cumulatively lost attributes. Hence, by contraposition, if feature enables perfect reconstruction, it must capture the cumulatively lost attributes. This motivates us to learn an expanding set of features supplement the expanding lost attributes as I increases. Specifically, z = f(xo) using an encoder f. Then we partition z into T disjoint subsets {zi f==1 At each t, optimize f that {z, k= compensates for a pre-trained DM's reconstruction error. Intuitively O with no lost attribute; if {ziM=z captures the cumulatively lost attributes till t, Z must capture the newly lost attribute to enable perfect reconstruction + 1; until T 1. |T
first map Xo to feature
by induction, starting from
U ZT learns all hidden attributes in compact and modular vector space.
We term our approach as DiTi to highlight our leverage of Diffusion Time-step as an inductive bias for feature learning. We summarize the paper structure and our contributions below:
In Section 3.1
we formalize the notion of good feature with a definition of disentangled repre- sentation Higgins et al. (2018) and provide a preliminary introduction to DM in Section 3.2.
In Section 4.1 we build theoretical framework that connects diffusion time-steps and the hidden modular attributes, which motivates simple and practical approach for unsupervised disentangled representation learning, discussed in Section 4.2
Section 5) by extensive experiments on CelebA Liu et al. (2015), FFHQ Karras et al. (2019) and Bedroom Yu et al. (2015), our DiTi feature brings significant improvements in attribute inference accuracy and enables counterfactual generation, verifying its disentanglement quality.
2 RELATED WORKS
DM for Representation Learning. There are three main approaches: 1) DDIM Song et al. (2020) inversion aims find latent X7 feature, by DDIM sampling with the learned DM, can recon- struct the image Xg. However, this process is time-consuming, and the resulting latent is difficult to interpret Hertz al. (2022). 2) Kwon et al. (2022) uses the bottleneck feature of the U-Net at all stmess However, the feature is not compact and difficult for downstream leverage. 3)
Atrribute name is illustrative. Our method no attribute supervision, explicitly names them.
2

---

Published as a conference paper at ICLR 2024
PDAE (entangled)
DITi (disentangled)
Classifier of 40 attributes on CelebA
1000
PDAE 60. 2)
DiTI (Acc. 62.
young
young
DDmansioos
200
5000
male
+
male
0.02
0.06
0.10
0.14
0.18
high cheek
high cheek
Classifier weight value
(a) Examples of counterfactual generation
(b) Histogram of classifier weight value
Figure 2: (a) Counterfactual generations on CelebA by manipulating out 512 feature dimensions (iee,
simulating the edit of a single z;). A disentangled representation enables editing a single attribute (e.g. gender) without affecting others (e.g., lighting) and promotes faithful extrapolation (e.g., no artifacts). (b) Histogram of the classifier weight value. More dimensions of DiTi weights closed and 0 (explanations in the text).
The closest to our works are Preechakul et al. 2022): ; Zhang et al. 20222) However, they learn
time-step-indedt step-independent z to supplement DM reconstruction error. Without explicit design enforce modularity, the learned feature entangles all the attributes. At time-step f. it leaks information about attributes lost at later time-step (i.e., + .,..., 7). Hence, instead of learning the lost attribute 2, the reconstruction may digress to use the spurious correlation between leaked information and lost attribute. In Section we validate this deficiency by observing failed counterfactual generation and lower attribute prediction accuracy.
Disentangled Representation separates the modular hidden attributes that generate data, which can be formally defined from group troup-theoretical view Higgins al. (2018). Our work focuses the unsupervised learning setting. Existing methods typically learn a VAE Burgess et al. (2017) GAN Jeon al. (2021) with information bottleneck. Yet they lack explicit inductive bias that necessary for unsupervised disentanglement Locatello et al. (2019). Our work reveals that the connection between hidden attributes and diffusion time-steps is effective inductive bias towards disentangiement (Section 4.1)
3 PRELIMINARIES
3.1 DISENTANGLED REPRESENTATION
A good feature can be formally defined by the classic notion of disentangled representation. We aim to provide preliminary introduction below, and refer interesting readers to HHiggins et al. (2018); Wang et al.] (2021) for more formal details. Each sample erated from hidden attribute vector z E Z in the vector space through an injective mapping $ 2 X' (i.e., different samples generated by different attributes). In particular, 2 can be decomposed into the Cartesian product of N subspaces Z Z x x Zn. where each 2, is called modular attribute, as its value can locally intervened without affecting other modular attributes (e.g., changing "cxpression" without affecting "gender" or "age"). A disentangled representation mapping f X Z that inverses 4 i.e., given each image x, f(x) recovers the modular attributes feature vector z = [Z]..... ZNI, which possesses desired properties of a good feature in the following common views:
X' in the image space X' gen-
Counterfactual generation. A counterfactual image of by intervening the i-th modular attribute with 9; (e.g., aging) denoted as 9i x (e.g., an older x). Specifically, 9: x is generated by first mapping to by f. editing z; according to Ii without changing other attributes, and finally feeding the modified to 4 fact, g, x formally known as group action (1994). Figure shows counterfactual images generated by editing three attributes.
Sample inference. A modular feature facilitates robust inference in downstream tasks about any subset of {z, N e.g., linear classifier eliminate task -ikiirreeevant environmental attributes by assigning larger values to the dimensions corresponding to z; and 0 to other dimensions. In Figure by explicitly disentangling the modular attributes (Section our DiTi achieves higher accuracy in
3

---

Published as a conference paper at ICLR 2024
attribute prediction than PDAE Zhang al. (2022) (without disentangling), while more dimensions
of our classifier weight are closed to (corresponding to zj) and 0 (environmental attributes).
3.2 DENOISING DIFFUSION PROBABILISTIC MODEL
Denoising Diffusion Probabilistic Model Ho et al. 2020) (DM) is latent variable model with fixed forward process, and learnable reverse process. Details are given below.
Forward Process. It is fixed to a Markov chain that incrementally adds Gaussian noise to each sample Xg I" time-steps, producing a sequence of noisy samples X1...., ......,,y, Xy latents of the same dimensionality as the original sample Xy- Given Xo and a variance schedule $1. .rr (i.e., how much noise is added each time-step), each x adheres the following noisy sample distribution:
((x/1x0) = (xe: Va,xo, (1 - T), where a₃ == 1 Bb, Cr
II Os
Reverse Process. It corresponds to a learned Gaussian transition po(x; (x)) parameterized by
0, starting at p(xT) := (xx;; 0, I). Each po po(x,_1(xx) computed two steps: 1) Reconstruct
Xo from X2 with (x,, t). where ug is learnable U-Net Ronneberger et al. (2015). 2) Compute
q (xx-1|Xe, uo (x,, t)), which has a closed-form solution given in Appendix. Training is performed
by minimizing the reconstruction error made by us wxxx, t) each time-step t:
LDM =
E 1,N0,e
āt -t 0t
|xo
uol - vI - āge, 1)
where c < (O.II) I) is random noise. Note that we formulate the U-Net to predict xe (equivalent the c-formulation Ho et al., (2020). ). As DM has no explicit feature encoder, we learn one to capture the modular attributes by leveraging the inductive bias in the diffusion time-steps.
METHOD
4.1 THEORY
We reveal that in the forward diffusion process, fine-grained attributes lost at earlier time-step compared to coarse-grained ones, where the granularity is defined by the pixel-level changes when altering attribute. We first formalize the notion of attribute loss, i.e., when DM fails distinguish the noisy samples drawn from two overlapping noisy sample distributions q Xx XX)) and (yy:)yo).
Definition. (Attribute Loss) Given a e-parameterized DM optimized by Eq. we say that attribute Z, is lost with degree at t when Ex,ex [Err(xo, Yo = 9s Xo, t)] 2 where Err(xo. Yo. ==
1 LE [1 xoll 2 (x(x
11000 yoll)]
(llyo xoll < l15o yo []]]
(yyilyo)
with (.) denoting the indicator function and &n = wo(x;), = uo(y:) denoting the DM recon- structed samples from xt, Yt. respectively.
Intuitively, Err(xo. Vo. t) measures attribute loss degree by how likely a DM falsely reconstructs x, drawn from q(xe (xo) closer to Yo instead of Xo (vice versa). Hence, when Xo and yo differ in attribute 2; modified by gi. larger Err(xo, yo. means that the attribute is more likely lost.
Theorem. (Attribute Loss and Time-step) 1) For each 2,, there exists a smallest time-step t (Zz), such that Z, is lost with degree at each E (t(Z.),..., 2) 3( such that t(Z,) > t(2,) whenever ||xo -::0---330 9.-X) | is first-order stochastic dominans over Xoo 9, -Xoll with Xg ~ X uniformly.
Intuitively, the first part of the theorem states that lost attribute will not be regained as time-step t increases, and there is a time-step t( ((2)) when Z; becomes lost (with degree tau) for the first time. The second part states that when ||xo 9: Xp is more likely to take a larger value than |xo- 9j Xn |I. the attribute Z, (i.e., a coarse- coarse-grained attribute) is lost at a larger time-step compared to Z, (i.e., fine-grained attribute). We have the following proof sketch:
4

---

Published as a conference paper at ICLR 2024
DITi
Frozen DM Me
X
X1
X2
XT
Encoder
Decoder & inmon
Z
Z
Zg EZ,
Illustration of our DiTi. We break down Eq. 5 at each time-step. On the right, we show the detailed network design, where Xg denotes the reconstructed Xo by the pre-trained DM.
Figure 3:
1) Increasing induces further attribute loss. We show in Appendix that Err(xo, Vo. OVL (g(x, x Xo), ((Yelyo)) for an optimal DM, where OVL is the overlapping coefficient Inman & Bradley Jr 1989), , i.e., overlapping area of probability density functions (PDFs). In particular,
Err(xo. t) = 1 ( (g(x)))))))
= 1 erf
[Ivā. - xo)1] -0000 2/2(11 - o.)
)
(4)
As cr decreases with an increasing from Eq. [ and the error function erf(-) strictly increasing.
Err(xo. Vo. t) is strictly increasing in given any Xo. Vo. Hence Ex.eX [Er (No.Yo = 9i Xo, xo, 2 Ex,Ex [Err(xo. Yo 9: Xx, t(Z,)) for every t 2 t t(z),, which completes the proof.
2) Coarse-graimee attributes are lost at larger t. Given that erf(-) is strictly increasing and ||x0
gx-Xpll || is first-order stochastic dominant over | gj Xg we have Expex x [Err(Xo, Di Xo, xo,t)] > Ex,ex [Err(Xp, 9j Xu, t)] at every time-step & using Eq. 4 Hence t t(2,) > t(2,) under any vari- ance schedule {B:)T=1 such that Z, is not lost at f(2,). completing the proof. Note that in practice, DM leverages a large ' with small {3, 1 This ensures that Err(xo, Yo, t) slows increases with t according to Eq. Hence empirically, this theorem tends hold.
4.2 PROPOSED APPROACH
The previous theoretical analysis leads to two interesting groundings: 1) Attribute loss is linked with the eиror of reconstructing Xo from the noisy x,. 2) Increasing time-step f in forward diffusion pro- cess causes expanding cumulative loss of attributes, with fine-grained attributes lost at smaller t and caars-graaieed ones lost at a larger t. They serve as an effective inductive bias for unsupervised learning: making up for the reconstruction error can retrieve lost attributes, 2) an expanding set of features captures the expanding cumulatively lost attributes. This motivates our approach below.
As illustrated in Figure our model includes frozen DM U-Net ue pre-trained D using Eq. until convergence, a trainable encoder / (trained to become a disentangled representation) that maps each image (without noise) z back to image space given t. In training, given z fxxo), we partition it into T disjoint subsets {z,) 1 At each time-step we construct new feature z, = [zi...., ., Z,, O, ..... O). which is only about {z, Y by masking Zı+1,..., Zr as O. Our training loss requires 2, to compensate for the reconstruction eтror made by ug at each time-step t, given by:
d-dimensional feature z E Rd. and trainable decoder g that maps
C = E t,Xo,c
[
Vullaa,n + - Vi-āāe,t) + (2.((Z.
(5)
where de, Wt are time-step weight and compensate strength. We follow PDAE Zhang et al. (2022)
to use fixed values computed from the variance schedule {3,) (details in Appendix).
To see the effect of Eq. 5 consider t = 1, only Z1 is used to compensate for reconstruction error, hence Z1 effectively captures a (fine-grained) grained) modular attribute Z, with t t(2,) 1 (lost at t 1) (such Z, always exists for some threshold in our definition). At t = 2. the cumulatively lost attributes correspond to Z, and 2,, where t(2,) 2. As z₁ already captures the attribute Z,,
5

---

Published as a conference paper at ICLR 2024
Zz is encouraged to further learn the attribute 2, to minimize the reconstruction error at t 2. Eventually with a large enough T. Xy becomes pure noise and all attributes are lost. By the above induction, {z, ) captures all the hidden attributes as modular feature. In contrast, the most related work PDAR differs from Eq. 5 by replacing g(z:) with g This means that PDAF uses the full-dimensional feature compensate the reconstruction error all time-steps, which fails to leverage the inductive bias discussed in Section 4.1 We show in Section S that this simple change leads to drastic differences in both inference and generation tasks.
Implementation Details. We highlight some design considerations with ablations in Section 5.2
be smaller than T. e.g., we use feature dimension d = 512 that is smaller than T = 1. 000. This means that adjacent time-steps may share modular feature z, We compare the choice of k later.
Number of Subsets k. In practice, the number of subsets for partitioning
Partition Strategy. The simplest way is to the same number of feature dimensions for all modular features 2; (i.e., balanced). However, we empirically find that features corresponding time-step 100-300 requires more dimensions to improve convergence of Eq. 5 We design an imbalanced partition strategy and compare it with the balanced one.
Optimization Strategy. In training, we experiment detaching the gradient of 21,-.., 2 to form z. i.e,, training only 2, at time-step f. We find that it leads to improved disentanglement quality at the cost of slower convergence.
5 EXPERIMENTS
5.1 SETTINGS
Datasets. We choose real-world datasets to validate if DiTi learns a disentangled representation of the generative attributes: 1) Celebrity Faces Attributes (CelebA) Liu et al. (2015) is large-scale face attributes dataset. Each face image has 40 binary attribute labels, delineating facial features and characteristics, such as expressions, accessories, and lighting conditions. 2) Flickr-Faces-HH (FFHQ) Karras et al. (2019) contains 70,000 high-quality face images obtained from Flickr. 3) We additionally used the Labeled Faces in the Wild (LFW) dataset Huang et al. (2007) that provides continuous attribute labels. 4) Beedroom is part of the Large-scale Scene UNderstanding (LSUN) dataset Yu et al. (2015) that contains around 3 million images. We apply off the-tff image classi- fiers following Yang et al. (2021) to determine scene attribute labels.
Evaluation Protocols. Our evaluation is based on the properties of a disentangled representation (Section 3.1. For inference, we perform unsupervised learning on CelebA train split or FFHQ and test the linear-probe attribute prediction accuracy on CelebA test split, measured by Average Preci- sion (AP). We also evaluate the challenging attribute regression task using LFW, where the metrics are Pearson correlation coefficient (Pearson's 's r) and Mean Squared Error (MSE). For counterfac- tual generation, we extend the conventional interpolation and manipulation technique [reechakul eet al. (2022) to tailor for our needs: 1) Instead of interpolating the whole features of an image pair, interpolate only feature subset while keeping its complement fixed. With disentangled repre- sentation, only the attribute captured by the subset will change in the generated counterfactuals. e.g., changing only expression between two faces. In manipulation, an image is edited by altering its feature along the direction of an attribute classifier weight, where the classifier is trained on whole features. We constrain the classifier by ProbMask Zhou et al.] (202T) to use a small subset of the most discriminative feature dimensions (32 out of 512 dimensions). For disentangled representa- tion, this should have minimal impact on the generation quality, as each single attribute is encoded in compact feature subset. Detailed algorithms are in Appendix.
Baselines include 3 representative lineups: 1) Conventional unsupervised disentanglement methods 3-TCVAE Chen et al. (2018) and IB-GAN Jeon et al. (202I): 2) Self-supervised learning meth- ods SimCLR Chen et al. (2020); 3) The most related works Diff-AE Preechakul et al. (2022) and PDAE al. (2022) that also learn representation by making up DM's reconstruction errors.
Implementation Details. We followed the network design of encoder f and decoder g in PDAE and adopted its hyper-parameter settings (e.g., de, W, in Eq. 4 details in Appendix). This ensures that any emerged property of disentangled representation is solely from our leverage of the inductive bias in Section 4.1. We also used the same training iterations PDAF, ie., 290k iterations on CelebA,

---

Published as a conference paper at ICLR 2024
CelebA
FFHQ
Method
AP
Pearson's
MSE 4
AP1
Pearson's r
MSE
B-TCVAE Chen et al. 2018
45.0
0.378
0.573
43.2
0.335
0.608
[B-GAN
Jeon et al. 2021
44.2
0.307
0.597
42.8
0.260
0.644
Diff-AE
Procchakul et al 2022
60.3
0.598
0.421
60.5
0.606
0.410
PDAE Zang et al. 2022
60.2
0.596
0.410
59.7
0.603
0.416
SimCCR Chen et al. 2020
59.7
0.474
0.603
60.8
0.481
0,638
DITi (Ours)
62.3
0.617
0.392
61.4
0.622
0.384
Table I: AP (%) on CelebA attribute classification and Pearson's r, MSE on LFW attribute regression. The first column shows the training dataset.
Days
Saand
celppeppy
fecu
Supaag
s OM
VowN Rig
drng wia
axpeare
1912
-596
6%
DiTi Acc.
SimCLR Acc.
DITI Acc. PDAE Acc.
2%
15%
Rey
Dcubae Clde
Del
Muskan
Strliig
Morth Open
No Brord
Gootie
Figure 4: Improvements in attribute classification precision our DITi over PDAE (bottom) over SimCLR (top). Improvements more than 2% are highlighted with red bars. Negative values are marked with blue bars.
500k iterations on FFHQ, and 540k iterations on Bedroom. Hence our DiTi training is efficient as PDAE. Our experiments were performed on 4 NVIDIA A100 GPUs.
5.2 INFERENCE
Comparison with DifT-AE and PDAE. In T our proposed DiTi significantly outperforms the two related works. We break down the improved accuracy over PDAE on each attribute Figure bottom, where DíTi is consistently better on all attributes. The result, together with the previous analysis of Figure Pb. validates that DiTi exhibits the sample inference property representation. It also supports our analysis on the baselines' deficiency in Section 2 i.e. they may digress to use spurious correlation to supplement reconstruction error. For example, the most improved attribute is "Brown Hair", which strongly correlates with the female gender.
disentangled
Comparison with SimCLR. SimCLR does not perform well in attribute inference, surpassed even by PDAE. This is because SimCLR learns representation invariant to augmentations (e.g., color jittering). which disregards its related attributes, such as hair or skin color. This verified by the large improvements of DiTi over SimCLR "Pale Skin" accuracy in Figure top. The deficiency cannot be addressed by simply removing augmentations, which leads to severe performance degra- dation (see Appendix). Note that SimCLR slightly outperforms DiTi some unnoticeable attributes (e.g., earrings). This suggests that some fine- aie-ggrined attributes are not well disentangled. To tackle
this, we will improve model expressiveness (e.g., using Stable Diffusion Rombach et (2022) as the frozen DM) and explore more advanced training objectives Song et al. (2023) as future work.
Comparison with 3-TCVAE and IB- GAN. It is perhaps not surprising that existing unsupervised disentanglement methods perform poorly. Their previ- ous successes are limited to synthetic datasets, and their models are far less expressive compared to DM. In fact, their pursuit of disentangled representation by information bottleneck proves to be ever-elusive Locatello
Imbalance Detach
AP
Pearson's
MSE
16
62.1
0.619
0.389
32
62.0
0.615
0.392
64
62.3
0.617
0.392
128
61.9
0.616
0.391
64
61.9
0.604
0.410
62.5
0.422
64
62.4
0.604
0.405
Table 2: Ablations on DiTi designs on CelebA. Imbalance and Detach: using our partition and optimization strategy.
CovreellsiveLoocatlllo et al. (019).
Ablation on #Subsets k. From Table 2 first 4 lines, we observe that inference results are not sensitive to the subset number l (details Section 4.2p. We choose à = 64 with the highest AP.
7

---

Published as a conference paper at ICLR 2024
Early
Middle
Late
y
x to y
Pair
Pair
Pair
y tox
Pair
Paic
Figure 5: Counterfactual generations on FFI by interpolating feature subset of 3 image pairs (x,y). We partition (z.H-z according to early t E (0, T/3). middle t E (T/3, 2T and late t E (27/3, Ti. For each subset, we show results in 2 directions (x + y.y x) under 3 interpolation scales (color gradient scalef),
This means that our model can learn up to 64 hidden attributes, whose combinations are certainly diverse enough for our chosen datasets.
Ablation on Partition & Optimization Strategy. We devise an imbalanced partition strategy to allocate more feature dimensions for z, corresponding to & {100, ... 300], as this time-step range contributes the most to the overall loss (Figure A9). This leads to faster convergence and improved performance in Table 2 (line vs. line 5). For optimization strategy, tried detaching the gradient of zı..... 2t-1 at time-step t to prevent them from capturing the lost attribute at t. Yet leads to slower convergence, which especially hurts the challenging regression task. We leave it future work to explore more principled ways of improving feature modularity.
5.3 COUNTERFACTUAL GENERATION
FFHQ Results. Figure 5 shows DiTi generations by interpolating a feature subset while fixing its complement. Different from full-feature interpolation (Figure 77 there two directions: x y fixes the complement to the corresponding feature values of x. and y - x fixes to those of y. By identifying how images are changing during interpolation (left to right for top, right to left for bottom) and comparing the changes across the 3 columns, we have several observations:
1. Interpolating different subsets modifies different attributes. For example, subset of early t mainly controls micro-expressions, such as mouth corner,, cheeks, or eye details. The interpolation hardly changes a person's identity, but certainly moves one's temperament towards the other person. In contrast, subset of middle t is more responsible for identity-related attributes (e.g., proportion of facial features) and accessories (e.g., eyeglasses). Finally, late controls attributes such as hairstyle, pose, and even position of the face in the image (e.g., the bottom right face moves towards center).
2. From early to late t, the corresponding feature subset controls a more coarse-grained feature, where the granularity is defined based on the pixel-level changes. For example, changing hairstyle or removing hat (subset of late t) modifies more pixels than changing facial feature proportions (subset of middle t). This is in line with our theoretical analysis in Section 4.1 Notably, both our feature subset partition and the correspondence between subsets and attributes granularity are known a priori, without relying manual discovery techniques such as latent traversal.
3. Our generated counterfactuals have minimal distortion or artifact, and demonstrate modular prop- erties (e.g., subset of late controls pose with minimal impact on facial features and accessories). These are strong proofs that DiTi learns a disentangled representation Besserve et al. (2020) We show Appendix that all generative baselines do not exhibit these traits.
8

---

Published as a conference paper at ICLR 2024
Diff-AE original
PDAE original
DITI original
Attr.
Indoor Lighting
Cluttered
Carpet
Figure 6: Counterfactual generations on Bedroom by manipulating out 512 feature dimensions.
Diff-AE
PDAE
DiTi
Figure 7: Results of interpolating the whole feature on FFHQ. Baselines have distortions during interpolation.
Bedroom Manipulation. Bedroom is a more challenging dataset, where room scenes are much more complex than facial images (e.g., various layouts, decorations, objects). As shown in Figure DiTi is the only method that generates faithful counterfactuals. 1st row: only DiTi can alter lighting without making significant changes to room furniture. 2nd row: baselines fail to modify the rustic feeling without artifacts or changing room layouts. 3rd row: DiTi makes the room more cluttered by introducing additional objects. Last row: only DiTi adds additional carpet.
Whole-feature Interpolation. Under this conventional setting, as shown in Figure 7 only DiTi smoothly interpolates between the attributes of each image pair, with minimal distortion and artifact along the trajectories. This means that our DiTi can generate faithful counterfactuals (i.e., valid im- ages in the support of data distribution), which validates disentanglement quality Besserve et al. (2020. For example, if feature only captures the attribute "smile" (i.e., disentangled), interpolat- ing this feature between smiling person and non-smiling leads to the gradual transition of expression. However, if a feature entangles multiple attributes, such interpolation will alter all of them or even distortion when the interpolated feature of the data distribution.
CONCLUSION
We presented novel unsupervised method learn a disentangled representation, which leverages the inductive bias of diffusion time-steps. In particular, reveal inherent connection between time-step and hidden modular attributes that generate data faithfully, enabling simple and effective approach disentangle the attributes by learning time-step-speciff feature. The learned feature improves downstream inference and enables counterfactual generation, validating its disentangle- ment quality. As future work, will seek additional inductive bias to improve disentanglement, e.g., using text as a disentangled template by exploring text-to-image diffusion models, and devise practical optimization techniques enable faster convergence.
ACKNOWLEDGEMENT
This research is supported by Microsoft Research Asia, the National Research Foundation, Sin- gapore under its AI Singapore Programme (AISG Award No: S-GG2-RP-2021-00 MOE AcRF Tier 2 -MOE0019-T22--6666
Wallenberg-NTUU Presidential Postdoctoral Fellowship, the Lee Kong Chian (LKC) Fellowship fund awarded by Singapore Management University, and the DSO Re- search Grant (Fund Code MG22C03).

---

Published as a conference paper at ICLR 2024
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1988-1828, 2013.
Michel Besserve, Arash Mchrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. ICLR, 2020.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerehner. Understanding disentangling in beta-vae. ICLR, 2017.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources disen- tanglement in variational autoencoders. NeurlPS, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35 (1):53-65, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint 02X:622208.01666, 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 18/1 02.02230 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6855 2020.
Gary Huang, Vidit Jain, and Erik Leamed-Miller. Unsupervised joint alignment of complex images. In ICCV, 2007.
Henry F Inman and Edwin L Bradley Jr. The overlapping coefficient measure of agreement between probability distributions and point estimation of the overlap two normal densities. Communications in Staristics-theor and Methods, 18(10):8851-3874 1989.
Insu Jeon, Wonkwang Lee, Myeongjang Pyeon, and Gunhee Kim. Ib-gan: Disentangled represen- tation learning with information bottleneck generative adversarial networks. In AAAI, 2021.
Thomas W. Judson. Abstract Algebra: Theory and Applications (The Prindle, Weber & Schmidt Series Advanced Mathematics). Prindle Weber Schmidt, 1994.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
10

---

Published as a conference paper at ICLR 2024
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. ICLR, 2022.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In international conference on machine learning, 2019.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffu- sion autoencoders: Toward meaningful and decodable representation. In CVPR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-2411 Springer, 2015.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Processing Systems, 34:1 82255-8240. 2021.
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative repre- sentations for scene synthesis. International Journal Computer Vision, 129:1451-1466 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arxiv: 550.03365, 2015.
Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Counterfactual zero- shot and open-set visual recognition. In CVPR, 2021.
Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised representation learning from pre-trained diffusion probabilistic models. Advances in Neural Information Processing Systems, 35:22117- 22130, 2022.
Xiao Zhou, Weizhong Zhang. Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In CVPR, 2021.
11


Reviews:
[{"review_id": "wPztMP4DDp", "reviewer": "Reviewer_Kc2k", "summary": "This paper presents a unique perspective on exploiting the image-noise ratio across different time steps in diffusion models and offers a framework for learning useful and disentangled representations from diffusion models. The authors argue that as t increases, images progressively loss information starting from details to global structures during noise injection, and learning a complementary feature representation conveying the corrupted information can help make the representation more disentangled. Experiments are performed on multiple image datasets.", "questions": "Could the authors provide a brief discussion for [1]?\n\n\n---\n[1] Diffusion Based Representation Learning, arxiv", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of systematically studying the representation changes during the noise injection of diffusion models is interesting and has great potential in further understanding the diffusion models. \n\n2. The paper is written in good presentation quality in general. The definitions such as attribute loss are adequately formulated and illustrated with figures. \n\n3. The quantitative results in Table 1 seem strong. \n\n4. The method is built on top of pretrained diffusion models, which I believe can help reduce the cost of representation learning.", "weaknesses": "The empirical discussions on counterfactual generation (section 5.3) are relatively weak as the effectiveness is only supported by a few examples. Especially for the bedroom experiments, I personally feel like it's not easy to justify the authors' claim that 'DiTi is the only method that generates faithful counterfactuals' based on the given examples.\n\nAnd since this paper focuses on unsupervised representation learning, more empirical results regarding how the learned disentanglement can improve the overall representation quality can be a huge plus. \n\nFor example, do the learned disentangled representations enable human intervention to mitigate spurious correlations? \n\nHow is the quality of the representation when evaluated in a more general setting such as ImageNet-style recognition? \n\nOne concern I do have is that as diffusion is trained by image reconstruction, the learned feature will inevitably contain considerable information regarding the background (probably in late step t), and how will this affect the general representation quality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYVF71pa8c", "reviewer": "Reviewer_Cjby", "summary": "This work aims to disentangle the modular attributes in the DDPM framework by exploiting the granularity of feature details at different time steps of the diffusion models. The work finds that the fine feature components are encoded in the earlier time steps of the forward diffusion process while the coarse details are at the later time steps. A encoder decoder approach with partitioned features  at different time steps are proposed to allow for disentangled representations. Experiments are performed on bedroom, CelebA and FFHQ datasets where the method outperforms prior art.", "questions": "- From point 2 above, it will be great to include the plot to validate the time steps and loss. \n-The results presented in the manuscript hold for unconditional models. Do the findings also extend to conditional models? \nAlso see weaknesses above.\n\n\nMinor: The equations are missing parenthesis on expectation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and easy to read. The motivation of the work related to learning the disentangled modular attributes in diffusion models is promising. Claims are supported with theoretical reasoning. \n\n+ The experiments are performed on different face datasets such as CelebA, FFHQ, and the Bedrooms dataset. The approach outperforms prior work on metrics such as AP, Pearson-r, and MSE. \n\n+ Ablations are provided regarding the choice of partitioning and optimization strategies.", "weaknesses": "- The qualitative results shown in figure 5 do not show consistent improvements wrt to modular editing. In the second row, for example, the eyeglasses appear in the middle range and disappears again in the final ranges.  Are the time steps plotted in a cumulative fashion within the  early/middle/late “t” range. \n\n- On the use of timesteps [100-300] and it corresponding to the maximum value of the loss: One would expect the loss to be greater at the later stages when the image becomes complete noise. Eg: it should be difficult to construct image from t =1000 to t =700. Can a plot be included for different samples to validate this? \n\n- Comparison to SIMCLR: SimCLR does obtain a higher accuracy on attributes affecting local appearances (e.g., “Hat”). We postulate that its contrastive training has some effects in regularizing f as an injective mapping” . This is not clear. What is being implied in these findings. Does this mean that the proposed approach depends on attribute correlations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5FPxJglnen", "reviewer": "Reviewer_5NeX", "summary": "This work proposes DiTi, a method that learns disentangled representations in an unsupervised fashion. The work first provides insights into the inductive bias of Denoising Diffusion Probabilistic Models that can be leveraged for learning decoupled features, with theoretical grounding. Then the work proposes to leverage the inherent connection between timesteps and modular attributes to learn a set of features from the residuals. Finally, the feature can be used for both downstream inference (e.g., attribute prediction) and counterfactual generation, surpassing prior works in both tasks.", "questions": "* Do the subsets have to be uniform? Does it benefit the method if we have more features for coarser or finer features?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work provides a link between the timesteps and the modular attributes with an intuitive explanation and theoretical proof.\n* This work proposes a simple yet effective approach to learn disentangled features along with the diffusion model training that follows from the discoveries during the analysis.\n* This work outperforms previous unsupervised feature learning methods that do not disentangle the features on attribute classification tasks. This work also shows higher-quality counterfactual generations compared to previous works.", "weaknesses": "* This work trains models from scratch. Due to the introduced term for disentangled feature learning, the model needs to be re-trained.\n* The authors only evaluated the proposed method with datasets that are domain-specific (e.g., CelebA, FFHQ). The method has not been evaluated on large models trained on more diverse and general image datasets (e.g., LAION). Therefore, it does not indicate whether the method can scale to more complicated settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SANkVZSPin", "reviewer": "Reviewer_1YJJ", "summary": "I was one of the reviewers for this paper for NeurIPS 2023, I have carefully revisited the paper again for the ICLR submission, and write my new reviews as below.\n\nThis paper studies unsupervised representation learning with generative diffusion probabilistic models with specific exploration of the diffusion steps. The main research idea originates from the observation/assumption that image attributes are gradually lost along the diffusion process with increasing levels of Gaussian noises. The authors propose DiTi (named after Diffusion Time-step) to learn a step-specific feature to capture the information of lost attributes, and then leverage this feature of modular attributes as the inductive bias for unsupervised learning. Experimental tests of the feature are conducted on two settings, namely the attribute classification and counterfactual generation.", "questions": "In general, I don’t have specific questions to ask at this time and still have mixed feelings as in my first-time review of this work. I think this paper has its own merits/advantages and drawbacks/disadvantages at the same time, and thus keep my tentative rating as borderline but will update it according to other reviewers and further discussions later.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper elaborates well on the concept of attribute loss, which is the fundamental observation for the proposed DiTi method, and is in general easy to follow.\n\n- The idea to learn a step specific feature to encode the lost attribute information as the inductive bias for unsupervised learning is reasonable and intuitive.\n\n- Experimental results show improvement over several baseline methods on CelebA, FFHQ and LSUN-Bedroom datasets.\n\n- Compared to the previous manuscript from NeurIPS, it seems that the authors have incorporated the comparison of the proposed work with several existing works that touch on the representation learning with DMs, which helps to clarify the differences between this paper and existing literature (but this part is also a little biased, as specified below).", "weaknesses": "Since this is my second time reviewing this paper, I think most of my questions concerning the technical details have been resolved and clarified. While I acknowledge that the paper is well-written and has its merits to enlighten the studies of combining representation learning and diffusion models, there are a few of my concerns.\n\n- First, I echo with my previous NeurIPS reviewers folks on the concern: whether it is the direct direction to actually integrate DMs into the current representation learning paradigm? Based on our previous discussions, the proposed method has a higher computational/time cost compared to its contrastive counterparts. And if the argument here is because the contrastive paradigms lose attribute information, then it seems to me the proposed DiTi may not be the optimal method to introduce the attribute inductive bias.\n \n- Second, in the related work section, the authors include a comparison with the current DM for representation learning methods. I find this new part slightly biased, because the approaches in 2) and 3) are not really proposed under the context of representation learning. For instance, (Kwon et al., 2022) proposed the bottleneck feature of the U-Net under the context of image editing; similar to (Preechakul et al., 2022), in which separate autoencoders are learned to achieve editing purposes. I think while these works do touch the latent features/attributes in DMs, they tackle different scenarios other than representation learning. In other words, the first concern is not resolved by comparing these works.\n\n- Going towards the technical level, one concern (less urgent in my sense) is that the proposed DiTi method shares quite similarities with PDAE, with the main difference in the disentanglement, with the cost of more hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)