Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models
Authors: Zijian Zhang
Zhou Zhao
Zhijie Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Diffusion Probabilistic Models
Representation Learning
Autoencoders

Abstract: Diffusion Probabilistic Models (DPMs) have shown a powerful capacity of generating high-quality image samples. Recently, diffusion autoencoders (Diff-AE) have been proposed to explore DPMs for representation learning via autoencoding. Their key idea is to jointly train an encoder for discovering meaningful representations from images and a conditional DPM as the decoder for reconstructing images. Considering that training DPMs from scratch will take a long time and there have existed numerous pre-trained DPMs, we propose \textbf{P}re-trained \textbf{D}PM \textbf{A}uto\textbf{E}ncoding (\textbf{PDAE}), a general method to adapt existing pre-trained DPMs to the decoders for image reconstruction, with better training efficiency and performance than Diff-AE. Specifically, we find that the reason that pre-trained DPMs fail to reconstruct an image from its latent variables is due to the information loss of forward process, which causes a gap between their predicted posterior mean and the true one. From this perspective, the classifier-guided sampling method can be explained as computing an extra mean shift to fill the gap, reconstructing the lost class information in samples. These imply that the gap corresponds to the lost information of the image, and we can reconstruct the image by filling the gap. Drawing inspiration from this, we employ a trainable model to predict a mean shift according to encoded representation and train it to fill as much gap as possible, in this way, the encoder is forced to learn as much information as possible from images to help the filling. By reusing a part of network of pre-trained DPMs and redesigning the weighting scheme of diffusion loss, PDAE can learn meaningful representations from images efficiently. Extensive experiments demonstrate the effectiveness, efficiency and flexibility of PDAE.

Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models
Zijian Zhang'
Zhou Zhao
Zhijie Lin
Department of Computer Science and Technology. Zhejiang University
"ea Al Lab
{ckezzj zwwozhou) (arrzhhoou@juju.... @zju. aaju.eddu edu. Linzj@seal com
Abstract
Diffusion Probabilistic Models (DPMs) have shown a powerful capacity of generat- ing high-quality image samples. Recently, diffusion autoencoders (Diff-AE) have been proposed to explore DPMs for representation learning via autoencoding, Their key idea is to jointly train encoder for discovering meaningful representations from images and a conditional DPM as the decoder for reconstructing images. Considering that training DPMs from scratch will take long time and there have existed numerous pre-trained DPMs, we propose Pre-trained DPM AutoEncoding (PDAE), a general method adapt existing pre-trained DPMs the decoders for image reconstruction, with better training efficiency and performance than Diff-AE. Specifically, we find that the reason that pre-trained DPMs fail reconstruct image from its latent variables is due to the information loss of forward process, which causes gap between their predicted posterior mean and the true one. From this perspective, the bussiifierguuide sampling method can be explained as comput- ing an extra mean shift fill the gap, reconstructing the lost class information samples. These imply that the gap corresponds to the lost information of the image, and we can reconstruct the image by filling the gap. Drawing inspiration from this, we employ a trainable model to predict mean shift according to encoded represen- tation and train it to fill as much gap as possible, in this way, the encoder is forced to learn much information as possible from images to help the filling. By reusing a part network of pre-trained DPMs and redesigning the weighting scheme diffusion loss, PDAE can learn meaningful representations from images efficiently. Extensive experiments demonstrate the effectiveness, efficiency and flexibility PDAE. Our implementation is available at https//giitub.oomcccc//////
Introduction
Deep generative models such variational autoencoders (VAEs) 25
39 generative adversarial
networks (GANs) [3]. autoregressive models [50 48, normalizing flows (NFs) 38
and energy- based models (EBMs) 9 45) have shown remarkable capacity synthesize striking image samples.
23
Recently, another kind of generative models, Diffusion Probabilistic Models (DPMs) 43 14]
are
further developed and becoming popular for their stable training process and late-of-theearr sample quality []] Although a large number of degrees of freedom in implementation, the DPMs discussed in this paper will refer exclusively those trained by the denoising method proposed in DDPMs (14)
Unsupervised representation learning via generative modeling is popular topic in computer vision. Latent variable generative models, such GANs and VAEs, a natural candidate for this, since they inherently involve latent representation of the data they generate. Likewise, DPMs inherently
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

yield latent variables through the forward process. However, these latent variables lack high-level semantic information because they are just sequence of spatially corrupted images. In light of this, diffusion autoencoders (Diff-AB) [6] explore DPMs for representation learning via autoencoding. Specifically, they employ an encoder for discovering meaningful representations from images and conditional DPM as the decoder for image reconstruction by taking the encoded representations as input conditions. Diff-AE is competitive with the state-of-the-art model on image reconstruction and capable of various downstream tasks.
Following the paradigm of autoencoders, PDAE aims to adapt existing pre-trained DPMs to the decoders for image reconstruction and benefit from it. Generally, pre-trained DPMs cannot accurately predict the posterior mean of z from De in the reverse process due to the information loss of forward process, which results in gap between their predicted posterior mean and the true one. This is the reason that they fail to reconstruct an image (mo) from its latent variables (æt). From this perspective, the classifier- guided sampling method [ can be explained as reconstructing the lost class information in samples by shifting the predicted posterior mean with an extra item computed by the gradient of classifier to fill the gap. Drawing inspiration from this method that uses the prior knowledges (class label) to fill the gap, we aim to inversely extract the knowledges from the gap, i.e., learn representations that can help to fill the gap. In light of this, we employ a novel gradient estimator to predict the mean shift according to encoded representations and train it to fill as much gap possible, in this way. the encoder is forced to learn much information possible from images to help the filling. PDAE follows this principle to build an autoencoder based on pre-trained DPMs. Furthermore, we find that the posterior mean gap in different time stages contain different levels of information, so we redesign the weighting scheme diffusion loss to encourage the model to learn rich representations efficiently. We also the convergence of our model. Based on pre-trained DPMs, PDAE only needs less than half of the training time that Diff-AE costs complete the representation learning but still outperforms Diff-AE. Moreover, PDAE also enables some other interesting features.
part of network of pre-trained DPMs to accelerate
2 Background
2.1 Denoising Diffusion Probabilistic Models
DDPMs 14
employ a forward process that starts from the data distribution g(æo) and sequentially corrupts to (0, I) with Markov diffusion kernels g( (z+=1) defined by fixed variance schedule (3.)2- The process can be expressed by:
T ( = II q(<1|(1111)
q( x1-1) = Nax: v11 1,01-1 B,I)
(1)
where {xt)T=1 are latent variables of DDPMs. According to the rule of the sum of normally distributed random variables, can directly sample H; from zo for arbitrary t with g( z0)
N(x;;; vā,zo (1 (1-ā,) ā,) where a, 1 8, and ā; II' 0ze-
The reverse (generative) process is defined as another Markov chain parameterized by 0 to describe the same but reverse process, denoising arbitrary Gaussian noise to a clean data sample:
T
Po( = (xx Mo 1), Eo(xx,t))
po(wor:) = p(ær) II po [x)),
(2)
where p(2r) N (x>: 0, I). employs po po (ax)) of Gaussian form because the reversal of the diffusion process has the identical functional form as the forward process when 3r is small The generative distribution can be represented as (zoo
43
(Furr))) )dx1.
Training is performed to maximize the model log likelihood f q ( ) log pe (2xx)d by minimizing the variational upper bound of the negative one. The final objective is derived by some parameteriza- tion and simplication [14]
Ceimpte( (0) = Em....e ||e
colVoaaa + vI -I- āec, t)]
(3)
where is function approximator to predict from 2t.-

---

2.2 Denoising Diffusion Implicit Models
define non-Markov forward process that leads the same training objective with DDPMs, but the corresponding reverse process can be much more flexible and faster to sample from. Specifically, one can sample Φt-1 from 2, using the Eg of some pre-trained DDPMs via:
DDIMs
44
v1- āi - E + v d o eg 6 () + ofte,
04-1
(4)
Vāx
where 62 (0, 1) and or controls the stochasticity of forward process. The strides greater than 1 are allowed for accelerated sampling. When o = 0, the generative process becomes deterministic, which is named as DDIMs.
2.3 Classifier-guided Sampling Method
Classifier -esiffeer-guieed sampling method
43
46 8 shows that one can train classifier Po (22) noisy
data and use its gradient V Va, Yy y to guide some pre-trained unconditional DDPM to sample towards specified class y. The conditional reverse process
be approximated by a Gaussian similar
to that of the unconditional one in Eq. ( but with a shifted mean:
Pe ( 1 y) (( (Z + Eo( t) Va, log (y)(a))) Ee(z,,
(5)
For deterministic sampling methods like DDIMs, one can use score-basse conditioning trick [46
45
to define new function approximator for conditional sampling:
Co( 1) <n < () - v1 ā. =., logPo 2..)
(6)
More generally, any similarity estimator between noisy data and conditions can be applied for guided sampling, such as noisy- -LIP guidance 33 31
3 Method
3.1 Forward Process Posterior Mean Gap
Generally, one will train unconditional and conditional DPMs by respectively learning (=( (2x)
(i Me (e, t), Eg 6)) and pa pa( y) = N (x (a y, (), Eg De y/ f)) to ap- proximate the same forward process posterior ( [rt mo) = (4, vo); 1- 3.I). Here y is some condition that contains some prior knowledges of corresponding 2g. such as class label. Assuming that both Eg is set to untrained time dependent constants, under the same experi- mental settings, the conditional DPMs will reach a lower optimized diffusion loss. The experiment in Figure can prove this fact, which means that ( y, () is closer pa(e; ao) than ns (æ,,) l). This implies that there exists a gap between the posterior mean predicted by the unconditional DPMs (a, ()) and the true one (jae( .0)). Essentially, the posterior mean gap is caused by the information loss of forward process so that the reverse process cannot recover it in æ;-1 only according x;.
1-a:
introduce some knowledges of vo for DPMs. like y here, the gap will be smaller. The more information of æo that contains, the smaller the gap is.
Moreover, according to 5). the Gaussian mean of massifier-guidee conditional reverse process contains an extra shift item compared with that of the unconditional one. From the perspective of posterior mean gap. the mean shift item can partially fill the gap and help the reverse process to reconstruct the lost class information in samples. In theory, if in Eq.(5) contains all information of Zo. the mean shift will fully fill the gap and guide the reverse process reconstruct 25p On the other hand, if we employ a model to predict mean shift according to our encoded representations z and train it to fill as much gap as possible, the encoder will be forced to learn as much information as possible from To to help the filling. The more the gap is filled, the more accurate the mean shift is, the more perfect the reconstruction is, and the more information of 35 that z contains. PDAE follows this principle to build an autoencoder based on pre- pre-trained DPMs.
3.2 Unsupervised Representation Learning by Filling the Gap
Following the paradigm of autoencoders, we employ encoder z = (xoo) for learning compact and meaningful representations from input images and adapt pre-trained unconditional DPM pe( (x,) (w (x t), (xx,,)) to the decoder for image reconstruction.
3

---

co(as, Eo(ar.
0003
Blbiud 2014
6003
A M(LT)
6002
Golm (x))))
0004
skip connections
6003
60 E
forwerd pess
Figure 1: Comparison of diffusion loss between unconditional and conditional DPM trained on MNIST 28
Figure 2: Network and data flow of PDAE. The gray part represents the pre-trained DPM, which frozen during training.
Specifically, we employ gradient estimator Go t) to simulate Va, log p(=((=)); where p( (x,) some implicit classifier that we will not use explicitly, and use it to assemble conditional DPM ( ( ( =) (x;-1;; po æt, t) + (x,,, t) (Z,, 2, t). Ex t)) as the decoder. Then
Pe,v
we train like regular conditional DPM by optimizing following derived objective (assuming the e-prediction parameterization is adopted):
((V(p) Exo.t,r delle co(xx:t) t) VoiVI-a ai 3e
Ex(x,, (Z (Ze, Ex (o))),,)))) e)|
(7)
where Vararo v1 age and d is a new weighting scheme that we will discuss in Section 3.4 Note that we use pre-trained DPMs so that are frozen during the optimization. Usually we set Eg = 1-8-1 3,I to untrained time-dependent constants. The optimization is equivalent to minimizing || [x,,, (). (Wr, (xoo)), t) (Tr, (e)))-us(a: ()) 1 which forces the predicted mean shift Eg (x,,,)) Go (Zz, (zo), t) to fill the posterior mean gap f4: (I,, xD) (1
T-a;
With trained Go 2, t), we can treat it as the score of an optimal classifier p(z[2;) and use the classifier-guice guided sampling method (q. for DDPM sampling or use the modified function approximator Eg in Eq. for DDIM sampling, based on pre-trained co (Iz,, t). We put detailed algorithm procedures in Appendix ??.
Except the semantic latent code 2, we can infer a stochastic latent code ær deterministic generative process of DDIMs in reverse:
36 by running the
Øt-1 (a+11
Ct
VI ai Co(x;,
+ /1- uit1 (o(
(8)
Vān
This procedure optional, but helpful to near-exact reconstruction and real-image manipulation for reconstructing minor details of input images when using DDIM sampling.
We also train a latent DPM p_ (Zz- to model the learned semantic latent space, same with that in Diff-AE [36]. With trained latent DPM, we can sample z from it to help pre-trained DPMs to achieve faster and better unconditional sampling under the guidance of GZ t,t).
3.3 Network Design
Figure 2 shows the network and data flow of PDAE. For encoder E,g, unlike Diff-AE that uses the encoder part of U-Net [40]. we find that simply stacked convolution layers and linear layer is enough learn meaningful z from Fo. For gradient estimator Gy, we use U-Net similar to the function approximator eg of pre-trained DPM. Considering that Eg also takes æ; and as input, we can further leverage the knowledges pre-trained DPM by reusing its trained encoder part and time embedding layer, so that we only need to employ new middle blocks, decoder part and output blocks of U-Net for G.- To incorporate into them, we follow [8] extend Group Normalization [53] by applying scaling & shifting twice normalized feature maps:
AdaGN(h, I, z2 = zz (roGrropNNrm(h)) + (b) +
(9)

---

carly-stag
criticl-stage stage
late-stage
V1.5
DDP9
image
1)
fg
T
mixed sampling procedure linput 0-9 when classilier guided sampling
7 4 6 q
5 5 5 /
/
ate stage
/ /
2 3 4
5 6 9
attical-stage
8 4/4 D 4
3
7
3
6 6
3
early-tage
160
Signalu Nr Ratoo DNA
Figure 3: Investigations of the effects of mean shift for different time stages. We perform a 50-step-grid- search for (41, 12) pairs to find the shortest critical- stage that can ensure high accuracy of conditional generation. For MNIST (28), it is (400, 700).
Figure 4: Normalized weighting schemes of diffusion loss for different DPMs rel- ative to the true variational lower bound loss. Linear variance schedule is used.
where [te, to] and [=., 23] are obtained from a linear projection t and z, respectively, Note that we still use skip connections from reused encoder to new decoder. In this way, Ge is totally determined by pre-trained DPM and can be universally applied to different U-Net architectures.
3.4 Weighting Scheme Redesign
We originally worked with simplified training objective like that in DDPMs [4], i.c. setting X, = 1 in (q. but found the training extremely unstable, resulting slow/non- convergence and poor performance. Inspired by P2- P2-weighting [7). which has shown that the weighting scheme of diffusion loss can greatly affect the performance of DPMs, we attribute this phenomenon to the weighting scheme and investigate it in Figure g
Specifically, train unconditional DPM and noisy classifier on MNIST 28 and divide the diffusion forward process into three stages: early-stage between 0 and t1, critical between t and to and late-stage between to and shown in the top row. Then we design mixed sampling procedure that employs unconditional sampling but switches to classifie-guuide sampling only during the specified stage. The bottom three rows show the samples generated by three different mixed sampling procedures, where each row only employs classifier-guii -uiiidd sampling during the specified stage on the right. As we can see, only the samples guided by the classifier during critical-stage match the input class labels. We can conclude that the mean shift during critical-stage contains more crucial information to reconstruct the input class label in samples than the other two stages. From the view of diffusion trajectories, the sampling trajectories are separated from each other during critical-stage and they need the mean shift to guide them towards specified direction, otherwise will be determined by the stochasticity of Langevin dynamics. Therefore, we opt to down-weight weight the objective function for the in early- and late-stage to encourage the model to learn rich representations from critical critical-stage. Inspired by P2- 12-weighting 77). redesign weighting scheme of diffusion loss (A, in Eq. (7) in terms of signal-to-noise ratio 24 (SNR(t) = )
SNR(L) ( SNR(F)) )",
N 1+ SNR(F)
(10)
where the first item is for etrly-stage and the second one for late-stage. 7 hyperparameter that balances the strength of wein-wegghting between two items. Empirically we set y 0.1. Figure 4 shows the normalized weighting schemes of diffusion loss for different DPMs relative the true variational lower bound loss. Compared with other DPMs, our weighting scheme down-weights the diffusion loss for both low and high SNR.
Experiments
To compare PDAE with Diff-AE 36), we follow their experiments with the same settings. Moreover, we also show that PDAE enables some added features. For fair comparison, we use the baseline DPMs provided by official Diff-AE implementation pre-trained models (also baselines), which
5

---

25:5
PIAE
220
800
160
0.00
700
8:00
tiresdep
Figure 5: Left: Predicted 2n by denoising æ; for only step. The first pre-trained DPM and the second row use PDAE. Right: Average posterior mean gap for all steps.
have the same network architectures (hyperparameters) with their Diff-AE models. For brevity, we use the notation such as "FFHQQ22-1 1012288----------1111 to name our model, which means that we use baseline DPM pre-trained with 130M images and leverage it for PDAE training with 64M images, on 128 x 128 FFHQ dataset [21]. with the semantic latent code of 512-d. We put all implementation details in Appendix ?? and additional samples of following experiments in Appendix ??.
4.1 Training Efficiency
We demonstrate the better training efficiency of PDAE compared with Diff-AE from two aspects: training time and times. For training time, we train both models with the same network architectures (hyperparameters) 128 x 128 image dataset using Nvidia A A100-SXM4 GPUs for distributed train- ing and set batch size to 128 (32 for each GPU) to calculate their training throughput (imgs/sec.// 100). PDAE achieves throughput of 81.57 and Diff-AE achieves that of 7..41. Owing to the reuse of the U-Net encoder part of pre-trained DPM, PDAE has less trainable parameters and achieves a higher training throughput than Diff-AE. For training times, we find that PDAE needs about 4 of the number of training batches (images) that Diff-AE needs for loss convergence. We think this is because that modeling the posterior mean gap based on pre-trained DPMs easier than modeling conditional DPM from scratch. The network reuse and the weighting scheme redesign also help. As result, based on pre-trained DPMs, PDAE needs less than half of the training time that Diff-AE costs to complete the representation learning.
4.2 Learned Mean Shift Fills Posterior Mean Gap
We train a model of 100-22212--3002551----A-- and show that our learned mean shift can the posterior mean gap with qualitative and quantitative results in Figure Specifically, we select some images my from FFHQ, sample 2; va + vT azé for different and predict á from 3r by denoising them for only one step (i.e., 2o m. -a-VI-art using pre-trained DPM and PDAE respectively. As
Va,
see in the figure (left), for large PDAE can predict accurate noise from I; and reconstruct plausible images, which shows that the predicted mean shift fills the posterior mean gap and the learned representation helps recover the lost information of forward process. Furthermore, we randomly select 1000 images from FFHQ, sample æ, va,ao + /1 â,e and calculate their average posterior mean gap for each step using pre-trained DPM: | (2,, 2z) μs (z )) and PDAE: [| (I,, zo) ( t) + Eg @2 t) G E (xx). t)) respectively. shown in the figure (right). As we can see, PDAE predicts the mean shift that significantly fills the posterior mean gap.
4.3 Autoencoding Reconstruction
We use 'TFHQQ22-110M-22 12-64M" to run some autoencoding reconstruction examples using PDAE generative process of DDIM and DDPM respectively. As we can see in Figure 6 both methods generate samples with similar contents to the input. Some stochastic variations (36 occur in minor details of hair, eye and skin when introducing stochasticity, Due to the similar performance

---

DDIM Input (7-100, interee ry)
DDIM
DDPM
(T-100 random o))
Figure 6: Autoencoding reconstruction examples generated by different sampling methods, Each row corresponds to an example.
with
Table 1: Autoencoding reconstruction quality of 1TFH0128-130--25122666 on CelebA-HO.
Model
Latent dim SSIM f LPIPS | MSE
StyleGAN2 (W inversion)
512
0.677
0.168
0.016
StyleGAN2 (W+ inversion)
7.168
0.827
0.114
0.006
VQ-GAN [0]
65,536
0.782
0.109
3.61e-3
VQ-VAB2 37
327,680
0.947
0.012
4.87e-4
NVAE 47
6,005,760
0.984
0.001
4.85e-5
Diff-AE 130M (T=100, random #r) 33
512
0.677
0.073
0.007
PDAE @ 664M (T=100, random wr)
512
0.696
0.094
0.005
DDIM @ 130M (T= 100) [
49,152
0.917
0.063
0,002
Diff-AE @ 130M (T=100, inferred ar))
36
49,664
0.991
0.011
6.07e-5
PDAE @64M (T=100, inferred ar)
49,664
0.993
0.008
5.48e-5
between DDPM and DDIM with random ær, we will always use DDIM sampling method later experiments. We can get near-exact reconstruction if use the stochastic latent code inferred from aforementioned ODE, which further proves that the stochastic latent code controls the local details.
To further evaluate the autoencoding reconstruction quality of PDAE, we conduct the same quantita- tive experiments with Diff-AE. Specifically, we use "FFHQ128" 100QQQ22--30--1122- 130M-2512-64M" to encode-and- reconstruct all 30k images of CelebA-HQ (20) and evaluate the reconstruction quality with their average SSIM (52), LPIPS (56) and MSE. We use the same baselines described in [36), and the results are shown in Table T We can see that PDAE is competitive with the state-of-the-an NVAE even with much less latent dimensionality and also outperforms Diff-AE in all metrics except the LPIPS for random Ty. Moreover, PDAE only needs about half of the training times that Diff-AR AE needs for representation learning, which shows that PDAE can learn richer representations from images more efficiently based pre-trained DPM.
4.4 Interpolation of Semantic Latent Codes and Trajectories
to encode them into (z', .x/)) and (z² x and run PDAE generative process of DDIM starting from Slerp(æ/) æ A) under the guidance of G (I,, Lerp(x', 2); t) with 100 steps, expecting smooth transitions along A. Moreover, from the view of the diffusion trajectories, PDAE generates desired sam- ples by shifting the unconditional sampling trajectories towards the spatial direction predicted by Gæ 2, t). This enables PDAE to directly interpolate between two different sampling trajectories. Intuitively, the spatial direction predicted by the linear interpolation of semantic latent codes, G (3r, Lerp(= 22; A), t). should be equivalent to the linear interpolation of two spatial directions
Given two images v and x3 from FFHQ, we use "FFHQ128-
predicted by respective semantic latent code, Lerp( ( z1, 2²,1). C. (Zt, z² 2²,();
(); 2). We present
some examples of these two kinds of interpolation methods Figure 7 As we can see, both methods
generate similar samples that smoothly transition from one endpoint to the other, which means that

---

0.1 0.2
0.8
0.9
Figure 7: Interpolation examples generated by the first row use the guidance of Go (Zi, Lerp(z*, 2²; t) and the second row use the guidance of Lerp(C (a z², t), Go (xi, 2², t); A).
For each example,
Smiling
No Beard
Male
Young
Attribute manipulation examples generated by H214A-52228---2-51222222 For each example, we manipulate the input image (middle) by moving its semantic latent code along the direction of corresponding attribute found by trained linear classifiers with different scales.
Figure 8:
(ar Lerp(z), =2;A), Cerp(GGs(aa 2 1), ( x², (); A). so that G z, 1) can be seen as function of z analogous to a linear map. The lincarity guarantees meaningful semantic latent space that represents the semantic spatial change of image by a linear change of latent code.
4.5 Attribute Manipulation
We can further explore the learned semantic latent space in supervised way. To illustrate this, we train a model of "CelobA-HQI28-355 2228-522-25512-22 and conduct attribute manipulation experiments by utilizing the attribute annotations of CelebA-HQ dataset. Specifically, we first encode an image to its semantic latent code, then move along the learned direction and finally decode it to the manipulated image. Similar Diff-EE, we train linear classifier to separate the semantic latent codes of the images with different attribute labels and use the normal vector of separating hyperplane (i.e. the weight of linear classifier) the direction vector. We present some attribute manipulation examples in Figure g As we can see, PDAE succeeds in manipulating images by moving their semantic latent codes along the direction of desired attribute with different scales. Like Diff-AE, PDAE can change attribute-relevant features while keeping other irrelevant details almost stationary using the inferred ær of input image.
4.6 Truncation-like Effect
we can obtain truncation-like effect in DPMs by scaling the strength of classifier guidance. We have assumed that Go (mr, 2, 1) trained by filling the posterior mean gap simulates the gradient of some implicit classifier, and it can actually work desired. In theory, it can
According to g 15
8

---

FID
Dataset
Model
223
T=10
T=20
T=50
T=100
FFHQ
DDIM
31.87
20.53
15.82
11.95
12.
Diff-AE
21.95
18.10
13.14
10.55
PDAE
20.16
17.18
12.81
10.31
Horse 555
DDIM
25.24
14.41
7.98
5.93
Diff-AE
12.66
9.21
7.12
5.27
PDAE
11.94
8.51
6.83
5.09
Bedroom 55
DDIM
14.07
9.29
7.31
5.88
18
Diff-AE
10.79
8.42
6.49
5.32
200
290
PDAE
10.05
7.89
6.33
5,47
CelebA
DDIM
18.89
13.82
8.48
Figure 9: The truncation-like like effect by scaling (a y. t) with 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0 respectively.
Diff-AE
12.92
10.18
7.05
5.30
for "|mageNet64-77M.--88MM
PDAE
11.84
9.65
7.23
5.19
Table 2: FID scores for unconditional sampling.
also be applied in truncation-like effect. To illustrate this, directly incorporate the class label into GT y. t) and train it to fill the gap. Specifically, we train model of 1amageNer64-77M-y-33B0 and use DDIM sampling method with 100 steps generate 50k samples. guided by the predicted mean shift with different seales for a truncation-like effect. Figure 9 shows the sample quality effects of sweeping over the scale. As that of classifier-gu guided sampling method, which helps to build connections between filling the posterior mean gap and classifier-guide guiieed sampling method. The gradient estimator trained by filling the posterior mean gap is an alternative to the noisy classifier.
see, it achieves the truncation-like effect similar to
4.7 Few-shot Conditional Generation
on CelebA 200 and aim to achieve conditional sampling given small number of labeled images. To achieve this, we train latent DPM p. (=.-i(
Following D2C 42 we train model of "CelebA64-722--2512-225
semantic latent space and a latent classifier pa (ylz) using given labeled
images. For binary scenario, the images are labeled by binary class (100 samples, 50 for each class).
For PU scenario, the images ei- ther labeled positive or unlabeled (100 positively labeled and 10k unla- beled samples). Then we sample z from (z)-2(1 and accept with the probability of (yl 2). We use the accepted generate 5k sam- ples for every class and compute the FID score between these samples and all images belonging to corre- sponding class dataset. We com- pare PDAE with Diff-At Diff-AE and D2C.
Table 3: FID scores for few-shot conditional generation using "CelebA64-22M-2512-881"
Scenario
Classes
PDAE
DII-AE
D2C [42
Naive
Male
11.21
11.52
13.44
25.70
Binary
Female
6.81
7.29
9.51
14.16
Blond
16.96
16.10
17.61
24.78
Non-Blond
8.13
8.48
8.94
1.12
Male
9.41
9.54
16.39
25.70
Female
8.97
9.21
12.21
14.16
PU
Blond
6.34
7.01
10.09
24.78
Non-Blond
7.17
7.91
9.09
1.12
We also use the naive approach that computes the FID score between the training images and the corresponding subset of images in dataset. Table 3 shows that PDAE achieves better FID scores than Diff-AE and D2C.
4.8 Improved Unconditional Sampling
As shown in Section under the help of z, PDAE can generate plausible images in only one step. If we get in advance, PDAE can achieve better sample quality than pre-trained DPMs in the same number of sampling steps. Similar to Diff-AE, we train a latent DPM on semantic latent space and sample from it improve the unconditional sampling of pre- pre-trained DPMs.
Unlike Diff-AE that must take as input for sampling, PDAE uses an independent gradient estimator as a corrector of the pre-trained DPM for sampling. We find that only using pre-trained DPMs the last few sampling steps can achieve better sample quality, which may because that the gradient estimator is sensitive to in the last few sampling steps and the stochasticity of sampled

---

will lead to out-of-domaing samples. Asyrp 27 also finds similar phenomenon. Empirically, we carry out this strategy in the last 30% sampling steps. We evaluate unconditional sampling result on 9FFHQ128-1300-25122
"Horse 1 28- 288-130--5512-- 1--6-MM.. "Bedroom 18-1120-255111 12--00M" and using DDIM sampling method with different steps. For each dataset, we calculate the FID scores between 50k generated samples and 50k real images randomly selected from dataset. Table 2 shows that PDAE significantly improves the sample quality of pre-trained DPMs and outperforms Diff-AE. Note that PDAE can be applied for any pre-trained DPMs as an auxiliary booster to improve their sample quality.
(CelebA64-72M-2512-BBB
Related Work
Our work based on an emerging latent variable generative model known as Diffusion Probabilistic Models (DPMs) 43 [14, which are now popular for their stable training process and competitive
sample quality. Numerous studies [34 24 g T 19 46, 30) and applications [3] 26 18 32 57
6
29
16 have further significantly improved and expanded DPMs.
Unsupervised representation learning via generative modeling is popular topie in computer vision. Latent variable generative models, such as GANs [13], VA (25 [9. and DPMs, are a natural candidate for this, since they inherently involve latent representation of the data they generate. For GANs, due to its lack of inference functionality, one have to extract the representations for any given real samples by an extra technique called GAN Inversion [54] which invert samples back
51 either have limited reconstruction quality or need significantly higher computational cost. VAEs explicitly learn representations for samples, but still face representation generation trade-off challenges 49 42)
into the latent space of trained GANs. Existing inversion methods [58 35 4
I 2
prassentttioongeeeratioono
VQ-VAE 449 37 and D2C [42 overcome these problems by modeling latent variables post-hoc in different ways. DPMs also yield latent variables through the forward process. However, these latent variables lack high-level semantic information because they are just sequence of spatially corrupted images. In light of this, diffusion autoencoders (Diff-AE) [66] explore DPMs for representation learning via autoencoding. Specifically, they jointly train an encoder for discovering meaningful representations from images and conditional DPM as the decoder for image reconstruction by treating the representations as input conditions. Diff-AE is competitive with the state-of-the-arr model on image reconstruction and capable of various downstream tasks. Compared with Diff-AE, PDAE leverages existing pre-traine pre-trained DPMs for representation learning also via autoencoding, but with better training efficiency and performance.
concurrent work with the similar idea the textual inversion of pre-trained text-to-image DPMs Specifically, given only 3-5 images of a user-provided concept, like an object or style, they learn to represent it through new "words" in the embedding space of the frozen text-to-image DPMs. These learned "words" can be further composed into natural language sentences, guiding personalized creation in an intuitive way. From the perspective of posterior mean gap, for the given new concept, textual inversion optimizes its corresponding new "words" embedding vector to find a best textual condition (c), so that which can be fed into pre-trained text-to-image DPMs (Ea(z,, c, t)) to fill as much gap (E- - eg (x,, 0, t)) as possible.
6 Conclusion
In conclusion, we present a general method called PDAE that leverages pre-trained DPMs for representation learning via autoencoding and achieves better training efficiency and performance than Diff-AE. Our key idea is based on the concept of posterior mean gap and its connections with classifier sampling method. A concurrent work, textual inversion of pre-trained text-to-image DPMs, can also be explained from this perspective. We think the idea can be further explored to extract knowledges from pre-trained DPMs, such interpretable direction discovery [51], and leave it as future work.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (Grant No. 62072397 and No.6 1836002), Zhejiang Natural Science Foundation (LR (LR19F020006) and Yiwise.
10

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432-4441, 2019.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegantt: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296-8305, 2020.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 34:17981-11 1-7993, 2021.
[4] David Bau, Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Inverting layers of a large generator. In ICLR Workshop, volume 2, page 2019.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009. 00713, 2020.
[6] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilv:: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.029) 2021.
[7] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. arXiv preprint arXiv:2204.0022 2022.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances Neural Information Processing Systems, 34, 2021.
[9] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[11] William Feller. On the theory of stochastic processes, with particular reference to applications. In Proceedings of the [First] Berkeley Symposium on Mathematical Staristics and Probability, volume 1, pages 403 403-433. University of California Press, 1949.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar Xiv:22220.00. ffv:22008006688, 2022.
[13] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[16] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fast- diff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204,0 09934,9944.55 2022.
[17] Rongjie Huang. Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Pro- gressive fast diffusion model for high-quality text-to-speech. arXiv preprint arXiv:2207.06388 2022.
[18] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint 10X.:22044014099 2021.
[19] Alexia Mrricoeur-Mattineau, Ke Li, Rémi Piché-Taillefer Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 2021.
[20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1XXx:11710010116,2007 2017.
11

---

[21]
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 1401-4410, 2019.
[22] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, pages 8110-8119, 2020.
[23] Diederik P Kingma and Prafulla Dhariwal. Glow; Generative flow with invertible Ixl convolu- tions. arXiv preprini arXiv: / /100/00000003 2018.
[24] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models.
arXiv preprint arXiv:2107 100,00000700000 2021.
[25] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint
1311v:1312.611 14, 2013.
[26] Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009. 0.000, 2020.
[27] Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 2201062200.10600 2022.
[28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324,, 1998. [29] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yucting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 2022.
[30] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
[31] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprini 1rXiv:2112.05744 2021.
[32] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[33] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, nya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini arXiv:2112. 10741, 2021.
[34] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[35] Guim Peramna, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint 16Xiv:6611.06355 2016.
[36] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn Diffusion autoencoders: Toward meaningful and decodable representation. arXiv preprint arXiv:2111. 15640, 2021.
[37] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images
with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[38] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In
International conference machine learning, pages 1530-1538. PMLR, 2015.
[39] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation
and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assiiee intervention, pages 234-241. Springer, 2015.
[41] Chitwan Saharia, Jonathan Ho. William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint (04::2004.07366 2021.
[42] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34, 2021.
[43] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
12

---

[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.022000 2020.
[45] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprini arXiv:201 1. 13346,, 2020.
[47] Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder, Advances in Neural Information Processing Systems, 33:19667-1967, 2020.
[48] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[49] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[50] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks.
In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[51] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning, pages 9786-9796. PMLR. 2020.
[52] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for im- age quality assessment. In The Thrity-Seventh Asilomar Conference Signals, Systems & Computers, 2003, volume 2. pages 1398-1400 leee, 2003.
[53] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[54] Weihao Xia, Yulun Zhang. Yujiu Yang. Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. arXiv preprint arXiv:2107. 200.00080., 2021.
[55] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1.50. 03365,, 2015.
[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[57] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. arXiv preprini arXiv:2 104. 104.03670, 2021.
[58] Yun Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In European conference on computer vision, pages 597-613. Springer, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasig new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "m68yJcLOE1d", "reviewer": "Reviewer_jf2c", "summary": "The paper presents a new learning framework for diffusion probabilistic models. Unlike the most existing diffusion probabilistic models, which mainly embed the data into a predefined distribution through the Markov chain, the proposed framework aims to learn a robust representation feature through a pre-trained diffusion models. The authors also introduce a weighting scheme redesign function to further improve the representation learning in the diffusion models. Extensive experiments are conducted on multiple scenarios (e.g. FFHQ, CelebA, Bedroom).", "questions": "- What's the key difference between the proposed framework with the latest state-of-the-art diffusion autoencoders (Diff-AE) [21]. While the authors consider Diff-AE as a baseline, the two works seem to be similar. The authors should clearly explain what is the difference between them.\n- Besides, The qualitative comparison to the Diff-AE baseline is missing, even after considering the results in the Appendix. What's more, the quantitative improvement in Tables 1, 2, and 3 is limited compared to the  Diff-AE baseline.\n- It takes me a hard time buying the $x_0\\to x_t$ in Figure 1. If I understand correctly, this should be a diffusion processing in general. Then, does this contain any pre-trained parameters?\n- Could please the authors illuminate the difference between these results shown in Figure 5? While the authors claimed to \"some stochastic variations\", it seems to be very hard to capture such a conclusion.", "limitations": "Limitations are not discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The task of representation learning via diffusion probabilistic models is novel and interesting.\n- The overall idea and the approach towards addressing it seems reasonable. \n\n### Weaknesses\n- The paper is not well written and hard to read.\n- The technical novelty is limited. Neither the designed network nor the theory are new. \n- The qualitative results are poor, and the quantitative improvement is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "d8JMWd3UVaQ", "reviewer": "Reviewer_4H5f", "summary": "The authors propose an unsupervised method for data representation and reconstruction, utilizing a pre-trained unconditional diffusion probabilistic model (DMP) in order to avoid the long training time of such models.\nThe method incorporates a gradient estimator, that learns the posterior mean shift, which is then used as a condition in order to guide the sampling process towards the image reconstruction. The authors also propose a new weighting scheme for training the objective function. Learning the gradient estimator encourages the encoder to learn meaningful data representation, and the sampling process to produce better data reconstruction.\nThe authors provide quantitative and qualitative evaluations showing improvements in representation learning, image reconstruction, and image sampling.", "questions": "- Line 16: \"and train it like a normal DMP\": this causes some confusion, as the DMP part of the model is fixed. Also, the objective function aims to learn image reconstruction rather than image generation.\n\n- It would be nice to see the results of the experiment shown in section 3.4, but when using the gradient-estimator as classifier-guided sampling, instead of a separate classifier, as done on MNIST.\n\n- In the experiment section, the prediction of x_0 is done from x_t in only one step. What does this \"one-step\" mean? Isn't the whole reverse process required in order to retrieve x_0?\n\n- It would be great to have an experiment that supports the claim that this method requires less training time than its competitors.", "limitations": "The author did not address their method's limitations.\nThe described work has no potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The problem is known to be valid and challenging, and the level of novelty is reasonable.\n\n- The method, idea, and motivation are well explained in the paper. The authors provided a good intuition about their proposed network architecture.\n\n- The authors well-explained their weighting method by describing their insights from an experiment that shows differences in the result with and without the classifier-guided sampling\n\nWeaknesses:\n\n- There is no algorithm that explains how the sampling step works in the author's method (how does the sampling-guided method is employed when using the model described in the paper).\n\n- I suggest showing Figures 1,2 in separate locations (figure 2 should be closer to the text that describes it, along with figure 3).\n\n- According to the quantitative results, the improvement of data representation is not significant. However, there is an improvement in the rest of the experiments, such as in unconditional sampling.\n\n- Overall proofreading is required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GD9yWj9EvuX", "reviewer": "Reviewer_eiJZ", "summary": "This paper presents a method of learning representations out-of pretrained unconditional diffusion models. Different from DiffusionAE which learns the encoder and condition DPM together, this paper proposes to keep a pretrained unconditional diffusion models unchanged, and learn the gradient of classifier guidance which takes a latent variable as the additional input, and the latent variable in encoded by a learnable encoder. Empirical results show that this method leads to better reconstruction and few-shot conditional generation results compared to DiffusionAE with less training expense. Qualitatively the method learns meaningful representations, and it can also improve the sample quality of unconditional diffusion models with a learned latent DPM on the latent variables. ", "questions": "- One baseline worthwhile to compare is DiffusionAE + classifier-free guidance. If the proposed method can beat this baseline, it will be more convincing that the improvement is given by the novel formulation $G_\\psi$, instead of the guidance formulation which is not the contribution of this paper. \n\n- For classifier(-free) guidance, it is known that results are better if the diffusion models are also conditional, i.e., the modified score is given by $\\nabla_{x_t} [\\log p(x_t|c) + \\omega \\log p(c|x_t)]$ instead of $\\nabla_{x_t} [\\log p(x_t) + \\omega \\log p(c|x_t)]$. I'm wondering if similar conclusion applies here. I.e., if you condition both the diffusion model and the classifier gradient $G_\\psi$ with the latent variables $z$, whether you can get better performance than the current formulation. ", "limitations": "No limitation and potential negative social impact are discussed . ", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- This paper is well written and easy to follow. It clearly states its connection with the previous close related work, DiffusionAE, and points out the advantage over the previous work. \n- The proposed method is technically sound and naturally combines learning representation and classifier guidance. \n- Various experiments are carried out and results are compared with baseline methods, making the method more convincing. \n\nWeaknesses: \n- IMO the novelty of this work is kind of limited. Compared to DiffusionAE, this work includes the classifier guidance and the latent codes are embedded into the learnable guidance term instead of the diffusion model, expect for which all other components remains the same as DiffusionAE, such as reversing DDIM sampler for latent code and learning latent DPM for z. Technically speaking I don't think there's a significant novel method proposed here.\n- Compared to DiffusionAE, this method includes guidance term so it is not quite surprising to me that it can outperform DiffusionAE. It's not clear to me whether the improvement is due to the guidance mechanism or due to the novel formulation $G_\\psi$. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6EmKglXS1z", "reviewer": "Reviewer_XByM", "summary": "Whereas prior work (DiffAE) uses a conditional DPM as the decoder in an auto-encoder setup, this work attempts to leverage pretrained unconditional DPMs instead. The proposed formulation extends the classifier-guidance solution for latent autoencoder semantic codes z. However, instead of explicitly training classifiers p(z | x_t), taking its corresponding score, they propose to directly fit to the posterior mean gap of the pretrained, unsupervised, and frozen DPM. Results demonstrate higher quality reconstruction than DiffAE, with an overall simpler training setup.", "questions": "- See weaknesses\n- How does the closeness between the pretrained DPM's domain and the target domain affect the efficacy of this method? If the pretrained DPM were from a completely unrelated domain, would this just mean that the posterior gap is larger and that the training time of this method would take much longer?", "limitations": "Limitations were not discussed. Example topics would include deep fakes etc.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n- paper was clearly written. Formulation was easy to follow\n- prior works using classifier guidance had some issues regarding the scale of the guidance vector being used, but I believe the proposed formulation gets around that issue by fitting to the posterior mean gap instead.\n\nweaknesses\n- In section 3.4, it's not clear how the start and end of the critical stage are determined.\n- L124 Except the latent code z (awkward phrasing)\n- One thing that would've been nice to see is whether it's truly necessary to use another DPM for modeling the generative distribution $z$. DiffAE justifies their choice of a DPM by stating that a VAE's objective would have been difficult to tune, but perhaps a simple auto-encoder with a post-fit GMM over the latent vectors might suffice?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept