Diffusion-Based Representation Learning
Authors: Korbinian Abstreiter
Stefan Bauer
Bernhard Schölkopf
Arash Mehrjou
Venue: ICLR 2022 Submitted (2022)

Abstract: Score-based methods represented as stochastic differential equations on a continuous time domain have recently proven successful as a non-adversarial generative model. Training such models relies on denoising score matching, which can be seen as multi-scale denoising autoencoders. Here, we augment the denoising score-matching framework to enable representation learning without any supervised signal. GANs and VAEs learn representations by directly transforming latent codes to data samples. In contrast, the introduced diffusion based representation learning relies on a new formulation of the denoising score-matching objective and thus encodes information needed for denoising. We illustrate how this difference allows for manual control of the level of details encoded in the representation. Using the same approach, we propose to learn an infinite-dimensional latent code which achieves improvements of state-of-the-art models on semi-supervised image classification. As a side contribution, we show how adversarial training in score-based models can improve sample quality and improve sampling speed using a new approximation of the prior at smaller noise scales.

Under review as a conference paper at ICLR 2022
DIFFUSION-BASED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion-based methods represented as stochastic differential equations on a continuous time domain have recently proven successful as non-adversarial generative model. Training such models relies on denoising score matching, which can be seen as multi-scale denoising autoencoders. Here, we augment the denoising score matching framework to enable representation learning without any supervised signal. GANs and VAEs learn representations by directly transforming latent codes to data samples. In contrast, the introduced diffusion based representation learning relies on a new formulation of the denoising score matching objective and thus encodes information needed for denoising. We illustrate how this difference allows for manual control of the level of details encoded in the representation. Using the same approach, we propose to learn which achieves improvements of state-of-the- art models on semi-supervised image classification. As side contribution, we show how adversarial training in diffusion- based models can improve sample quality and improve sampling speed using a new approximation of the prior at smaller noise scales.
fifinite-iimessionalll latent code
INTRODUCTION
Diffusion- based models have recently proven successful for generating images (Sohl-Dickstein et al. (2015): Song & Ermon (2020); Song et al. (2020)), ), graphs (Niu et al. (2020), ) shapes (Cai et al. (2020) ) and audio (Chen al. (2020b); Kong al. (202T)). Two promising approaches apply step-wise perturbations to samples of the data distribution until the perturbed distribution matches a known prior (Song Ermon (2019); Ho et al. (2020). ). A model is trained to estimate the reverse process, which transforms samples of the prior to samples of the data distribution (Saremi et al.
(2018)) Diffusion models were further refined (Nichol & Dhariwal (202T): Luhman & Luhman (2021) and even achieved better image sample quality than GANs (Dhariwal & Nichol (2021); Ho al. (202T): MMhrjou et al.] (2017)). Further. (Son et al.] showed that these frameworks are discrete versions of continuous-time perturbations by stochastic differential equations and proposed diffusion-basee generative modeling framework on continuous time. Unlike generative models such GANs and various forms of autoencoders, the original form of diffusion models does not come with a fixed architectural module that captures the representation.
Learning desirable representations has been an integral component of generative models such as GANs and VAEs (Bengio et al. 2013); Radford et al. (2016) Chen et al. (0166); van den Oord et al. (2017): Donahue & Simonyan 2019); Chen et al. (2020a); Schölkopf al. (2021)). Considering diffusion-based methods as promising and theoretically grounded generative models, here we propose method to augment the underlying Stochastic Differential Equation (SDE) for learning latent data-generating code. The key idea is to provide a representation of the clean data as additional input to the model estimating the score function. Our approach is illustrated in Figure [ Recent works on visual representation learning achieve impressive performance on the downstream task of classification by applying contrastive learning (Chen et al. (2020cp; Grill al. (2020 Chen & (202)); CCaonn et al. 22021). However, training the encoder to output similar representation for different views of the same image removes information about the applied augmentations, thus the performance benefits are limited to downstream tasks that do not depend on the augmentation, which has to be known beforehand. For this reason, we do not apply contrastive learning in order not to restrict the learned representation specific downstream tasks and solve more general problem instead. We provide a summary of contrastive learning approaches in A. Similar to our approach, Denoising Autoencoders (DAE) (Vincent al. 2008)) be used encode representations that

---

Under review as a conference paper at ICLR 2022
dx f(x, f) dt g(t) dwo
X1
Xg
Ex(xa)
(x;) Vx, log q(x, x
[x5, 2) V., log (xxlx))1
Denolsing score matching
Conditional score matching Representation learning
Figure 1: Conditional score matching with parametrized latent code is representation learning. Denoising score matching estimates the score at each T4; we add latent representation 2 of the clean data Zo as additional input the score estimator.
can be manually controlled by adjusting the noise scale (Geras & Sutton 2015); Chandra & Sharma (2014): Zhang Zhang (2018). Note that unlike DAEs, the encoder in our approach does not receive noisy data as input, but instead extracts features based on the clean images. For example, this key difference allows DRL be used to limit the encoding to fine-grainde features when focusing on low noise levels, which is not possible with DAES. Even though diffusion models are considered non-adversarial, in this work we show that it can actually be an adversarial process by means of the weighting function À which has positive effect in the sampling quality of the diffusion-based models.
We begin by briefly revisiting the foundations diffusion- generative models in Section 1.1 In Section we present method for representation learning, propose how to apply adversarial training in Eifuuiionbbsse models Section 3.1] and motivate the use of fewer sampling steps in Section 3.2 We follow up with experimental results and evaluations of our proposed methods in For further clarity, the main contributions of this work are itemized the following.
Section
We present an alternative formulation of the denoising score matching objective, showing that the objective cannot be reduced to zero.
We introduce and evaluate Diffusion-baad based Representation Learning (DRL), a new frame- work for representation learning in diffusion-based generative models. We show how this framework allows for manual control of the level of details encoded in the representation. code and evaluate it the downstream
We extend approach to dnfinite-dimensee task semi- semi-supervised image classification, improving satte-of-the-an approaches.
Unlike the widely admitted advesarrsalial nature of these models, we show there exists an inherent component in the formulation that acts adversarially and can be leveraged to improve the sample quality.
We evaluate the effect of the initial noise scale and achieve significant improvements in sampling speed, which is a bottleneck in diffusion-based generative models compared with GANs and VAEs, without sacrificing image quality.
1.1 DIFFUSION-BASE BASED GENERATIVE MODELING
In the following, we give brief overview of the technical background for the framework of the diffusion-based generative model, for example, as described in (Song et al., 2021b). The forward diffusion process of the data is modeled as SDE on continuous time domain / E [0, 7). Let To E IRd denote a sample of the data distribution To ~ Po, where the data dimension. The trajectory (zz ( of data samples is function of time determined by the diffusion process. The SDE is chosen such that the distribution Por (xr for any sample a known prior distribution. Notice that the subscript 07' of por refers to the conditional distribution of the diffused data at time T given the data at time 0. For simplicity we limit the remainder of this paper to the colcaleed Variance Exploding SDE (Song et al. (20216)), that is,
Po can be approximated by
d() dw, dt
dx f(x, t) dt g(t) dw :=
(1)
2

---

Under review as a conference paper at ICLR 2022
where is the standard Wiener process. The perturbation kemnel of this diffusion process has closed-form solution being ( = - (z Zo, (o) - o d0))))). was shown by Anderson (1982) that the reverse diffusion process is the solution to the following SDE:
dæ = [f(,,)) g (1) log (x)) dt g(t) dw,
where W is the standard Wiener process when the time moves backwards. Thus, given the score function V log (z)) for all t E [0, we can generate samples from the data distribution pa(x). In order to learn the score function, the simplest objective is Explicit Score Matching (ESM) (Hyvärinen Dayan (005))) that is,
Ex, [1150 1) - Vz, pe
Since the ground-truth score function V., log pt (1:z) is generally not known, one can apply denoising score matching (DSM) (Vincent 12 which is defined as the following:
JPSM (0) = Ez, [Eas/zo (2), - Vz, log Po (22 (20)2=))111)
(4)
The training objective over all t is augmented by SSnn et al. (202bb) with a time-dependent positive weighting function A(1), that is, J DSM (0) = E, [A()) [0)]
DIFFUSION-BASED REPRESENTATION LEARNING
2.1 ALTERNATIVE FORMULATION OF DENOISING SCORE MATCHING
We begin this section by presenting an alternative formulation of the Denoising Score Matching (DSM) objective, which shows that this objective cannot be made arbitrarily small. Formally, the formula of the DSM objective can be rearranged as
JPSM (0) = Emp Egg so [II s, log Por (Z) Ve, (z))
+ +1|s0(Ze, t) V., s, log (x))
The above formulation holds, because the DSM objective in equation 4 is minimized when Vay, se (xx, t) = Vx,
and differs from ESM in equation onlly by a constant (Vincent ( (201 Hence, the constant is equal to the minimum achievable value of the DSM objective. A detailed proof is included in the Appendix A.2). It is noteworthy that the first term in the rhs of the equation dodes not depend on the learned score function z, for every 1 [0, T]. Rather, it is influenced by the diffusion process that generates H, from zo. This observation has not been emphasized previously, probably because has no direct effect on the learning of the score function that is handeled by the second term in the rhs of equation 5 However, the additional constant has major implications for finding other hyperparameters such as the function A(t) and the choice of (t) in the forward SDE. To the best of our knowledge, there is no known theoretical justification for the values of (t). While these hyperparameters could be optimized in ESM using gradient-based learning, this ability is severely limited by the non-vanishing constant in equation 5 Similarly, it impacts the behaviour of adversarial training in diffusion models, which we analyze in Section 4.2
Even though the non-vanishing constant in the denoising score matching objective presents a burden in multiple ways such as hyperparameter search and model evaluation, it provides an opportunity for latent representation learning, which will be described in the following sections.
2.2 CONDITIONAL SCORE MATCHING
Class-conditional generation can be achieved in diffusion-based models by training an additional time-dependent classifier Pe (yl xx) (Song et al. (2021b)). In particular, the conditional score for a fixed y be expressed as the sum of the unconditional score and the score of the classifier, that is, Va, log (z ly) = V Vz, log px + Va. log (y xz). We propose conditional score matching as an alternative way to allow for controllable generation. Given supervised samples (ar, y())), the new training objective for each time t becomes
J JCSM
Exz (Em. [2), y(zu)) V., log Pox (zy 11)
(6)
The objective in equation 6 is minimized if and only the model equals the conditional score function
N. log pa (xx (x0) ú) for all labels
3

---

Under review as a conference paper at ICLR 2022
2.3 LEARNING LATENT REPRESENTATIONS
Since supervised data is limited and rarely available, we propose to learn a labeling function y(z)) at the same time as optimizing the conditional score matching objective in equation 6 In particular, we represent the labeling function as a trainable encoder Egg : IR -> Re, where (x( maps the data sample zg to its corresponding code in the e-dimensional latent space. The code is then used as additional input to the model. Formally, the proposed learning objective for ffffusion-basee Representation Learning (DRL) the following:
JDRI,(0, (0) E..o..,
1, - Vz, log
To get a better idea of the above objective, we provide an intuition for the role of (x() in the input of the model. The model so("...) :)= : R x R x RR + IR is a vector-valued function whose output points different directions based on the value of its third argument. In fact, Ex selects the direction that best recovers Z from Zt. Hence, when optimizing over Φ, the encoder learns to extract the information from zu in a fedluced-dimeesione space that helps recover zy by denoising It. Notice that finding the denoising direction requires information from both 2o and 2; and Es can only extract the partial information from the source Zn-
We show in the following that equation is valid representation learning objective. The score of the perturbation kernel V.,, log Por ( is a function of only t, Ht and rg. Thus the objective can be reduced to zero if all information about 2o is contained in the latent representation (z())) When (xo) has no mutual information with zo, the objective can only be reduced up to the constant in equation 5 Hence, our proposed formulation takes advantage of the non-zero lower-bound of equation 5 which can only vanish when data information is distilled in a code provided as input to the model. These properties show that equation 7 is valid objective for representation learning.
Our proposed representation learning objective enjoys the continuous nature of SDEs, a property that is not available in many previous representation learning methods (Radford et al. (2016); Chen et al. Locatello et al. (2019)). In DRL,, the encoder is trained to represent the information needed to denoise wo for different levels of noise (l).. We hypothesize that by adjusting the weighting function A()). we can manually control the granularity of the features encoded in the representation and provide empirical evidence as support. When / - T that associated to high noise level, the mutual information of Ie and To starts to vanish, thus denoising requires all information about xg to be contained in the code. In contrast, when t + 0 that corresponds to low noise levels, I, contains coarse- -aanine features of za and only fine-graines properties may have been washed out due to the small magnitude of noise. Hence, the representation learns to keep the information needed to recover these fine-grained details. We provide empirical evidence to support this hypothesis in Section 4
is noteworthy that Eg does not need to be a deterministic function. In principle, it can be viewed as information channel that controls the amount of information that the diffusion model receives from the initial point of the diffusion process. With this perspective, any deterministic or stochastic function that can manipulate (0),,0)). the mutual information between Zo and 2t, be used. This opens up the room for stochastic encoders similar to VAEs that we call Variational Diffusion-based Representation Learning (VDLR) from this point onward. The formal objective of VDRL is
JVDRL.(0, (0, =Ee, [E (2 m (O))sss (Ie,, 222 Va, log Poe (o (1011)
(8)
+ (E)
2.4 SFRNITE-DIMESSINNA REPRESENTATION OF FINITE-DIMENSIONAL DATA
We now present an alternative version of DRL where the representation is function of time. Instead of emphasizing on different noise levels by weighting the training objective, as done in the previous section, we can provide the time /. input to the encoder. Formally, the new objective is the following:
Eg, [X(t)]sso (2,, t, Es t)) Va, log Por (zx [1]]
(10)
where (z(g in equation iss replaced by Ex t). Intuitively, it allows the encoder to extract the necessary information of zn required to denoise 2, for any noise level. This can be seen as a rich representation learning in the following way. Normally in autoencoders other static representation
4

---

Under review as a conference paper at ICLR 2022
learning methods, the imput data zo E R' is mapped to single point E R in the code space. We propose a richer representation where the imput zo is mapped to a curve in R instead a single point. Hence, the learned code is produced by the map 2o (Eo(zo, where the infinite-dimensional object (Eo t)) is the encoding for xo.
Proposition For any downstream task, the rnfiniteddimnnsinn using the objective in equation 70 is at least as good as Enite-dimensien static codes learned by the reconstruction of xn-
code (Zo, tE(o.T) learned
Proof sketch. The score matching objective can be seen as a reconstruction objective of zo condi- tioned on The terminal time I' is chosen large enough so that 2T is independent of Zp. hence the objective for t T is equal to reconstruction objective without conditioning. Therefore, there exists E [0, 7] where the learned representation ( E) is the same representation learned by the reconstruction objective of a vanilla autoencoder.
The full proof for Proposition ! can be found in the Appendix (T). A downstream task can leverage this rich encoding in various ways, including the use of either the static code for fixed I, or the use of the whole trajectory (Fog( ()) as input. We posit the conjecture that the proposed rich representation is helpful for downstream tasks when used for pretraining, where the value of t is either a model selection parameter or jointly optimized with other parameters during training, and evaluate it on semi-supervised image classification in Section 4.1.1 The current state-of-the-art model for many semi-supervised image classification benchmarks is LaplaceNet (Sellars et al. (2021), )). alternates between assigning pseudo-labed to samples and supervised training of classifier. The key idea is to assign pseudo-labels by minimizing the graphical Laplacian of the prediction matrix, where similarities of data samples are caleulated on a hidden layer representation in the classifier. Note that LaplaceNet applies mixup (Zhang et al. (2017)) that changes the input distribution of the classifier. We evaluate our method both with and without mixup on CIFAR- (Krizhevsky al. CIFAR-100 100 (Krizhevsky et al. (b)) and MinilmageNet (Vinyals et al. (0066)).
REMARKS ON DIFFUSION- BASED GENERATIVE MODELS
The following sections contain side-contributions on training and sampling in diffusion-basee models. First we want to note that diffusion models are claimed to minimize KL-Divergence when using the respective likelihood weighting (Song et al. (2021a)) However, the assumption of the model being curl-free is usually not enforced and thus this statement might not hold in practice.
3.1 ADVERSARIAL TRAINING IN DIFFUSION- -BASED GENERATIVE MODELS
In general, diffusion-baart based generative models enjoy the advantages of mon-adversa training. Hence, they do not suffer from mode collapse, a common problem observed in GANs (Thanh-Tung Tung et al. (2018)) but instead can be trained to minimize KL-Divergence. However, it has been shown that KL-Divergence is not a good indicator of perceptual image quality (Theis et al. (2016); /aays et al. (2017); Arjovsky et al. (2017). Hence, GANs were extended to maximize the variation lower bound of the family of f-divergences (Nowozin et al. 2016)). Similarly, (Song al. introduced the set of X-divergences between two probability distributions p and q, defined as
Dx(elg) = = / E Ex-pp( (A () lo (x) Vz (z)))
(11)
Note that X-divergences can express any f-divergence and that we can train on the respective diver- gence solely by adjusting A(z,t) t) in the denoising score matching objective. However, minimizing specific f-divergennce requires the knowledge of the ratio of densities (x))/qq(e) (z) that is not at hand.
Previous work on adversarial training in diffusion models added discriminator loss training the model to output visually appealing samples after each denoising step (Jolicoeur Marteerr--aainnau et al. (2020)). ). In contrast, we form min-max game to minimize the worst-case X-divergence defined by an additional adversary, which trained alternately with the model. Since we do not have access to the score function Vz, log (e.),, we approximate the X-divergence using the DSM objective, and
5

---

Under review as a conference paper at ICLR 2022
hence solve the following optimization problem:
1
min max 2 Ezo (Essaao [z aI log poe - Va, log px
(12)
+ [[5g((e,))) 1) Vx, (xx) dt.
As result, À is biased by the non-vanishing constant and might not explicitly focus on regions where the model is distant from the true score function. We hypothesize that the denoising score matching approximation can still yield the improvements of an adversarial divergence, and report the empirical evidence to support it in Section 4.2 For simplicity, we limit À to the set of linear functions on é, thus removing the dependence on x. Details are described in the Appendix in A.8 In order to prevent omission of any values of t in the training. we interpolate between training on the DEL and D2 objective with a hyperparameter PKL which determines the percentage of training on DKL.
3.2 THE CHOICE OF INITIAL NOISE SCALE
The initial noise scale controls the quality and diversity of the generated samples. It is proposed by Song & Ermon 2020) that the initial noise scale has to be chosen so that the sampling trajectory can traverse from every mode the other one. Even though this looks good in terms of diversity, it can be quite wasteful because the noise must be large enough so that the whole empirical data distribution covered. In fact, it is not necessary to traverse from every mode every other mode directly. We can traverse from a mode to a number of nearest modes and then from those modes to the farther ones. This way, we can choose a significantly smaller initial noise scale that saves us many steps in the sampling trajectory.
According to Song & Ermon 2020). o (T') should be set numerically equal the maximum pairwise distance of images, which is approximately 50 for CIFAR-I0 training images. However, we noticed that smaller values of a (T) are sufficient for generating diverse images. When using smaller initial noise scales, we can further approximate the noise distribution using the sum of gaussian noise z (0, (T)) and an additional uniform random variable in the image domain u ~ U[[0, 1) ensuring the generated images cover the whole image domain. We evaluate qualitative diversity and FID of generated images for various initial noise scales in Section 4.3
RESULTS
4.1
DIFFUSION-BASED REPRESENTATION LEARNING
For all experiments, we use the same function (l), L E [0, 1] as in Song et al. 2202 which is att) 7min (amax/ / /ii)) where amin 0.01 and max = 50. Further, we use 2d latent space and set A(t) = 0² (t), which has been shown to yield the KL-Divergence objective (Song et al. (2021a)) Our goal is not to produce state-of-the-as image quality, rather showcase the representation learning method. Because of that and also limited computational resources, we did not carry out an extensive hyperparameter sweep (cf. A.5 for details). Note that all experiments were conducted single Tesla V100 GPU, taking up to 30 hours of wall-clock time, which only amounts to 15% of the iterations proposed in Song al. 26216) We illustrate how the representation encodes information for denoising in Appendix in Figure
We first train a DRL model with LI Lerreullarization on the latent code on MNIST (eCunn & Cortes (2010)) and CIFAR- 10. Figure[ishwws shows samples from a grid over the latent space and a point cloud visualization of the latent values 2 = (zo) For MNIST, we can see that the value of 21 controls the stroke width, while 22 weakly indicates the class. The latent code of CIFAR-10 samples mostly encodes information about the background color, which weakly correlated to the class. The use of probabilistic encoder (VDRL) leads to similar representations (cf. 6 We further want to point out that the generative process using the reverse SDE involves randomness and thus generates different samples for single latent representation. The diversity of samples however steadily decreases with the dimensionality of the latent space, which is empirically shown in Figure of the Appendix.
Next, analyze the behavior of the representation when adjusting the weighting function A(t) to focus on higher noise levels, which can be done by changing the sampling distribution t. To this end, sample & E [o, 1] such that (() () is uniformly sampled from the interval [amin; mmax] = [0.01, 50).

---

Under review as a conference paper at ICLR 2022
3
5
5
5
0
3
6
6
9
y 4 9
7
4 7 7
9
7
1
(a) Generated samples
(b) Latent codes
(c) Generated samples
(d) Latent codes
Figure 2: Results of DRL. model trained on MNIST (a-b) and CIFAR-10 (c-d) using uniform
sampling of t. Samples generated from grid of latent values ranging from -1 to 1. The point clouds visualize the latent representation of test samples, colored according to the digit class.
5
5
3
8
5
4
5
2
}
7
A
4
d
0 0
5
6
0
6 0
60
20
(a) Generated samples
(b) Latent codes
(c) Generated samples
(d) Latent codes
Figure 3: Results of a VDRL model trained with a focus on high noise levels on MNIST (a-b) and CIFAR-10 (c-d). Samples generated from grid of latent values ranging from -2 to 2. The point clouds visualize the latent representation of test samples, colored according to the digit class.
Figure shows the resulting representation of VDRL (ef. Fig. for DRL results). As expected, the latent representation for MNIST encodes information about classes rather than fine-grained features we can control the granularity of features encoded in the latent space. For CIFAR-10, the model again only encodes information about the background, which contains most information about the image. A detailed analysis of class separation in the extreme case of training on single timescales is included in the Appendix (A.4).
such as stroke width. This validates our hypothesis of Section [23 2.3tion2.33tarr
Overall, the difference in the latent codes for varying X(t) shows that we can control the granularity encoded in the representation of DRL. While this ability does not exist in previously proposed models for representation learning, it provides significant advantage when there exists some prior information about level detail that we intend to encode in the target representation.
4.1.1 APPLICATION TO SEMI-SUPERVISED IMAGE CLASSIFICATION
representation (Es t) 0,71 on semi- supervised image classification, where we use DRL and VDRL as pretraining for the LaplaceNet classifier. Table depicts the classifier accuracy on test data for different pretraining settings. Details for architecture and hyperparameters are described in A.7 Note that we do not apply any of the methods proposed in Section 3
In the following, we evaluate the
Our proposed pretraining using DRL significantly improves the baseline and often surpasses the state-of-the-ard performance of LaplaceNet. Most notable are the results DRL and VDRL. without mixup, which achieve high accuracies without being specifically tailored to the downstream task of classification. Note that pretraining the classifier as part of an autoencoder did not yield any improvements (cf. 6). Combining DRL with mixup yields inconsistent improvements, results reported Table 7 In addition, DRL pretraining achieves much better performances when only limited computational resources available (cf. 5)
We further evaluate the infinite dimensoaiimesiooall representation on few-shot image classification using the representation different timescales input. The detailed results are shown in the Appendix (A.6).
7

---

Under review as a conference paper at ICLR 2022
LaplaceNet None No
LaplaceNet None
Ours DRL
Ours DRL
Ours
Pretraining Mixup
VDRL
Yes
No
Yes
No
Dataset
#labels
CIFAR-10
100
73.68
75.29
74.31
64.67
81.63
500
91.31
92.53
92.70
92.31
92.79
1000
92.59
93.13
93.24
93.42
93.60
2000
94.00
93.96
94.18
93.91
93,96
4000
94.73
94.97
94.75
95.22
95.00
CIFAR-100
1000
55.58
55.24
55.85
55.74
56.47
4000
67.07
67.25
67.22
67.47
67.54
10000
73.19
72.84
73.31
73.66
73.50
20000
75.80
76.07
76.46
76.88
76.64
MinilmageNet
4000
58.40
58.84
58.95
59.29
59.14
10000
66.65
66.80
67.31
66.63
67.46
Table 1: Comparison of classifier accuracy in % for different pretraining settings. Scores better than the SOTA model (LaplaceNet) are in bold. "DRL" pretraining is our proposed representation learning, and "VDRL" the respective version which uses probabilistic encoder.
PKL
0.05
0.5
0.95
FID 4 IS f
3.40
3.36
3.33
3.37
3.62
9.73
9.77
9.79
9.50
9.44
Table 2: FID and Inception Score for different interpolations between maximum likelihood training and training based on the adversarial X. PK = 1.0 corresponds to original training.
In summary, the representations of DRL and VDRL achieve significant improvements compared that of an autoencoder or VAE for several values of / and are similar for close to 1.
Overall the results align with the theoretical foundation in Proposition that the rich representation of DRL is at least as good as the static code learned using reconstruction objective. It further shows that in practice, the fimiite-dimesiionll code is superior the static representation for the application to downstream tasks by significant margin.
4.2 ADVERSARIAL TRAINING IN DIFFUSION BOSSED GENERATIVE MODELS
We evaluate our approach of optimizing the adversarial A-Divergence on the task of synthetic image generation of CIFAR-I0 images. We evaluate our approach for different values for PKL. which is used to interpolate between the original and adversarial training objective. The resulting FID and Inception Scores are displayed in Table The results show that for any value of PKl S 0.95, the sample quality is improved significantly. However in contrast to our intentions, the adversary converges to extreme value of (cf. 14). While our adversary learns put focus on values of / where the model is bad, the model apparently cannot improve in this region (cf.
in the first 10k iterations and does not change afterwards
Overall, the higher image quality after training with the adversary indicates that diffusion models can be improved using adversarial training. Since the loss is not significantly reduced, more complex adversaries in Hardy al. (2018) will only have the same effect predefined X. As shown in experiments, there is much room for improvement by searching for a good X. Furthermore. additional improvements might be achieved in the future by introducing inductive biases allowing the model to capture more high-frequency information, thus being able to reduce the loss for small values i.
4.3 THE CHOICE OF INITIAL NOISE SCALE
In the following, we evaluate image quality and diversity for different initial noise seales. Note that do not change a (T)), but instead evaluate generated images for different initial times livits which
8

---

Under review as a conference paper at ICLR 2022
Gaussian
Uniform Gaussian
tinst
(tiintt)
FID І
FID |
0.5
0.71
218.95
25.02
0.6
1.66
75.11
5.15
0.7
3.88
12.57
2.98
0,8
9.10
3,05
2.99
0.9
21.33
2.97
2.94
1.0
50.00
3.01
2.99
Table 3: FID for different initial noise scales evaluated on 20k generated samples.
(a) finst 0.5 (b) linc 0.6
(c) finst 0.7
(d) tinst 0.8 (e) finit 0.9 (f) tinet 1.0
(g) limit 0.5 (h) limit 0.6
finit 0.7 finit 0.8
(k) timit 0.9
() finit 1.0
Figure 4: Generated image samples for differrnt values of tinie- Top row ((a)-(f) uses the gaussian prior, bottom row ((g)-()) uses the version with an additional uniform random variable in the prior.
implicitly define the initial noise scale (f This reduces the number of sampling steps per image, which is 1000 finit and thus directly proportional to finit. Table shows the FID of generated images for various values of tsitt- As we can see, the first 200 sampling steps can safely be replaced by approximating the prior directly either with the gaussian the additional uniform distribution. Interestingly, using the sum of the uniform and gaussian random variables as a prior seed leads to improved image quality. This approximation for (1.)(1) allows steps by 30% without sacrificing image quality, which is further supported by the visual quality of generated samples shown Figure 4 Further, note that FID occasionally lower for values of finit < 1.0 than for fiit = 1. This suggests that to these timescales, our prior approximates the distribution better than the diffusion model when starting at finit 1.0.
reduce the number of sampling
CONCLUSION
We presented Diffusion-based Representation Learning (DRL), new objective for representation learning based on conditional denoising score matching. In doing so, we turned the original non- vanishing objective function into one that can be reduced arbitrarily close zero by the learned representation. We showed that the proposed method learns interpretable features in the latent space. In contrast to previous approaches, denoising score matching as foundation comes with the ability to manually control the granularity of features encoded in the representation. We demonstrated that the encoder can learn to separate classes when focusing on high noise levels and encodes one-grained features such as stroke-widtt when mainly trained on low level noise. In addition, we proposed an infiniteddimensional representation and demonstrated its effectiveness for downstream tasks such as few-shot classification. Using the representation learning pretraining for classifier, were able to improve the results of LaplaceNet, state-of off-the-art model on superviee image classification. As iide-contributions improve sample quality and were able to increase sampling speed using an approximation of the density smaller noise scales.
further showed how adversarial training in cefffusion-basee models can

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility, the code used to run all experiments is attached as supplementary material and will be published based upon acceptance.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models.
Stochastic Processes and
their Applications. 12(3):313-326,
1982. ISSN 0304-4149.
doi: https://doi.org//00
1016/0304-4144188)99003333 URL
Btticle/pL/330441122906
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3798) 2013.
Jane Bromley, James Bentz, Leon Bottou, Isabelle Guyon, Yann Lecun, Cliff Moore, Eduard Sackinger, and Rookpak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7:25, 08 1993, doi:
8001493000339.
Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation, 2020.
Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
B. Chandra and Rajesh Sharma. Adaptive noise schedule for denoising autoencoder. volume 8834, pp. 535-542, 11 2014. ISBN 978-3-319-12636-4, doi: 10.1007/978-3- --77778---1-----1 12637-1.67.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR, 2020a.
Nanxin Chen, Yu Zhang, Heiga 7en, Ron J. Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi supervisee learners. CoRR, abs/2006. 10029, 2020c. URL
hhttps://aratt org/ abs 2006. 10029
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets, 2016.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. 2019.
Leon A. Gatys, Alexander S. Feker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Con- trolling perceptual factors in neural style transfer, 2017.
Krzysztof J. Geras and Charles Sutton. Scheduled denoising autoencoders, 2015.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. Md-gan: Multi-discriminator generative adversarial networks for distributed datasets. CoRR, abs/1811.03388 2018. URL http://
arxiv. org. abs www.b////........ 03850
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. CoRR, abs/2006.1 1239, 2020. URL btL.http:: bttaps://ariive htwpa//wwww.rrr///////d///// 11220066.111111 11239

---

Under review as a conference paper at ICLR 2022
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282 2021.
Aapo Hyvärinen and Peter Dayan, Estimation of non- nonmoozall statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Alexia Jolicceur-Martinaau, Rémi Fiché-Taillefer Rémi Tachet des Combes, and loannis Mitliagkas. Adversarial score matching and improved sampling for image generation. CoRR, abs/2009.055 05475, 2020. BRLbttps. //axiiv arxiv. org
abs /ttpg/bbb//000055222
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). a. URLhttp: www. CS www.otoooooooooooooo edu/ krizz cifar. html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. URL htt.:tttt://wtt CS toronto. cetrrootto.dddd//e vcdu/-krize /ttrii/ccfae far.farttmm far.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:://ann
lecun. cww/
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021.
Arash Mchrjou, Bernhard Schölkopf, and Saced Saremi. Annealed generative adversarial networks. arXiv preprint 1XXYv:7700.075000 2017.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. CoRR, abs/2102.09672, 2021. URL htK///ttpss///xxxx.orr////////// 09672
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score-based generative modeling, 2020.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization, 2016.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830. 2011.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.
Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster
analysis. Journal of Computational and Applied Mathematics, 20:53-65, 1987. ISSN 0377-0427. URL https: www.wwwwww..ooo sci wcw.nccieencctt.....c.....cce...
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv:1805.083366 2018.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEFF, (115)6612-344 2021.
12

---

Under review as a conference paper at ICLR 2022
Philip Sellars, Angelica I. Avilés-Rivee Rivero, and Carola-Bibiane Schönlieb. Laplacenet: A hybrid energy-neural model for deep semi- classification. CoRR, abs/2106.04527, 2021. URL https://arxiz org/ abs 2106. 04527
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. CoRR, abs/ (ass/997.00600, 2019. URL
05600
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. CoRR, abs/2006.09011, 2020. URL.nttps://arr....rrrr//bbbb/b/
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models, 2021a. /oRL:/ttpae//r...oooo//////////
58v
Yang Song. Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021b.
Hoang Thanh-Ture Tung, Truyen Tran, and Svetha Venkatesh. On catastrophic forgetting and mode collapse in generative adversarial networks. CoRR, abs/1807.04015, 2018. URL http://
www.lv.orggabb/118e88 04015
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models, 2016.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. CoRR, 0bbs/1711.0099 2017.
00937
Pascal Vincent. A connection between matching and denoising autoencoders. Neural Computa- tion, 1317):1661-1677, 201 1, doi: 10. 10.1162/NECO a 00142.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference Machine Learning, ICML '08, pp. 1096-1 1103, New York, NY, USA, 2008. Associ- ation for Computing Machinery. ISBN 978 178555258, doi: 10. 10.1115//39015.111 13902994 URL
wtt.e//.0011111111333311/01 01.....1111111111111 1390294
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. CoRR, abs/1606.040000, 2016. URL htttht//////x
erq/abs/ /aobbb//666... 04080
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. CoRR, abs/1710.09412,, 2017. URL
09412
Qianjun Zhang and Lei Zhang. Convolutional adaptive denoising autoencoders for hicrarchical feature extraction. Front, Comput. Sci., 12(6): dec 2018. ISSN 2095-2228, doi: 10. 1007/8177044006-610777,00 URL htthhttts///dee /oii /
01000000--6-1111111111111111--11 6107-0
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 RELATED WORK ON CONTRASTIVE LEARNING
The core idea of contrastive learning is to learn representations that are similar for different views of the same image and distant for different images. In order to prevent the collapse of representations to a constant, various approaches have been introduced. SimCLRv2 directly includes a loss term repulsing negative image pairs in addition to the attraction of different views of positive pairs (Chen et al. (020c ). In contrast, B YOL relies solely on positive pairs, preventing collapse by enforcing similarity between the encoded representation of an image and the output of a momentum encoder applied to a different view of the same image (Grill et al. (2020)) An additional approach relies on online clustering and was proposed in SwAV (Caron al. (202T)) Training in SwAV is based on enforcing consistency between cluster assignments produced for different views of an image. Each
of these methods rely on the foundation of Siamese networks (Bromley et al. [993] ). which were
shown to be competitive for unsupervised pretraining for classification networks on its own when including a stop-gradient operation on one of the branches (Chen & He (020
A.2 DENOISING SCORE MATCHING
The following is the proof for the new formulation of the denoising score matching objective in equation 5
Proof. It shown by Vincent (2011) that equation 4 is equal explicit score matching up to constant which is independent of 0, that is,
Exp Ex (115 t) Va, log (x) EEr, (2),) - Vz, log pe (xx) +c.
(13)
(14)
As a consequence, the objective is minimized when the model equals the ground-truth score function se (2x,,) t) Vx (zr). Hence we have:
Ezo Egg 8o ||| 2 log):(Ft) - V.. log (2aa
(15)
Ez, [I] [ 0, (x - Vx, po +c
(16)
(17)
Combining these results leads to the claimed exact formulation of the Denoising Score Matching objective:
J PSM (0) Eno (Es, Imo (11,,( Va, log 1z Ez, [] 1) - V,, lo ( +c
(18)
(19)
Em, [11s0(21,, 1) - V., log [1
(20)
Enp (E png |ll V., lo - Vz. log par (111=0)
-E., (E,, Imo V (og V.. log (
(21)
+ [Is0(2r;;)) V., lx:))
A.3 REPRESENTATION LEARNING
Here we present the proof for Proposition]] 1 stating that the infinite-dine dimensional code learned using DRL is at least as good as static code learned using a reconstruction objective.
Proof. We assume that the distribution of the diffused samples time t T matches known prior (r( That is, p( p(2)))or (2p) dxy = Pr (zT). In practice is chosen such that this assumption approximately holds.
14

---

Under review as a conference paper at ICLR 2022
Now consider the training objective in equation 10 time T, which can be transformed to reconstruction objective the following way:
X(T) |] T, (s (ico..T) T)) - V
por ( Por
1 (zO)
(22)
2
= = E E Eprepp(er)
(sooE, T, (T(
2o - 2T
(23)
02(T)
=X(T)o (T) E E ExyepT(ar)
(D (T), x 20122]
(24)
=X(T)o- [Doo(Fs(Too so moll2]]
(25)
where replaced the model with Decoder model so (3T,, T, E ( 7)) Do (mn. (T)))aa
a²(T)
and replaced the score function of the perturbation kernel Ver log Par Pur (IT (zy) with its known closed- form solution #o-eT determined by the Forward SDE in equation Hence the learned code at time (T() 1' is equal to code learned using a reconstruction objective.
We model a downstream task as minimization problem of a distance d Ω x Ω2 + R in the feature space Ω between the true feature extractor g : R - Ω which maps data samples zg features space S and a model feature extractor he R - Ω doing the same given the code as input. The least as good as the static code:
following shows that the cefiniteddimensinne representation
inf min Ess, (((hh(EE(tt))) g(x))) s min Ezg [d( (egth(EEgg (zo, T)). g(xo))]
(26)
10
9
7
5
3
3
3
4
1
4
6
5
2
8
9
/
7
7
1
5
5
0
3
1
/
7
9
6
3
5
3
22
1/7
9
9
1
3
6
6
0
//7
7
9
6
2
0
3
05
-2
1 17
1775
9
8
3
0
0
77
4
9
8
083
7 4
7
8
0
8
6
-2
0
2
9
6
6
0
8
Z1
(a) Samples generated from grid latent values range from -2 to 2
(b) Latent representation test samples, colored according to the digit class
Figure 5: Samples and latent distribution model trained on MNIST using KL-divergence and uniform sampling of
A.4 TRAINING ON SINGLE TIMESCALES
To understand the effect of training DRL on different timescales more clearly, we limit the support of the weighting function A(t) to a single value of f. We analyze the resulting quality of the latent representation for different values of using the silhouette score with euclidean distance based on the dataset classes Rousseeuw (987), It compares the average distance between point to all other points in its cluster with the average distance to points in the nearest different cluster. Thus we measure how well the latent representation encodes classes, ignoring any other features. Note that
15

---

Under review as a conference paper at ICLR 2022
3
1
22
0
-2
-3
-2
Z1
(a) Samples generated from grid of latent values range from -1 to
(b) Latent representation of test samples, colored according the class label
Figure 6: Samples and latent distribution of VDRL model trained CIFAR-10 using uniform sampling of
8
8
8
8
3
9
0
8
8
3
774
4
8
3
2
22
9
7
4
4
8
3
6
0
9 7
7
9
4
8 4
2
6
6
6
-1
1
4
T
4
9
-
6
6
6
-2
1 7
7
7
5
6
6
6
-2
0
2
6
Z1
(a) Samples generated from grid of latent values range from -1 to
(b) Latent representation of test samples, colored according to the digit class
Figure 7: Samples and latent distribution of a DRL model trained on MNIST using uniform sampling of (focus on high noise levels).
after learning the representation with different distribution of it is necessary to perform additional training with uniform sampling of and frozen encoder to achieve good sample quality.
Figure II shows the silhouette scores of latent codes of MNIST and CIFAR-10 10 samples for different values of In alignment with our hypothesis of Section 2.3 training DRL on small and thus low noise levels leads to almost the opposite is the case for a range of t which differs between the two datasets. The decline in encoded class information for high values of can be explained by the vanishing difference between distributions perturbed samples when t gets large. This shows that the distinction among the code classes represented by the silhouette score is controlled by A(t)-
encoded class information in the latent representation, while
16

---

Under review as a conference paper at ICLR 2022
Z2
-2
-5.0
-2.5
Z1
(a) Samples generated from grid of latent values range from -1 to
(b) Latent representation of test samples, colored according the class label
Figure 8: Samples and latent distribution of DRL model trained CIFAR-10 using uniform sampling of (focus on high noise levels).
2
3
0
1
2
1
1
1
1
2
S
f
3
Figure 9: Samples generated starting from 2; (left column) using the diffusion model with the latent code of another To (top row) input. It shows that samples denoised correctly only when conditioning on the latent code the corresponding original image Zp-
A.5 ARCHITECTURE AND HYPERPARAMETERS
The model architecture we use for all experiments is based on "DDPM++ cont. (deep)" used for CIFAR-10 10 in [Song et al. (0021b). is composed of downsampling and upsampling block with residual blocks at multiple resolutions. We did not change any of the hyperparameters of the optimizer. Depending resolution, and the number of residual blocks per resolution in order to reduce training time.
the dataset, we adjusted the number of resolutions, number of channels per
For representation learning, of the model, followed by another three dense layers mapping to a low dimensional latent space. Another four dense layers map the latent code back to higher-dimensional representation. It is then
encoder with the same architecture the downsampling block
17

---

Under review as a conference paper at ICLR 2022
8
3
8
2
6
0
8
5
0
9
7
7
9
9
7
4
7
7
7
2
2
8
2
8
2
3
0
8 8
9
7
9
7
9
9
3
6
2
6
3
8
8
8 8
77
9
9
9
9
9
9
7
7
9
3
3
3
2
8
0
2
8
9
9
7
9
7
9
9
9 7 9 9 9
8
0
8
3
2
8
6
8 6
9
999
9
7
9
9
9
7
9
3
0
8
8
5
3
2
8
9 2
0
9
9
9
9
7
9
7
9
9
0
3
1
2
6
6
3
2
9
9
9
9
9
7
9
7
7 9 9 9
5
2
3
2
8
8
3
2
33
9 9
9
7
7
9
5
9
8
2
8
3
2
0
7
6
2
9
7
7
9
9
9
9
9
8
2
3
0
3
2
0
2
3
5
9 7
9
9
9
9
7
9
9
(a)
2-dimensional
(b) 4-dimensional
3
3
3
3
3
3
3
9
9
9
9
9
9
9
9
9
9
3
3
3
3
3
3
3
3
3
3
9
9
9
9
9
9
9
9
9
3
3
3
3
2
3
3
3
3
9
9
9
9
9
9
9
9
9
3
3
3
3
3
3
3
3
3
3
9
9
9
9
9
9
4
9
9
9
3
3
3
3
3
3
3
3
3
9
9
9
9
9
9
9
9
3
3
2
3
3
3
3
3
7
9
9
9
9
9
9
9
9
9
3
3
3
3
3
3
3
3
3
3
9
9
9
9
9
4
9
9
9
3
3
3
3
3
3
3
3
3
9
9
9
9
9
9
9
9
9
3
3
3
3
3
3
3
3
3
3
9
9
9
9
9
5
3
3
3
3
9
9
9
9
9
9
9
8
9
(c) 6-dimensional
(d)
16-dimensional
Figure 10: Samples generated using the same latent code for each generation, showing that the randomness of the code-conditional generation of DRL reduces in higher dimensional latent spaces.
0.10
0.2
Silhouette score
Silhouette score
0.08
0.06
0.1
0.04
0.0
0.02
0.00
-0.
-0.02
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
(a) MNIST
(b) CIFAR-10
Figure 11: Mean and standard deviation of silhouette scores when training a DRL model on MNIST (left) and CIFAR-10 (right) using single over three
given input to the model in the same way the time embedding. That is, each channel is provided with a conditional bias determined by the representation and time embedding at multiple stages of the downsampling and upsampling block.
18

---

Under review as a conference paper at ICLR 2022
SM
SM
VSM
AE VAE
AE
28
37
VAE
96
Accurscy
26
35
34
33
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.3
1.0
(a) 100 labels
(b) 1000 labels
Figure 12: Classifier accuracies for few shot learning given 8-dimensional representations learned
using DRL (SM), VDRL (VSM), Autoencoder (AE) and Variational Autoencoder (VAE).
Regularization of the latent space For both datasets, we use a regularization weight of 10-6 when applying Ll-regularization, and a weight of 10-7 when using probabilistic encoder regularized with KL-Divergence.
MNIST hyperparameters Due to the simplicity MNIST, we only use two resolutions of size 28 28 x 32 and 14 x 14 x 64, respectively. The number of residual blocks at each resolution is set to two. In each experiment, the model is trained for 80k iterations. For uniform sampling of we trained the models for an additional 80k iterations with frozen encoder and uniform sampling of t.
CIFAR-10 hyperparameters For the silhouette score analysis, we use three resolutions of size 32 32 32, 16 16 x 32, and 8 x 32, again with only two residual blocks at each resolution. Each model trained for 90k iterations.
CIFAR-10 (deep) hyperparameters While representation learning works for small models already, sample quality on CIFAR-10 is poor for models of the size described above. Thus for models used to generate samples, we use eight residual blocks per resolution and the following resolutions: 32 x 32 x 32, 16 x 16 x 64, x 8 x 64, and 4 4 x 64. Each model is trained for 3004 iterations. Note that this number of iterations not sufficient for convergence, however capable of illustrating the representation learning with limited computational resources.
A.6 EVALUATION OF THE INFINITE- INFNNIEE-DMMNSSIN DIMENSIONAL REPRESENTATION
In order to evaluate our
representation, we conduct an ablation study where we
compare our proposed method with Autoencoders (AH) and Variational Autoencoders (VAE)
CIFAR-10 images. We measure the accuracy of an SVM provided by sklearn (Pedregosa et al. 2011 with default hyperparameters trained on the representation of 100 (resp. 1000) training samples and their class labels. For our time-dependent representation, this is done for fixed values of t between 0.0 and 1.0 steps of 0.1. This done for both DRL and VDRL, where encoder regularized by including an additional KL-Divergence term in the training objective. DRL and AE were regularized using LI -norm, and the regularization weight was optimized for each model independently.
probabilistic
Results for few-shot learning with fixed representations are shown in Figure As expected, the ac- curacies when training on the score matching representations highly depend on the value of t. Overall our representation achieves much better scores when using the best f, and performs comparable to AE and VAE for 1.0. This aligns with Proposition claiming that our representation learning method for 1.0 (. similar to a static code learned using reconstruction objective. Note that the shape of the time-dependent classifier accuracies resemble the one of the silhouette score of CIFAR- 10 in II This is not surprising, since both training on single values of t and learning a time-dependent representation both trained find the optimal representation for given value of We further
19

---

Under review as a conference paper at ICLR 2022
Pretraining using DRL
Dataset
#labels
No pretraining
Improvement
CIFAR-10
100
64.12
69.79
+5,67
500
86.24
88.28
+2.04
1000
87.48
88.56
+1.08
2000
89,99
89.52
-0,47
4000
90.15
91.13
+0.98
CIFAR-100
1000
45.14
48.04
+2.90
4000
59,86
60.34
+0,48
10000
64.83
65.80
+0,97
20000
65.77
66.39
+0.62
MinilmageNet
4000
47.18
50.75
+3.57
10000
58.66
58.62
-0.04
Table 4: Classifier accuracy % with and without DRL as pretraining of the classifier when training for 100 epochs only.
want point out that representation learning through score matching enjoys the training stability of diffusion-baas generative models, which is often not the case in GANs and VAEs.
A.7 SEMI-SUPERVISED IMAGE CLASSIFICATION
Architecture and Hy pyperaaraeetess In all experiments, our encoder has the same architecture the classifier, where the hidden layer used to measure similarities for assigning pscudo-labels in LaplaceNet used as the latent code in representation learning. For all experiments, the input to the encoder is included as trainable parameter of the model and initialized with t = 0.5. As done in the original paper, we train the model for 260 iterations, where each iteration consists of assigning pseudo-labels and training is preceded by 100 supervised epochs on the labeled data. We use the small WideResNet model WRN-28- Sellars et al. (202T) and the same hyperparameters as the authors.
epoch of supervised training the assigned pseudo-labels. The
Evaluation with limited computation time In the following include more detailed analysis on the scenario few supervised labels and limited computational resources. Besides LaplaceNet and its version without mixup, we include ablation study of encoder pretraining as part of an autoencoder using binary cross entropy as reconstruction objective. In addition, we propose to improve the search for the optimal value of / by model selection, since the gradient for / is usually noisy and small. Thus we include additional experiments where we chose the initial & based on the minimum training loss after 100 epochs of supervised training. The optimal & is approximated by calculating the training loss for 1l equally spaced values of t in the interval [0.001, 1]. The results shown Table While mixup achieves no significant improvement the few-label case trained using 100 epochs, we can see that simple autoencoder pretraining consistently improves classifier accuracy. More notably however, our proposed pretraining based on score matching achieves significantly better results than both random initialization and autoencoder pretraining. In the L-search, we observed that for all datasets, our proposed method selects the interval [0.4. 0.6] during training. While this shows that our the approach of selecting / based on supervised training loss is not working, it demonstrates that the parameter t can very well be learned in the training process, making the downstream task performance robust to the initial value of t. In our experiments the final value t was always in the range [0.4, 0.6], independent of the initial value of t.
0.9, however it moves towards
20

---

Under review as a conference paper at ICLR 2022
CIFAR-10 100 labels
CIFAR-100 1000 labels
MinilmageNeel 4000 labels
Pretraining
Options
None
64.12
45.14
47.18
None
mixup
54.06
46.28
47.64
DRL
69.79
48.04
50.75
DRL
(-search
67.07
47.08
50.31
Autoencoder
64.99
46.88
48.52
Table Comparison of classifier accuracy in % for different pretraining methods in the of few supervised labels when training for 100 epochs only.
CIFAR-10 100 labels
CIFAR-100 1000 labels
MinilmageNeet 4000 labels
Pretraining
None
73.68
55.58
58.40
DRL
Autoencoder 74.31
58.95 55.85
58.84
55.41
57.93
Table 6: Classifier accuracy in % for autoencoder pretraining compared with the baseline and score matching pretraining. No mixup applied for this ablation study.
Ours Basic
Ours Basic Yes
Ours Mixup-DRL Yes
Ours VDRL No
Ours
Pretraining
VDRL
Mixup in sup. training
No
Yes
Dataset
#labels
CIFAR-10
100
74.31
64.67
70.40
81.63
77.51
500
92.70
92.31
92.55
92.79
91.46
1000
93.24
93.42
93.14
93.60
93.33
2000
94.18
93.91
93.80
93.96
94.27
4000
94.75
95.22
94.75
95.00
94.87
CIFAR- 100
1000
55.85
55.74
55.15
56.47
55.65
4000
67.22
67.47
67.09
67.54
67.52
10000
73.31
73.66
74.36
73.50
73.20
20000
76.46
76.88
77.04
76.64
76.68
MinilmageNet
4000
58,95
59,29
59.46
59.14
59.36
10000
67.31
66.63
67.31
67.46
66.79
Table 7: Evaluation of classifier accuracy in %, including the setting using mixup during pretraining (right column). DRL pretraining is proposed representation learning, and "Mixup-DRL the respective version which additionally applies mixup during pretraining. "VDRL" instead uses probabilistic encoder.
21

---

Under review as a conference paper at ICLR 2022
2.00
Po
Rc.
1.50
10-2
0.75
0.25
1.0
0.0
0.2
0.4
0.6
(a) Distrihution of after adversary convergence (h) Final training loss each for training adversarial distribution and uniform distribution
I
Figure 13: Comparison of the distribution of and the respective loss with and without including the adversary Xa-
22

---

Under review as a conference paper at ICLR 2022
1.00
0.75
0.50
0.25
0.00
0.25
0.50
-1.00
600000
610000
620000
630000
640000
650000
Iteration
Figure 14: Value of parameter over training, where 2a is the slope of the adversary Xa-
A.8 ADVERSARIAL TRAINING
For evaluating adversarial training, we initialize our model using a pretrained checkpoint which has been trained for 600k iterations. We then alternate training iterations between the model and the adversary, which is done for an overall number of 50k iterations.
In order to prevent À from increasing indefinitely, we propose to fix it to certain function class. For simplicity, one can remove A's dependence on æ, thus setting X(t) A(z, t). It was shown by Song et al. bb221b that setting A(t) (t) yields the KL-Divergence objective DKL. Hence we chose A(t) = ( to have KL-Divergence as the initial divergence when X'(() = 1 1A E [0. T]. We further propose to limit X' to the set of linear functions with slopes ranging from - 1 to 1. Formally, we set X (t) = 2at + (1 - a), E 11, 1]. Note that using this formulation, X, (t) can be seen as probability distribution over [0, 7). In addition, noticed that in practice sampling according to X' X(t)) instead of weighting the loss consistently achieves better results and thus used this approach our evaluation in Section
23


Reviews:
[{"review_id": "SEdtPhPdQW_", "reviewer": "Reviewer_VdM1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThe method proposed in this paper is simple and easy to use. Experimental results also demonstrate improvement over baselines.\n\n## Weaknesses\n\n1. The idea of representation learning with multi-scale denoising score matching is not new. Very similar ideas have already been discussed in at least [2][3][4].\n\n1. Using \"diffusion-based\" and \"score-based\" interchangeably in the paper makes it very confusing. I highly recommend sticking to one name. Given the dependency of the proposed approach on denoising score matching losses, I feel \"score-based\" might be a more appropriate naming.\n\n2. Proposition 1 is quite straightforward. Everyone who understands the proof of denoising score matching should know clearly that it is equivalent to the Fisher divergence up to a constant. It feels inappropriate to write down the expression of the constant as if it were your own contribution.\n\n3. The second sentence in section 3 is misleading. The work of Song et al. 2021a didn't claim diffusion models are trained to minimize KL-divergence—it is only true for a particular weighting of the score matching losses called the likelihood weighting. Also, the statement holds for the likelihood weighting even if the score model is not curl-free.\n\n4. In the second paragraph of section 3.2, authors claimed that the maximum pairwise distance of images is 170 for CIFAR-10. This is incorrect according to song and ermon 2020, where the maximum pairwise distance is reported to be around 50 for images with pixel values in [0, 1].\n\n5. The adversarial training of lambda divergences is not principled because the unknown constant of denoising score matching can depend on lambda. Also authors should cite and discuss the related work [1] which discusses adversarial training for score matching losses.\n\n## References\n[1] Jolicoeur-Martineau, Alexia, et al. \"Adversarial score matching and improved sampling for image generation.\" ICLR 2021.\n\n[2] K. J. Geras and C. Sutton. Scheduled denoising autoencoders. arXiv preprint arXiv:1406.3269,\n2014.\n\n[3] B. Chandra and R. K. Sharma. Adaptive noise schedule for denoising autoencoder. NeurIPS 2014.\n\n[4] Q. Zhang and L. Zhang. Convolutional adaptive denoising autoencoders for hierarchical feature\nextraction. Frontiers of Computer Science, 2018.\n\n\n", "summary_of_the_paper": "This paper proposes to leverage the recently proposed mixture of denoising score matching losses of score-based generative models for representation learning. Since the representation encoder is conditioned on time, it can be viewed as an infinite dimensional representation. Results demonstrate improvement over previous semi-supervised learning techniques by pre-training classifiers with the method proposed in this paper.\n", "summary_of_the_review": "Authors propose a simple idea to extract multiple-scale features with denoising score matching. However, very similar approaches have already been proposed in previous works and authors fail to cite and discuss.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LJy1XLINBU", "reviewer": "Reviewer_Wb9v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written and the method seems sound. The topic of score-based generative models as an alternative to GANs and VAEs is worth exploring and the authors propose an interesting way to generate powerful representations. \n\nThe fact that pretraining with such method allows to improve the state-of-the-art in the semi-supervised classification setting is an important contribution, even though the improvement seems less significant on datasets with more classes than CIFAR-10. It would also be desirable to have standard deviations associated to the classification accuracy values to better assess statistical significance of the result.\n\nResults on the generation task do not seem particularly impressive but it is fine for it not to be the main focus. \n\nA couple of typos should be fixed: page 1 the acronym SDE has not yet been defined; page 3 Densoising --> Denoising.", "summary_of_the_paper": "The paper presents a novel technique for representation learning by means of generative models. In particular, non-adversarial score-matching methods are improved to control the level of detail in the representations and learn a infinite-dimensional code. This is particularly useful for the semi-supervised learning setting, where it is shown that pretraining via the proposed model allows strong class separation and leads to state of the art results.", "summary_of_the_review": "Pros:\n- Interesting extension of current non-adversarial generative models, with expanded theoretical results\n- Powerful representation learning capabilities and SoTA on semi-supervised classification\n- Adequate experimental assessment and ablation\nCons:\n- Results on image generation are not very impressive\n- Statistical significance of the classification results could be better assessed\n\nI think that overall the paper is a valuable contribution and expands the knowledge on representation learning techniques in a meaningful way. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JeEKVkUWEN", "reviewer": "Reviewer_pE4h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths.\n\n    The idea, presented in this paper, is novel.  The motivation behind the method, taking its origin in the rethought conditional score matching objective, is clearly explained.\n\n\n2. Weaknesses.\n\n    a. Firstly, the submitted manuscript misses the detailed review of related works on representation learning. Therefore, it is rather difficult to realize which context the authors put their work in. The restriction of limited computational resources was mentioned, but it is still unclear what the formal restriction is.\n\n    b. In the absence of context, it is difficult to understand the choice of a baseline for the quantitative evaluation. The last two years were very successful for representation learning in the domain of images. Such methods as SimSiam [2], SimCLRv2 [3], BYOL [4] et cetera may be considered as reliable models for comparison. However, they are not even mentioned in the paper. Also, the evaluation protocol used in those works could be applied to the proposed DRL approach.\n\n    c. Though the idea of infinite-dimensional representations looks appealing, I did not understand how it can be applied for the downstream tasks. During the experiment, described in Sec. A5, the authors chose the specific timestamp according to the best classifier accuracy. Thus, the final representation is still point-based. Am I right, that for any other task, e.g. image retrieval, one should again test several values of $t$ and choose the best one?\n\n\n3. Questions and suggestions.\n\n    a. It is claimed in Sec. 3.1. that $\\lambda$-divergences can express any $f$-divergence. However, as far as I can understand, paper [5] has shown that $\\lambda$-divergence can bound the KL-divergence. Could you please put the proof sketch or the reference to one of the previous results?\n\n    b. What is the motivation behind the sum of a Uniform r.v. and Gaussian r.v. (Sec. 4.3)?\n\n    c. In Eq. 23 the decoder should be defined as $D_\\theta \\left(E_\\phi \\left(x_0, T\\right)\\right)$ instead of $D_\\theta \\left(x_T, E_\\phi \\left(x_0, T\\right)\\right)$.\n\n\n4. References\n\n    [2] Chen et al. Exploring Simple Siamese Representation Learning. 2020.\n\n    [3] Chen et al. Big Self-Supervised Models are Strong Semi-Supervised Learners. 2020.\n\n    [4] Grill et al. Bootstrap your own latent: A new approach to self-supervised Learning. 2020.\n\n    [5] Song et al.  Maximum likelihood training of score-based diffusion models. 2021.", "summary_of_the_paper": "In this paper, the authors propose a new scheme for training representations using diffusion probability models (DDPM). In detail, the estimator $s_\\theta$ receives the encoded original image $E_\\phi \\left(x_0\\right)$ (or $E_\\phi \\left(x_0, t\\right)$) as additional input - the DDPM loss function causes the $E_\\phi$ encoder to extract the features needed for image reconstruction. Remarkably, this representation can be deterministic or stochastic, pointwise or infinite-dimensional (with time variable $t$).\n\nQualitative experiments on MNIST and CIFAR-10 show that the encoder can learn some meaningful features. A quantitative comparison with LaplaceNet [1] on CIFAR-10, CIFAR-100, and MiniImageNet, demonstrates an improvement in the quality of semi-supervised classification.\n\n[1] Sellars et al. LaplaceNet: A Hybrid Energy-Neural Model for Deep Semi-Supervised Classification. 2021.", "summary_of_the_review": "1. Pre-rebuttal rating.\n\n    Despite I admit the idea, presented in the paper, is interesting and novel, the evaluation does not seem conclusive for me. I ask the authors to motivate the choice of the baseline and explain what the use case of the presented approach is. Currently, I tend to rate the submission below the acceptance threshold.\n\n2. Post-rebuttal update\n\n    Having read the authors' feedback and other reviews, I keep the initial rating. While the presented ideas may be of some interest to the community, the evaluation of the approach should be more thorough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1UW5AIZp0zF", "reviewer": "Reviewer_XjvM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to follow and well-motivated. The idea of incorporating a latent code into the score network looks promising and interesting.  However, I have a few concerns:\n\n* A few important formulations and implementation details are missing. For example, what is the exact formulation of VDRL? It is only briefly discussed at the end of section 2.3 but without further introduction. For the semi-supervised learning experiments, what is the value of $t$ used to extract the latent code in inference time? Did you use the adversarial training and initial noise schedule tricks that are introduced in section 3?\n\n*  From the limited experiment description, I suppose that for the semi-supervised learning, the encoder is first pretrained by the DRL objective and then finetuned by the original LaplaceNet algorithm. In that case, it seems that the most contribution leading to the performance is still from the LaplaceNet, and the improvement is tiny. A more informative experiment is to extract features directly from the learned encoder and train an L-2 SVM classifier on the top. \n\n* Section 3 and the corresponding experiment subsections look like a completely different topic that is not relevant to the paper title \"representation learning\". I would suggest removing those content and including more details and variants of representation learning experiments.\n\n", "summary_of_the_paper": "This paper proposes a revised version of score-based models for representation learning. Specifically, they add an additional input to the score network which is a latent code encoded from the clean sample by an encoder. The encoder can be further conditional on time. Using the pertaining algorithm they achieve improvement of SOTA semi-supervised learning methods. They also show how adversarial training and tuning of the initial noise schedule can improve the performance of score-based models. ", "summary_of_the_review": "Overall I appreciate this paper's idea of using score-based models for representation learning. However, I feel the writing and experiments can be improved before it gets accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes extracting multiple-scale features using denoising score matching. Reviewers pointed out the limited novelty in the work and that it does not cite various previous work and how it connects to them.  The paper needs some further  polishing on the writing, and in making the use of lambda divergences more rigorous and principled as explained in the comment of Reviewer VdM1 .
Decision: Reject