From Points to Functions: Infinite-dimensional Representations in Diffusion Models
Authors: Sarthak Mittal
Guillaume Lajoie
Stefan Bauer
Arash Mehrjou
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
diffusion models
score-based learning

Abstract: Diffusion-based generative models learn to iteratively transfer unstructured noise to a complex target distribution as opposed to Generative Adversarial Networks (GANs) or the decoder of Variational Autoencoders (VAEs) which produce samples from the target distribution in a single step. Thus, in diffusion models every sample is naturally connected to a random trajectory which is a solution to a learned stochastic differential equation (SDE). Generative models are only concerned with the final state of this trajectory that delivers samples from the desired distribution. \cite{abstreiter2021diffusion} showed that these stochastic trajectories can be seen as continuous filters that wash out information along the way. Consequently, it is reasonable to ask if there is an intermediate time step at which the preserved information is optimal for a given downstream task. In this work, we show that a combination of information content from different time steps gives a strictly better representation for the downstream task. We introduce an attention and recurrence based modules that ``learn to mix'' information content of various time-steps such that the resultant representation leads to superior performance in downstream tasks.


Under review as a conference paper at ICLR 2023
FROM POINTS TO FUNCTIONS: INFINITE-DIMENSIONAL REPRESENTATIONS IN DIFFUSION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion- generative models learn iteratively transfer unstructured noise to complex target distribution opposed to Generative Adversarial Networks (GANs) or the decoder of Variational Autoencoders (VAEs) which produce sam- ples from the target distribution in single step. Thus, in diffusion models ev- ery sample is naturally connected to a random trajectory which is a solution to a learned stochastic differential equation (SDE). Generative models are only con- cerned with the final state of this trajectory that delivers samples from the desired distribution. Abstreiter et al. (2021) showed that these stochastic trajectories be seen as continuous filters that wash out information along the way. Conse- quently, it is reasonable ask if there an intermediate time step at which the preserved information is optimal for a given downstream task. In this work, we show that a combination of information content from different time steps gives a strictly better representation for the downstream task. We introduce an attention and recurrence based modules that "learn to mix" information content of various time-steps such that the resultant representation leads to superior performance in downstream tasks.
INTRODUCTION
A lot of the progress in Machine Learning hinges on learning good representations of the data, whether in supervised or unsupervised fashion. Typically in the absence of label information, learn- ing a good representation is often guided by reconstruction of the input, as is the case with autoen- coders and generative models like variational autoencoders (Vincent et al., 2010; Kingma & Welling, 2013; Rezende et al., 2014); or by some notion of invariance to certain transformations like in Con- trastive Learning and similar approaches (Chen et al., 2020b;d; Grill et al., 2020). In this work, we analyze novel way of representation learning which was introduced in Abstreiter al. (2021) with a denoising objective using diffusion based models to obtain unbounded representations.
Diffusion-based models (Sohl-Dickstein et al., 2015; Song al., 2020; 2021; Sajjadi et al., 2018; Niu et al., 2020; Cai et al., 2020; Chen et al., 2020a: Saremi et al., 2018; Dhariwal & Nichol, 2021; Luhman & Luhman, 2021; Ho al., 2021: Mehrjou al., 2017: Nichol & Dhariwal, 2021) are generative models that leverage step-wise perturbations to the samples of the data distribution (eg. CIFARIO), modeled via a Stochastic Differential Equation (SDE), until convergence to unstruc- tured distribution (eg. N (0, I)) called, in this context, the prior distribution. In contrast to this diffusion process, "score model" is learned approximate the reverse process that iteratively con- verges to the data distribution starting from the prior distribution. Beyond the generative modelling capacity of score-based models, we instead use the additionally encoded representations to perform inference tasks, such classification.
In this work, we revisit the formulation provided by Abstreiter et al. (2021); Preechakul et al. (2022) which augments such diffusion- systems with encoder for performing representation learn- ing which can be used for downstream tasks. In particular, we look at the representation learning methodology from Abstreiter et al. (2021) and perform deeper dive into (a) the benefits of utilizing the trajectory or multiple points on as opposed to choosing just a single point, and (b) the kind of information encoded different points. Using trained attention

---

Under review as a conference paper at ICLR 2023
CFAR10
CHAR100
Hiri Image
33
72
43.0
60
43.0
24
Medel VSR
CEAR10
DIAR100
Mini Image
16
53.0
34
45.0
35
00
34
66
Figure 1: Downstream performance of single point based representations (MLP) and full trajec- tory based representations (RNN and Tsf) different datasets for both types of learned encoders: probabilistic (VDRL) and deterministic (DRL) using 64-dimensional latent space (Top) and a 128- dimensional latent space (Bottom).
mechanisms over diffusion trajectories, ask about similarity and differences of representations across diffusion processes. Do they encode certain interpretable features at different points, or redundant look at the whole trajectory? It is worth noting that while the representation itself is infinite- dimenssodlensinna,, we do discretize for performing our analysis.
Our findings be summarized follows:
We propose using the trajectory-based representation combined with sequential architec- tures like Recurrent Neural Networks (RNNs) and Transformers to perform downstream predictions using multiple points as it leads to better performance than just finding one- best point on the trajectory for downstream predictions (Abstreiter al., 2021).
We analyze the representations obtained at different parts of the trajectory through Mutual Information and ireention-bssd relevance to downstream tasks to showcase the differ- ences in information contained along the trajectory.
We also provide insights into the benefits of using more points on the trajectory, with satu- rating benefits as our discretization becomes finer. We further show that finer discretizations lead to even more performance benefits when the latent space is severely restricted, eg. just a 2-dimensional output from the encoder.
BEYOND FIXED REPRESENTATIONS
We first outline how diffusion- based representation learning systems are trained. Given some ex- ample Xo E gd which is sampled from the target distribution Po. the diffusion process constructs the trajectory (x, (xe))ejo.1 through the application of an SDE. In this work, we consider the Variance Exploding SDE (Song et al., 2021) for this diffusion process, defined
dx = f(x, 1) + g(()dw = - dw
(lo
where w is the standard Wiener process and o²() the noise variance of the diffusion process. This leads closed form distribution of x, conditional Xg pne pxe Xo) N Xo, (o2
2

---

Under review as a conference paper at ICLR 2023
00
6.35
-14
6.70
30 54 62 63
65
64
07
08 38 1.0
30 $1
62
63
04
65
6.
6J
08 00
10
(a) VDRL CIFARIO
(b) VDRL
CIFAR100
(c) VDRL Mini-ImageNett
00
6:00
6.40
2008
6.75
6.35
-66
-6.0
2808
30
1.0
(d) DRI. | CIFARIO
CIFARIOO
(l) DRL
Mini ImageNet
Figure 2: Normalized Mutual Information between different points on the trajectory. Cell (7,j) demonstrates the normalized mutual information, estimated with the MINE algorithm, between the representations at time & and t j.
(O) Given this diffusion process modeled through the Variance Exploding SDE, the reverse SDE takes similar form but requires the knowledge about the score function, i.e. Vx log p, for all [0, 1]. A common way obtain this score function is through the Explicit Score Match- ing (Hyvärinen & Dayan, 2005) objective,
Ex,
||se [|so Vx, log (xx))l
(2)
which suffers from just one hiccup, which that data about the ground-truth score function is not available. To solve this problem, Denoising Score Matching (Vincent, 2011) was proposed,
Ex
Ex,|x0
||s0 (x,, t) Vx, log (oc ]=]]
(3)
where the term log Poe (x Xo) is available due to its closed-form structure. Given that the above objective cannot be reduced to O, Abstreiter al. (2021) proposes the objective
Exp
Ex,
E
[|00(Xe, (xo 1),4) Xx, log Poe (x. xo)ll
(4)
where the additional input (x t) to the score function is obtained from a learned encoder. It pro- vides information about the unperturbed sample that might be useful for denoising data at time step I in the diffusion process. Training this system can lead to the objective being reduced to O, thereby providing incentive the encoder E. i) learn meaningful representations for each time From this, we obtain a trajectory-basee representation (Ex(xo. t) te) for each sample Xo. opposed finite sized representations obtained from typical Autoencoder (Bengio al., 2013; Vinyals al., 2016; Kingma & Welling, 2013; Rezende et al., 2014) and Contrastive Learning (Chen et al., 2020c; Grill al., 2020; Caron et al., 2021; Bromley et al., 1993; Chen & He, 2020) approaches.
Following the setup in Abstreiter et al. (2021), we consider two different versions of the encoder Eo(:.). (a) the VDRL setup. where the output :.(..) represents a distribution from which sample is used, and the distribution is regularized using KL--Dvergence term with the standard Normal distribution A(0, I), and (b) the DRL setup, where the output of the encoder is deterministic and regularized using L1 distance metric to be as close 0 as possible. Typically in all our experiments, we see that not only the trends hold with multiple seeds but also across these two types of encoders, substantiating the statistical significance of the trends.
is important to note that our goal here is strictly representation learning, and thus we use the representations obtained for downstream (multitask-) image classification. This should not be fused with generative modelling as the provided mechanism augments generative model for rep- resentation learning, but is not generative model its Since this representation learning
3

---

Under review as a conference paper at ICLR 2023
0.55
9.25
0.73
0.095
0.80
0.83
0.00
6.
(a) VDRL CIFARIO
(b) VDRL CIFAR100
(c) VDRL Mini-LmageNNet
0.12
4.00
9.40
1.
(d) DRL CIFARIO
(e) DRL CIFARIOD
(I) DRL Mini- Linie-e
Figure 3: Attention Scores provided to different points on the trajectories, which are obtained from diffusion based representation learning systems with probabilistic encoders (VDRL; top row) and with deterministic encoders (DRL; bottom row) across the following datasets (i) Left: CIFARIO, (ii) Middle: CIFAR 100, and (iii) Right: Mini-Imagenet.
paradigm be augmented with conditoond encoder model, this leads natural extension to trajectory-basee -basdd (unbounded) representation, in contrast to typical bounded representation learn- ing models like Autoencoders. Thus, this representation learning paradigm constructs functional map from the input space to a curve / trajectory in Rdd, where we refer to d as the dimensionality of this encoded space.
2.1 INFINITE-DIMER DIMNIEEDIMENSIONAL REPRESENTATION OF HNNIE-DIMENSION DIMENSIONAL DATA
Normally in autoencoders or other static representation learning methods, the input data xo R& is mapped to single point E I in the code space. However, our proposed algorithm learns richer representation where the input X is mapped encoder (-, t). Hence, the learned code is produced by the map + (Eo (o) where the infiniteddimensional object (Fs (x ()){((011) is the encoding for Xo-
curve in R instead of single point through the
The learned code least good static codes in terms of separation induced among the codes. Consider two input samples Xo and x/y, hence we have:
[x0, 0) Ex(x) 0)ll I sup [xo - Ex t)ll rE(o.1]
(5)
which implics that the downstream task at least recover the separation provided by finite- dimensional codes from the infinite-dimensionall code by looking for the maximum separation along the representation trajectory.
downstream task can leverage this rich encoding in various ways. Consider the classification task where we want find mapping giving Xo as the input to f. we define f H + {0, }} where the input to the classifier is the whole trajectory (xo, ())oo.))))o (o]o)))) Thus, the classifier can use RNN and Transformer models make use of the information content of the entire trajectories.
Rdd {0. 1} from input data the label space. Instead of
3 EXPERIMENTS
We first train two kinds of diffusion-based generative model as outlined in Abstreiter et al. (2021), based on probabilistic (VDRL) and deterministic (DRL) encoders respectively. After training, the encoder model is kept fixed. For all our downstream experiments, to obtain the trajectory based representation for each of the samples. While the trajectories lie in continuous domain [0, 1). sample at regular intervals with length 0.1, unless specified
use this trained encoder
4

---

Under review as a conference paper at ICLR 2023
3.0
3.10
1.00
2..0
X33
AU
AU
64
VDRL Background Color
(b) VDRL Foreground Color
(c) VDRL Location
(d) Objectt Shupe
144
9.3
5915
341
536
(c) DRL
Background Color
(f) DRL
Foreground Color
(g) DRL
Location
(h) DRL
Object Shape
Figure 4: Attention score profiles for different tasks under the Synthetic dataset, when using the VDRL framework (top) and the DRL framework (bottom). The scores reveal that almost all points in the trajectory store similar amounts of information about the background color, while the latter part of the trajectory encodes more information about the foreground object. In particular, information about the location is most heavily found near the end of the trajectory.
otherwise. This leads to a discretization of the trajectory, which is then used for various analysis as outlined below. Further, we consider the dimensionality of the latent space, that is, the output of the encoder, as 128 unless otherwise specified. Additional details about the architectures used, the optimization strategy and other implementation details can be found in Appendix B.
3.1 DOWNSTREAM PERFORMANCE REVEALS BENEFITS OF TRAJECTORY INFORMATION
To understand the benefits of utilizing the trajectory-based representations, we train standard Multi- Layer Perceptron (MLP) models at different points on the trajectory and compare it with Re- current Neural Network (RNN) (Hochreiter Schmidhuber, 1997: Cho et al., 2014) and Trans- former (Vaswani et al., 2017) based models that are able to aggregate information from different parts of the trajectory.
We evaluate the MLP, RNN and Transformer based downstream models on diffusion systems with both probabilistic encoders (VDRL) and also non-probabilistic ones (DRL). In Figure I, we see the performance of these different setups for the following datasets: CIFARIO (Krizhevsky et al., a), CIFAR100 (Krizhevsky et al., b) and Mini- Mini-lmageNet (Vinyals et al., 2016). Note that in contrast to MLP implementations. RNN and Transformer plotted across all time points for visual comparison. We typically see that RNN and Transformer based models perform better than there is no single point on the trajectory that encapsulates all the information necessary for opti- mal classification, and thus utilizing the whole trajectory opposed to individual points leads to improvements in performance.
the entire trajectory and the obtained performance
the peaks obtained by the MLP systems. This shows that
We further do this performance analysis for different dimensionality of the latent spaces, that is, when the trajectory representation is embedded in 64-dimensional Euclidean space (Figure 1: Top) and when emebdded 128 18-dimensional Euclidean space (Figure 1: Bottom). We see similar trends across the two settings, thus highlighting consistent benefits when using discretization of the whole trajectory.
3.2 MUTUAL INFORMATION REVEALS DIFFERENCES ALONG THE TRAJECTORY
In an effort to understand whether different parts of the trajectory based representation actually contain different types of information about the sample, we evaluate the mutual information between the representations at various points in the trajectory. 2018) to estimate the mutual information between the representations any two different points in the trajectory. Through this algorithm, we compute and analyse a normalized version of the mutual information. defined as NMI(X: Y) := I(X; Y) /)())H(( where I(- ; is the standard Mutual Information function (Cover, 1999) and H(-) is the entropy function.
use the MINE algorithm (Belghazi et al.,
5

---

Under review as a conference paper at ICLR 2023
0.33
0.33
0.33
0.20
0.20
0.20
6.15
0.15
0.15
0.09
0.05
0.05
0.00
0.5
0.3
9.2
9.3
04
0.5
0.6
077
0.6
1.0
0.09
0.3
0.1
0.2
9.3
94
0.5
06
07
06
69
1.0
0.00
0.3
9.3
0.2
53
94 0.5 06
Timestep
07
08
09 3.
(a) VDRL Background Color
(b) VDRL
Foreground Color
(c) VDRL
Digit Identity
0.40
0.40
0.35
0.35
0.35
0.223
0.23
0.29
0.20
0.20
0.15
0.03
0.03
0.05
0.00
02
0.1
04
05 Timssspp
0. 1.0
0.00
0.1 0.2
0.9
0.7
1.0
0.00
9.2 9.3
05 0.7 08
Timestpp
(d) DRL Background Color
(e) DRL
Foreground Color
(l) DRL Digit Identity
Figure 5: Attention score profiles for different tasks under the Colored-MNISS dataset, when using the VDRL framework (top) and the DRL framework (bottom). The scores reveal that almost all points the trajectory store similar amounts of information about the background as well as the foreground color, while the latter part of the trajectory encodes more information about the identity of the digit.
Figure illustrates the normalized mutual information between representations at different parts of the trajectory across three different datasets: 10FAR10, CIFAR1O0 100 and Mini-ImageNet as well as two different types of models: VDRL and DRI., where the former a probabilistic encoder and the latter doesn't. t. We see large normalized mutual information values near the principal diagonal and small values that are away from demonstrating that nearby representations the trajectory are similar whereas distant points in the trajectory are considerably different. This shows that different parts of the trajectory learn encode different kinds of information.
3.3 ATTENTION REVEALS RELEVANCE DIFFERENT PARTS OF THE TRAJECTORY
To complement the analysis in Sections 3. and 3.2, we train single-layered Transformer model for downstream prediction, which comes from learned embedding that queries information from different parts the trajectory. Through the analysis of the attention scores at different points in the trajectory, realize that the middle parts of the trajectory the most important, as illustrated in the high attention scores around f = 0.5 in Figure
This is in line with the performance results in Figure which also shows that amongst the single- point MLP-based systems, the best downstream performance reached near the middle of the tra- jectory. Attention score for any point in the trajectory, in single-layered Transformer network, can be understood as the weight or importance of that point in the whole trajectory for the task in consideration. For the three image-llassificttonn datasets that we experiment on, we see that the attention patterns are quite similar. However, in later sections we will provide analysis with more controlled settings and see that the attention score profiles show varied behaviour for different fea- tures, indicating and strengthening the claim from Section 3.2 that the trajectory indeed encodes different information different points.
3.4 PARSING SEMANTIC INFORMATION ENCODED ALONG THE TRAJECTORY
To better understand the different kind of information encoded in different parts of the trajectory, we expand our analysis into multi-task domains where each task relies information from different features in the input. We consider three different datasets for this fine-grained analysis: Synthetic, Colored-MNIST and CelebA

---

Under review as a conference paper at ICLR 2023
Scn
VDRL. Blury
VDRI.
Waaring
VDRL Blond Hair
(d) VDRL
Bald
Figure 8: Left: Jensen Shannon Divergence plot for the attention profiles for any pair of features in the CelebA dataset; Right: Attention Profile plots for subset of features in the CelebA dataset.
For all the analysis performed here, train the diffusion model with the dependenn encoder using Equation 4, and then keep frozen. We then perform inference to obtain the trajectory rep- resentations for different granularities for each data point. Then, for each task in the dataset, train a different single-layered transformer model and obtain attention scores over the trajectory corresponding to that particular task. This attention score over the points on the trajectory encodes the relevance of that area of the trajectory for the task, thereby providing insights on how much information about a particular feature encoded in which part of the trajectory.
Synthetic. Synthetic dataset consists of an object in scene. The scene consists of distinct background color and the object is as- sociated with distinct foreground, location as well as the object shape. The system consists of four tasks; determining the (a) back- ground color, (b) foreground color, (c) object location, and (d) ob- ject shape.
Figure 4 highlights the relevance of different parts of the trajecto- ries for the different tasks. We see that while the background color information is more less uniformly diffused over the whole tra- jectory, foreground information like object color, location and shape have a much more peaky distribution.
Fomplveed Oolor
Beciphand Cobe
oboee Soaps
Figure 6: Jensen Shannon Di- vergence plot for the attention profiles obtained for any pair of features for the Synthetic dataset using VDRL encoder.
We also refer the readers to Figure 6 which highlights the differ- the trajectory. Cell (i,j) in the
ences in attention distributions figure refers to the Jensen-Shannol divergence (JSD) between the distribution over trajectory obtained for task with that obtained for task j.
For additional details about the analysis as well as additional abla- tions and results using different granularities and latent dimension sizes, please check out Appendix D. We also provide examples of samples from this setup in the Appendix.
033
123
MNNITT We then extend our analysis to slightly more complex setting, where each sample consists of digit with dis- tinct foreground color and digit identity (0-9), along with a back- ground color on which the digit is embedded. The multi-task set- ting here consists of determining the (a) background color, (b) fore- ground color, and (c) digit identity.
Figure 7: Jensen Shannon Di- vergence plot for the attention profiles for any pair of fea- tures for the Colored-MNIST dataset using VDRL encoder.
Figure 5 shows which parts of the trajectory important for which task, and thus encode what kind of information. In this setting, we that both the background color and the foreground color infor- mation quite diffused over the trajectory, while the information about the digit identity is heavily present in the later parts of the trajectory.
We also refer the readers to Figure 7 for a birds-eye view on the differences in attention distributions over the trajectory for different tasks. Each cell (i,j) in the figure refers to the JSD between the
7

---

Under review as a conference paper at ICLR 2023
52
74
Jecuney
70
68
Cranulaity
Figure 9: Performance on different datasets evaluated at different granularities of the trajectory representation. Granularity-k the X-axis implies that the discretization of the trajectory represen- tation (obtained from the same frozen diffusion model), is done at & uniform points, which are then fed to Transformer model for downstream classification.
distribution over trajectory obtained for task i with that obtained for task j. A high divergence implies that the information corresponding to the two tasks is not present together simultaneously.
For additional details about the analysis as well as additional ablations and results using different granularities and latent dimension sizes, please check out Appendix E. We also provide examples of samples from this setup in the Appendix.
CelebA. Finally, we conduct experiments on the CelebA dataset (Liu et al., 2015), which is large- scale dataset that consists of images of celebrities as well as multiple binary labels for each corre- sponding the different attributes; eg. whether the celebrity in the image has brown hair not? We consider experimentation on CelebA to understand we can semantically understand what kind of features, (in a more real world setting), encoded in different parts of the trajectory. For full list of the attributes in consideration in this dataset, we refer the readers to Appendix F.
We refer the readers to Figure 27 for similar analysis of JSD between distributions over trajectories encoded by different features (or more formally; tasks corresponding to different features). While do see some clustering (eg. blond hair, black hair and brown hair all have similar attention profiles), there is also a lot of uniformity in the divergences. We believe that scaling and extending this setting to richer and more diverse multi-task, multi-feature domains would allow for much richer semantic separation between features.
We also highlight the individual attention profiles in Figure 27 for a subset of the features. shows that the distributions learned for different features are actually different, implying presence of com- plementary information along the trajectory (Figures 29 and 30 in the Appendix).
Overall, we highlight how information about different features is encoded in different regions of the trajectory. This can be leveraged by learning an automated task-conditioned system that learns to "look in more detail" at certain parts of the trajectory while ignoring the others. For additional details about the analysis as well as additional ablations, please check out Appendix
3.5 BENEFITS OF USING MORE POINTS IN THE TRAJECTORY
We now extend our analysis to understand the benefits of having more points in the trajectory, that is, of moving closer to the continuous-time domain. We do this by considering more and more fine-grained discretizations of the trajectories, which we denote as granularity. discretizes each trajectory representation by uniformly querying it at (k+1) different points, in J0, 1]. Thus, a granularity level of 2 indicates using the points (0.0, 0.5, 1.0} for downstream predictions.
granularity of
We refer the readers to Figure 9 which illustrates the benefits of having more points in the trajectory. In particular, even though a granularity level of 2 has access to the mid-point of the trajectory, doesn't do well when using a larger granularity. However, do notice that the benefits to having more points in the trajectory do start to saturate beyond a certain point.
We believe that the dimensionality of the latent code might have significant effect the saturating point. To this end, we perform experimentation on low-dimensional trajectories on the synthetic datasets in the next section.
8

---

Under review as a conference paper at ICLR 2023
100
100
90
Gramuder
Granuferity
60
Poneground Color
Badkgound Cord
locatton
Oltect Shaye
Paragrunn Color
Backyronn Coler
oV/
Footure
Feature
Figure 10: Improvements of increased granularity (finer discretization schedule) on different fea- tures for the Synthetic Dataset (Leff) and the Colored-MNIST Dataset (Right). The dimensionality of the latent space is heavily restricted to 2, and hence we see that with the coarsest discretization, performance on number of features suffers.
3.6 INTERPLAY BETWEEN CODE DIMENSIONALITY AND GRANULARITY
To provide additional analysis into the benefits of having this infinite-diment limeniediieeniooall representation, we consider the Synthetic and Colored-MNIST datasets and use a very small dimensionality for the latent space, which is the output of the encoder. In particular, consider the output of the encoder to be a 2-dimensional code, but unbounded in the time domain.
We discretize this 2-dimensional trajectory representation the analysis done in Section 3.5 to see how well a limited capacity (in latent bits per point on trajectory) code can do and how much does it benefit from the increase in dimensions through the time dimension?
different uniform points, similar to
Figure 10 highlights that for both the datasets, see substantial improvement when using more points on the trajectory when the latent code is severely restricted, further signifying the benefit of the temporal unbounded-nes: of the trajectory, and the monotonic improvement in performance with increase in granularity.
4 CONCLUSION
Through our analysis, we realize that the encoder Eg(-,t) actually learns different kinds of infor- mation different time-steps t. Typically the mid-points of the trajectory are the most important for downstream classification tasks but we uncover that using as many points on the whole trajec- tory, i.e. increasingly finer discretization of an infinite- dimensional object, is much better than just singular points on it. What kind of semantic information is encoded in the different parts of the trajectories? Can we show some benefits of the unboundedness of the trajectory? Through our anal- ysis. we provide insights into the differences of information stored along the trajectory, well as the benefits of its unbouneded structure especially in the domain of restricted latent dimensionality.
While we highlight some interesting properties of these trajectory-based representations well as the diversity of information over it, we believe that an important next step is to automate and learn the discretization process opposed to the heuristic based uniform discretization. We believe this could lead to a variable computation system, where the downstream model would learn on its own which part of the trajectory should it sample more finely than others, when conditioned on the task/feature used.
This kind of task-conditioned discretization process would not only be able use the whole trajec- tory information without heuristics but would also be able to leverage the structure which we show in our analysis in a more efficient and improved manner. We believe that this is an important direc- tion to obtaining task and feature centric representations which are more general than the finite-sized representations afforded by contemporary representation learning models.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We do not foresee any negative or unethical implications this work, which is in addition to the general impacts of advancement of Machine Learning and Representation Learning.
REPRODUCIBILITY STATEMENT
We perform all the experiments with multiple seeds, ranging from 10 depending the periments. For each run of the score-based diffusion model, we also perform the downstream ex- periments with multiple seeds obtain statistically significant results. We refer the readers the implementation details outlined in Appendix B and we will be open-soureing our code for ease of reproducibility.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion-based rep- resentation learning. arXiv preprint arXiv:2105.1422 2021.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning. pp. 531-540. PMLR, 2018.
Yoshua Bengio, Aaron Courville, and Paseal Vincent. Representation learning: A review and new perspectives. IEEE transactions pallern analysis and machine intelligence, 35(8):1798-1822) 2013.
Jane Bromley, James Bentz, Leon Bottou, Isabelle Guyon, Yann Lecun, Cliff Moore, Eduard Sackinger, and Rookpak Shah. Signature verification using "siamese" time delay neural net- work. International Journal of Pattern Recognition and Artificial Intelligence, 7:25, 08 1993.
doi: 10.1 10.1142/50218001493000033
Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. CoRR, abs/2006. 10029, 2020c. URL org/ aww. 6 10029.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04229 0429999.. 2020d.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint orXiv:1409.11 2014.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2/06.1 15282, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Aapo Hyvärinen and Peter Dayan. Estimation of normaazzdd statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Diederik Kingma and Max Welling. 1312v:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
11

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). a. URL htLLht/p:/wwee http://wwwww......o..ttt. toronto......o......... wdw/
wtt.caa....roo.................. cetarii/cffeee cifar. html.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced
research). b. URL http: /wwp toronto..oo...oooo.............. edu/ -kr /wtfarifrr....////// www.farrhmml..... html.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021.
Arash Mehrjou, Bernhard Schölkopf, and Saeed Saremi. Annealed generative adversarial networks. arXiv preprini arXiv:/705.. 00,00, 2017.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Chenhao Niu, Yang Song, Jiaming Song. Shengjia Zhao, Aditya Grover, and Stefano Ermon. Per- mutation invariant graph generation via score-based generative modeling, 2020.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakom. Diffu- sion autoencoders: Toward meaningful and decodable representation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International conference on machine learning, pp. 1286. PMLR, 2014.
Mehdi SM Sajjadi, Giambattista Parascandolo, Arash Mehrjou, and Bernhard Schölkopf. Tempered adversarial networks. International Conference on Machine Learning, pp. 4p.4451 4451-4459 PMLR, 2018.
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint arXiv: 880.08830, 2018.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2020.
Yang Song. Jascha Sohl-Dickstein. Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Compu- tation, 23()):16611167 2011. doi: 10.1162/NECO a.00142.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.
Matching networks for one shot learning. CoRR, fbss/606.04080. 2016. URL http:://aaak
org/abs/1600 606.60666 04080.
Sergey Zagoruyko and Nikos Komodakis. 16. 2016.
Wide residual networks.
arXiv preprint
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A CONNECTION TO SDEDIT
Our work has a close and reciprocal connection to Meng et al. (2021) that we refer to as SDEdit for convenience. SDEdit has been proposed as an image editing algorithm that transforms guide image x69) which can contain different levels of details (c.g. coarse colored strokes colored strokes on real images) to a realistic image that is to some extent faithful to the guide. In short, the way SDEdit performs this task is the following:
1. Train normal diffusion model the original image dataset. Let s0 t) be the learned time-dependent score function.
2. Add noise to the guide image so that it looks like sample from an intermediate distribution of the diffusion process at time to E (0, 1):
( (z o (t))
3. Then start the reverse diffusion process from the noisy guide at time to (instead of starting from pure noise) to obtain realistic image which is conditioned on the guide via the starting point. Let
SDEdi((z()) to,
be the output of this process.
A critical hyper-pan parameter of this algorithm is to which is chosen empirically in Meng et al. (2021) from within [0.3, 0.6]. We show the following a bidirectional connection between SDEdit and the infiite-dimensional trajectory-based representation of our work.
From SDEdit -> Ours: Notice that the guide 2: can be an abstract semantic description of the image, e.g. the left-most column of Figure 4 in Meng et al. (2021) may be viewed as a semantic segmentation of the image which is intended to be generated. The optimal choice for to corresponds to the amount of noise added to the clean guide such that æ(9) is still faithful to the original guide but at the same time is similar to the original images contaminated with the same amount of noise. Let Po be the distribution of the original images and sx00) ~ Po be a clean image. The time 4o is chosen such that (to) Pt. In other words, both original image a(to) and the guide a=(0) (to) contains similar information content at time to- This means that by starting the forward diffusion process from the clean image x, the information content of the guide is most conspicuous ((t)). Put it differently, consider the reverse procedure of SDEdit which instead of making realistic image from the guide, the goal is to estimate the guide from the realistic image (e.g. doing semantic segmentation from clean image.) Assume the estimation (segmentation) model is free to run diffusion process on the clean image x and choose an intermediate time within (0, 1) which is best to predict the guide (semantic segmentation). The above reasoning implies that the model chooses to.
From Ours + SDEdit: The core idea of the present work was to investigate the information
content of the representations corresponding to different time steps along the trajectories of the diffusion process. This was done by defining a downstream task whose solution requires extracting some abstract information from the images (e.g. location, identity or colour of the objects in the image) allowing the prediction model for this task to look at the entire trajectory as the information source instead of choosing the most useful intermediate time step. Our empirical results show that when the model is free mix information from more than one time step, the mixing weight is not concentrated on a single time step and distributed smoothly over various time steps. The implication of this result for SDEdit is that instead of using 2(5) (to) as the starting point of the reverse diffusion process, a mixing of (())( (ty).j E J should be a more suitable starting point where J is the a subset of time indices to which our downstream estimation model has attributed large weights. In this
case, more suitable means that entails a better trade-off between faithfulness to the guide and realisticness of the final image as defined in Meng et al. (2021).
13

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
Score Model: We use the implementation of the score model from Song et al. (2021) using variance-exploding SDE and the NCSN+ configuration, in particular a smaller version of the CI- FARI0 configuration provided on their codebase. We augment the score-model with an Encoder Es which is implemented as the 28 x Mide-ResNett architecture (Zagoruyko & Komodakis, 2016) that maps the input with time embeddings to a vector in IR4, where is the dimensionality of the latent space and is set 128 unless otherwise specified. The time embeddings for the encoder model are implemented in the exact same way as for the score inputs, as outlined in Song et al. (2021). We use the learning rate of 10 to optimize the score network.
(MLP) based Classification model, we consider network with single hidden layer, ReLU activation function, and 512 neurons. For the Recurrent Neural Network (RNN) model, we use GRU with 256 hidden units and for the transformer system, use Multi-Head attention system with 4 heads and two layers, with weight sharing between the layers. For our attention profile based analysis settings, we consider the same Multi-Head attention system but only use single layer instead of two, as it allows make the score (averaged over heads) more interpretable.
Downstream Model: For Maalti-Layer-errcephee
We train all the downstream models with dropout of 0.25 and perform hyperparameter optimization for the learning rate over the set (0.001, 0.00075, 0.0005, 0.00025, 0.0001, 0.00005). In particular, we found the hyperparameter optimization important when considering the granularity analysis.
C CIFARIO, CIFARIO0 AND MINI-IMAGENET
We train the score model for 70,000 iterations and then the downstream models for 100 epochs. For the performance of the models, profiles, we use a single layered Transformer model.
2 2-layered Transformer model while for attention score
D SYNTHETIC
We train the score model for 250,000 iterations and then the downstream models for 1500 epochs. Figure 11 shows some samples obtained from this dataset, showcasing the different features present as well as the diversity of these different features.
We additionally perform the Jensen-Shannon Divergence analysis between different features for different granularities, as well as visualize the attention score profiles for the different granularities as well. Furthermore, we do the same analysis with both the types of encoders: VDRL and DRL.
Figure 11: Sam-
The corresponding plots for the attention score profiles are present in Figures 12 17 for different latent space dimensionalities, different granularities and the different types of encoding schemes (VDRL and DRL). Further analysis into the performance on different features with different granularities and dimensionalities can found Figure 18.
ples from Synthetic Dataset
E COLORED MNIST
We train the score model for 250,000 iterations and then the downstream models for 1500 epochs. Figure 26 shows some samples obtained from this dataset, showcasing the different features present as well as the diversity of these different features.
9
3
1
2
4
7
4
8
2
We additionally perform the Jensen-Shane Shannon Divergence analysis between different features for different granularities, as well as visualize the attention score profiles for the different granularities as well. Furthermore, we do the same analysis with both the types of encoders; VDRL and DRL..
5
8
q
4
Figure 26: Sam-
ples from Colored-
MNIST Dataset
14

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
cm
cm
641
Trasie
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
Background Color
Foreground Color
Location
Object Shape
Granularity: 10
0.54
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 12: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 2 and the VDRL. encoder.
15

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
1-r
cm
cm
641
Trasie
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
Background Color
Foreground Color
Location
Object Shape
Granularity: 10
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 13: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 16 and the VDRL encoder.
16

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
1-r
cm
641
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
Background Color
Foreground Calar
Location
Object Shape
Granularity: 10
0.54
Tmenss
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 14: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 32 and the VDRL encoder.
17

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
3mm
cm
cn
641
Sratidy
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
cm
Background Color
Foreground Color
Location
Object Shape
Granularity: 10
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 15: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 2 and the DRL encoder.
18

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
1-r
cn
641
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
CHI
Background Color
Foreground Color
Location
Object Shape
Granularity: 10
0.54
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 16: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 16 and the DRL encoder.
19

---

Under review as a conference paper at ICLR 2023
Background Color
Foreground Color
Location
Object Shape
Granularity: 2
1-r
cm
641
Buckyround Color
Foreground Color
Location
Object Shape
Granularity: 5
Background Color
Foreground Calar
Location
Object Shape
Granularity: 10
0.54
Background Color
Foreground Color
Location
Object Shape
Granularity: 25
Figure 17: Attention score profiles for the synthetic dataset on the different features, using different granularities, with the dimensionality of the latent space as 32 and the DRL. encoder.
20

---

Under review as a conference paper at ICLR 2023
Feature Foreground Color
Feature Sackground Color
Reature Location
Feature Object Shape
103
Granuaritt
10
15.0
Dimensionnitt
Dimensianiiity
Dimensionalty
(a) VDRL Encoder
Foreground
Bacaground
Shape
Granufarity
22.0
2.0
15.0
32.0
2.0
16.0
52.0
16.0 Dimensionaliy
32.0
Dimensknality
Dimonsinality
Dimensionality
(b) DRL. Encoder
Downstream performance plots for the Synthetic Dataset for different features, when the score model is trained with different latent dimensionality and the downstream models are trained with different granularities for discretization.
Figure 18:
The corresponding plots for the attention score profiles are present in Figures 19 24 for different latent space dimensionalities, different granularities and the different types of encoding schemes (VDRL and DRL). Further analysis into the performance different features with different granularities and dimensionalities can be found in Figure 25.
F CELEBA
We train the score model 250,000 iterations and then the downstream models for 100 epochs. We additionally perform the Cansen-Shannon Divergence analysis between different features for two different types of encoders; VDRL and DRL. The corresponding plots for these analysis, as well as for the attention score profiles and performances different features, present in Figures 27 30. The figures also enumerate the different attributes present in the dataset.
21

---

Under review as a conference paper at ICLR 2023
644
2.30
835
0.35
630
630
9.0
2
Ø
Timecrop
Tieecop
Background Color
Foreground Color
Digit
Granularity: 2
2.0
815
8.15
0.15
810
8.00
0.0
8.0
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.00
6.30
0.30
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.0
0.0
Tereates
Threates
Thraskz
Background Color
Foreground Color
Digit
Granularity: 25
Figure 19: Attention score profiles for the Colored-MNISS dataset on the different features, using different granularities, with the dimensionality of the latent space and the VDRL encoder.
22

---

Under review as a conference paper at ICLR 2023
2.30
835
0.35
630
630
9.0
2
Trnvetrp
Timecrop
Tieecop
Background Color
Foreground Color
Digit
Granularity: 2
2.00
815
8.15
0.15
810
8.10
0.09
8.0
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.00
6.30
0.00
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.00
0.0
Thredee
Thrcates
Thrakkz
Background Color
Foreground Color
Digit
Granularity: 25
Figure 20: Attention score profiles for the ColOTed-MNIST dataset on the different features, using different granularities, with the dimensionality of the latent space 16 and the VDRL encoder.
23

---

Under review as a conference paper at ICLR 2023
2.3
835
8.95
0.35
630
690
9.0
2
Ø
Timecrop
Tineeop
Background Color
Foreground Color
Digit
Granularity: 2
2.30
815
8.15
0.15
810
8.10
0.19
8.09
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.00
6.00
0.00
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.00
0.0
Tereaee
Threites
Thrakkz
Background Color
Foreground Color
Digit
Granularity: 25
Figure 21: Attention score profiles for the ColOTed-MNIST dataset on the different features, using different granularities, with the dimensionality of the latent space 32 and the VDRL encoder.
24

---

Under review as a conference paper at ICLR 2023
6.45
2.30
8.5
8.95
0.35
630
630
9.00
2
Ø
Timecrop
Tieeeop
Background Color
Foreground Color
Digit
Granularity: 2
2.20
8155
8.1
0.15
810
8.00
0.09
8.09
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.00
6.30
0.30
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.00
0.0
Tereates
Threates
Background Color
Foreground Color
Digit
Granularity: 25
Figure 22: Attention score profiles for the Colored-MNIIS dataset on the different features, using different granularities, with the dimensionality of the latent space and the DRL encoder.
25

---

Under review as a conference paper at ICLR 2023
6.4
2.30
835
0.35
630
630
9.00
2
Ø
Tinvetrp
Timecrop
Tieeeop
Background Color
Foreground Color
Digit
Granularity: 2
2.00
815
8.15
0.15
810
8.10
0.19
8.0
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.00
6.00
0.00
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.0
0.0
Tereaee
Threites
Thrakkz
Background Color
Foreground Color
Digit
Granularity: 25
Figure 23: Attention score profiles for the ColOTed-MNIST dataset on the different features, using different granularities, with the dimensionality of the latent space 16 and the DRL encoder.
26

---

Under review as a conference paper at ICLR 2023
645
2.3
835
8.95
0.35
630
690
9.00
2
Ø
Tinvetrp
Timecrop
Tieeeop
Background Color
Foreground Color
Digit
Granularity: 2
2.30
815
8.15
0.15
810
8.10
0.09
8.09
Background Color
Foreground Color
Digit
Granularity: 5
3.69
6.0
6.00
0.30
Background Color
Foreground Color
Digit
Granularity: 10
0.45
0.45
0.45
0.9
0.0
0.0
Tereaee
Threite
Thrakkz
Background Color
Foreground Color
Digit
Granularity: 25
Figure 24: Attention score profiles for the Colored-MNISS dataset on the different features, using different granularities, with the dimensionality of the latent space 32 and the DRL encoder.
27

---

Under review as a conference paper at ICLR 2023
Feature Foreground Color
Feature Background Color
Feature Digit
100
Performance
Granularity
5
40
20
16.0 Dimensionality
32.0
2.0
16.0
32.0
2.0
16.0 Dimensionalitty
32.0
Dimensionality
(a) VDRL Encoder
Feature Foreground Color
Feature Background Color
Feature Digit
100
Performance
Granularity
40
16.0
32.0
2.0
16.0
32.0
2.0
16.0
32.0
Dimensionality
Dimensionality
Dimensionallity
(b) DRL Encoder
Downstream performance plots for the ColOred-MNIST Dataset for different features. when the score model is trained with different latent dimensionality and the downstream models are trained with different granularities for discretization.
Figure 25:
28

---

Under review as a conference paper at ICLR 2023
Bro
Figure 27: Jensen Shannon Divergence plot for the attention profiles for any pair of features in the CelebA dataset when using the Left: VDRL Encoder, and Right: DRL Encoder.
capiie mas
Figure 28: Downstream performance plots for the different features in the CelebA dataset when
using the Left: VDRL Encoder, and Right: DRL Encoder.
29

---

Under review as a conference paper at ICLR 2023
du fu 40
dur
qun
4ux
ub
Figure 29: Attention Score profiles for different parts of the trajectory- -based representation CelebA when using the VDRL stochastic encoder.
30

---

Under review as a conference paper at ICLR 2023
Sun
dur tu 40
dur
qun
4ux
un
ub
Figure 30: Attention Score profiles for different parts of the trajectory- based representation CelebA when using the DRL deterministic encoder.
31


Reviews:
[{"review_id": "xXsUtGkJVq5", "reviewer": "Reviewer_ZUwW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the representation learned by diffusion models equipped with an encoder. Based on the prior work (Abstreiter et al 2021), the paper obtained a time-dependent encoder along with the training of a diffusion model. \n\nIn contrast to the prior work (Abstreiter et al 2021), which uses the output of the encoder at a single timestep (i.e. t0), the paper considers the discretized trajectory at multiple timesteps simultaneously. It is done by using an RNN or a transformer to aggregate the sequential representations together and make a final decision. \n\nEmpirically, it investigates the benefits of the trajectory information for downstream tasks, analyzes the different information encoded in the trajectory in different time steps by mutual information, parses the semantic information along the trajectory and demonstrates the benefits of using more samples in the trajectory.\n", "summary_of_the_review": "This paper is clearly below the acceptance bar of ICLR due to the lack of novelty and motivation. ", "strength_and_weaknesses": "**Strength**\n1. The paper is clearly written and well organized.\n\n**Weaknesses**\n1. The paper is incremental compared to the prior work (Abstreiter et al 2021). In fact, it is exactly the same to the prior work except that it considers to leverage the \"noisy\" representations along the trajactory simultaneously for downstream tasks.\n\n2. I think the claim of \"points to functions\" and \"infinite dimensional representations\" should be modified, including the title. This is very misleading because some popular algorithms in machine learning like Kernel methods can really do this in function space. However, The proposed method (using RNNs) takes representations at finite time steps as input instead of the whole function.\n\n3. I did not see a clear advantage of diffusion models to learn representations. In fact, the paper does not compare to any other deep generative models like VAEs and GANs or discuss about this. It it not well motivated for me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lK1ISl00TF", "reviewer": "Reviewer_PmAJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a trajectory of representations produced by a time-dependent encoder that leverages the diffusion-based method is more informative and can benefit downstream networks (RNN, transformers, etc.) that are able to handle the mixture of the representations. The authors empirically demonstrate their idea on synthetic data and real-world image datasets such as CIFAR-10, mini-ImageNet, colored-MNIST and CelebA.", "summary_of_the_review": "The topic studied in this paper is important in the study of representation learning and diffusion-based method. However, some key perspectives regarding the proposed method are not comprehensively studied and remain unclear to the readers, which weakens the contribution of this paper. Moreover, the experimental details are not clearly provided. I am inclined to consider this paper below the acceptance threshold given the current version.  ", "strength_and_weaknesses": "**Strength**\n\nThe paper is well-written and organized, making it easy to follow. The claims in the paper are well supported by the corresponding experiments. The authors not only demonstrate the trajectory combination of representations is stronger, but also show which intermediate steps are more important. This observation is important to both diffusion-based model and representation learning community.\n\n**Weakness**\n\nThis paper only provides empirical studies on the representations, but lacks theoretical analysis on why the combination of representations is better. \n\nMost of the method part is heavily dependent on Abstreiter et al. and some claims are duplicated, which may weaken the contribution and novelty.\n\nThe authors only demonstrate their observations on one single diffusion schedule, while there are various options that may affects the observations. If the diffusion adopts a cosine schedule, can we still get the same conclusions? For example, can we still get the highest attentions scores around t=0.5, as shown in Fig. 3?  \n\nOn the effects of granularity, the timesteps are only considered to be uniformly partitioned. There could be more combinations to explore. For example, we can sample steps according to a cosine annealing decay weight for the combination. \n\nThe accuracy shown in the experiments are relatively lower than the results in both supervised and self-supervised cases. It is hard to say this gap is caused by the method or by using a weaker encoder backbone (I could not find the parameterization details of the Wide-ResNet encoder). It is unclear whether such observation could be helpful in the practical case. \n\nThe diffusion schedule is not specified, and some experimental details are not precise; please also see the clarity part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IezE10DPA9", "reviewer": "Reviewer_tLzz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way to solve downstream tasks using diffusion-based representation learning. The authors observe that representation learning inspired by diffusion doesn't give one, but infinitely many representations of the image (one for each time, t, of the diffusion). The work that introduced diffusion-based representation learning used one of those representations to optimize for a downstream task. Instead, this paper proposes to consider the whole trajectory of representations. Particularly, the authors propose to use a Transformer (or an RNN) to map from the trajectory to a single embedding that is then used to solve downstream tasks. Experimentally, this yields improved performance in a wide variety of settings.", "summary_of_the_review": "The paper proposes a better way to solve downstream problems with diffusion-based learned representations. The key innovation is to use the whole trajectory and not a single point to extract a representation that will be used to solve downstream tasks. The method is not evaluated against other ways to learn representations and hence its practical relevance is not clear. ", "strength_and_weaknesses": "Strengths:\n\n* The idea of the paper is simple, yet effective.\n* The paper is well-written.\n* The authors experiment in a wide variety of settings, from synthetic tasks to more realistic tasks, such as CelebA classification.\n* The authors demonstrate (using the attention profiles and the estimated Mutual Information) that the representations of the image are changing over the diffusion time.\n\nWeaknesses:\n\n* The paper does not compare with other methods of representation learning, e.g. Contrastive Learning. Given that the diffusion-based representation learning paper itself has not yet been published, I think it would be useful to compare this new work to more established methods for learning representations.\n* There is a lack of clarity for some things regarding the implementation. Why did the authors need to retrain the models from Abstreiter et. al? Aren't there open-source models available that could have been reused? If not, which models did the authors train? Is this the NCSN++ model with the Variance Exploding SDE (as it is implied from the Background Section)? Would anything change if instead of the Variance Exploding SDE, one used Variance Preserving or the sub-VP SDE?\n* In the experiments of Figure 1 (but also in subsequent experiments), it seems that we are comparing a Transformer that takes as input the whole trajectory vs an MLP that takes as input a single point. Does the transformer and the MLP have the same number of parameters? If the MLP has much less expressive power than the Transformer, it could just be that the observed benefits over the baseline are due to bigger or more powerful architecture (and not because of the trajectory vs single point).\n* It would have been more impactful to report numbers on more standard benchmarks such as CIFAR10 , CIFAR100 or ImageNet. This would give a better sense of how this method compares to other methods for learning representations (e.g. SimCLR).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7lHThqFJd_", "reviewer": "Reviewer_WYmT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses representation learning with diffusion models. The key idea is from Abstreiter et al., where a diffusion model is additionally trained on conditioning information given by an encoder. The benefit of this is that is becomes possible to minimize the denoising score matching objective to zero. This paper observes that different representations are learned at different timesteps of the diffusion, and measures the information contained about in the representations, finding that mid-points of the trajectory would give better performance on downstream applications. The paper also investigates other semantic information encoded in these representations.  ", "summary_of_the_review": "The paper performs an empirical study over diffusion-based representation learning. While the experiments are extensive, there are not really a lot of technical depth, nor critically useful empirical applications that can be learned from this paper. Even with the entire trajectory of representations, the downstream performance simply does not beat older methods such as contrastive learning. I am also not entirely convinced by this focus on \"infinite-size\" representations in the title and in parts of the paper -- coming up with \"infinite-size\" representations is easy (simply add redundancy would work), but such a representation has to be useful for downstream tasks. ", "strength_and_weaknesses": "Strength:\n- The paper is concise and easy to understand. \n- The paper presents extensive empirical studies over the representations, such as the timestep-dependent attention score, mutual information, and downstream application performance.\n\nWeakness:\n- Lack of technical novelty: both of the methods DRL and VDRL come from the Abstreiter paper, and it comes not at a surprise that the representations obtained from such models have infinite dimension. The paper mostly presents experimental evaluation results on this type of representations.\n- Utility of the representations are not obvious: there are fairly powerful representation learning methods that does not even require a diffusion model. For example, in Figure 1 and Figure 9, the classifier accuracy on CIFAR10 is lower than 80% whereas contrastive learning methods could easily achieve over 90% accuracy with linear classifiers. While the observations are interesting, it is not clear why we want to use diffusion representations for such downstream applications (why not do time-dependent contrastive learning?). Also, there are already works that try to learn diffusion-like methods with powerful representations [1]. All in all, the encoder learned here does not seem very useful empirically.\n- Lack of relevance towards existing diffusion models: probably the most interesting part is Figure 5 (c/f), where it suggests that digit identity is highly correlated with large timesteps. However, it is unclear that these findings can be generalized to \"standard\" diffusion models. \n\n[1] D2C: Diffusion-Denoising Models for Few-shot Conditional Generation, NeurIPS 21.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject