Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new way to solve downstream tasks using diffusion-based representation learning. The authors observe that representation learning inspired by diffusion doesn't give one, but infinitely many representations of the image (one for each time, t, of the diffusion). The work that introduced diffusion-based representation learning used one of those representations to optimize for a downstream task. Instead, this paper proposes to consider the whole trajectory of representations. Particularly, the authors propose to use a Transformer (or an RNN) to map from the trajectory to a single embedding that is then used to solve downstream tasks. Experimentally, this yields improved performance in a wide variety of settings.
strength_and_weaknesses: Strengths:

* The idea of the paper is simple, yet effective.
* The paper is well-written.
* The authors experiment in a wide variety of settings, from synthetic tasks to more realistic tasks, such as CelebA classification.
* The authors demonstrate (using the attention profiles and the estimated Mutual Information) that the representations of the image are changing over the diffusion time.

Weaknesses:

* The paper does not compare with other methods of representation learning, e.g. Contrastive Learning. Given that the diffusion-based representation learning paper itself has not yet been published, I think it would be useful to compare this new work to more established methods for learning representations.
* There is a lack of clarity for some things regarding the implementation. Why did the authors need to retrain the models from Abstreiter et. al? Aren't there open-source models available that could have been reused? If not, which models did the authors train? Is this the NCSN++ model with the Variance Exploding SDE (as it is implied from the Background Section)? Would anything change if instead of the Variance Exploding SDE, one used Variance Preserving or the sub-VP SDE?
* In the experiments of Figure 1 (but also in subsequent experiments), it seems that we are comparing a Transformer that takes as input the whole trajectory vs an MLP that takes as input a single point. Does the transformer and the MLP have the same number of parameters? If the MLP has much less expressive power than the Transformer, it could just be that the observed benefits over the baseline are due to bigger or more powerful architecture (and not because of the trajectory vs single point).
* It would have been more impactful to report numbers on more standard benchmarks such as CIFAR10 , CIFAR100 or ImageNet. This would give a better sense of how this method compares to other methods for learning representations (e.g. SimCLR).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. Certain parts of the experimental evaluation should be further clarified (see above). The proposed idea is original but incremental to the diffusion-based representation learning paper.
summary_of_the_review: The paper proposes a better way to solve downstream problems with diffusion-based learned representations. The key innovation is to use the whole trajectory and not a single point to extract a representation that will be used to solve downstream tasks. The method is not evaluated against other ways to learn representations and hence its practical relevance is not clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the representation learned by diffusion models equipped with an encoder. Based on the prior work (Abstreiter et al 2021), the paper obtained a time-dependent encoder along with the training of a diffusion model. 

In contrast to the prior work (Abstreiter et al 2021), which uses the output of the encoder at a single timestep (i.e. t0), the paper considers the discretized trajectory at multiple timesteps simultaneously. It is done by using an RNN or a transformer to aggregate the sequential representations together and make a final decision. 

Empirically, it investigates the benefits of the trajectory information for downstream tasks, analyzes the different information encoded in the trajectory in different time steps by mutual information, parses the semantic information along the trajectory and demonstrates the benefits of using more samples in the trajectory.

strength_and_weaknesses: **Strength**
1. The paper is clearly written and well organized.

**Weaknesses**
1. The paper is incremental compared to the prior work (Abstreiter et al 2021). In fact, it is exactly the same to the prior work except that it considers to leverage the "noisy" representations along the trajactory simultaneously for downstream tasks.

2. I think the claim of "points to functions" and "infinite dimensional representations" should be modified, including the title. This is very misleading because some popular algorithms in machine learning like Kernel methods can really do this in function space. However, The proposed method (using RNNs) takes representations at finite time steps as input instead of the whole function.

3. I did not see a clear advantage of diffusion models to learn representations. In fact, the paper does not compare to any other deep generative models like VAEs and GANs or discuss about this. It it not well motivated for me.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is clear.

Quality: the quality of the paper is limited. See details in the weaknesses above.

Novelty: the novelty is limited compared to the prior work (Abstreiter et al 2021).

Reproducibility: it seems reproducible.
summary_of_the_review: This paper is clearly below the acceptance bar of ICLR due to the lack of novelty and motivation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a trajectory of representations produced by a time-dependent encoder that leverages the diffusion-based method is more informative and can benefit downstream networks (RNN, transformers, etc.) that are able to handle the mixture of the representations. The authors empirically demonstrate their idea on synthetic data and real-world image datasets such as CIFAR-10, mini-ImageNet, colored-MNIST and CelebA.
strength_and_weaknesses: **Strength**

The paper is well-written and organized, making it easy to follow. The claims in the paper are well supported by the corresponding experiments. The authors not only demonstrate the trajectory combination of representations is stronger, but also show which intermediate steps are more important. This observation is important to both diffusion-based model and representation learning community.

**Weakness**

This paper only provides empirical studies on the representations, but lacks theoretical analysis on why the combination of representations is better. 

Most of the method part is heavily dependent on Abstreiter et al. and some claims are duplicated, which may weaken the contribution and novelty.

The authors only demonstrate their observations on one single diffusion schedule, while there are various options that may affects the observations. If the diffusion adopts a cosine schedule, can we still get the same conclusions? For example, can we still get the highest attentions scores around t=0.5, as shown in Fig. 3?  

On the effects of granularity, the timesteps are only considered to be uniformly partitioned. There could be more combinations to explore. For example, we can sample steps according to a cosine annealing decay weight for the combination. 

The accuracy shown in the experiments are relatively lower than the results in both supervised and self-supervised cases. It is hard to say this gap is caused by the method or by using a weaker encoder backbone (I could not find the parameterization details of the Wide-ResNet encoder). It is unclear whether such observation could be helpful in the practical case. 

The diffusion schedule is not specified, and some experimental details are not precise; please also see the clarity part.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is overall clear. I list the points that are not clear to the readers below:

Equation (5) looks similar to proposition 1 in Abstreiter et al. Is this result from there or derived by the authors? Could the authors provide the corresponding source, e.g., the reference or the proof? 

The attention score used in Fig. 3 is not mathematically defined. It is not clear the attention score is calculated with which variables.

The model architecture is not clarified. For example, what is the number of layers of the Wide-ResNet? 



**Novelty**

The proposed method has novelty in leveraging the combination of the time-dependent representations, while the method part is extended from previous works like Abstreiter et al. 

**Quality**

The paper provides reasonable empirical studies and lacks theoretical analysis. Some important ablations may be in need to improve the paper quality.

**Reproducibility**

The authors include some details for the algorithm and model parameterization, but the provided details are insufficient for reproducing their results.

summary_of_the_review: The topic studied in this paper is important in the study of representation learning and diffusion-based method. However, some key perspectives regarding the proposed method are not comprehensively studied and remain unclear to the readers, which weakens the contribution of this paper. Moreover, the experimental details are not clearly provided. I am inclined to consider this paper below the acceptance threshold given the current version.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses representation learning with diffusion models. The key idea is from Abstreiter et al., where a diffusion model is additionally trained on conditioning information given by an encoder. The benefit of this is that is becomes possible to minimize the denoising score matching objective to zero. This paper observes that different representations are learned at different timesteps of the diffusion, and measures the information contained about in the representations, finding that mid-points of the trajectory would give better performance on downstream applications. The paper also investigates other semantic information encoded in these representations.  
strength_and_weaknesses: Strength:
- The paper is concise and easy to understand. 
- The paper presents extensive empirical studies over the representations, such as the timestep-dependent attention score, mutual information, and downstream application performance.

Weakness:
- Lack of technical novelty: both of the methods DRL and VDRL come from the Abstreiter paper, and it comes not at a surprise that the representations obtained from such models have infinite dimension. The paper mostly presents experimental evaluation results on this type of representations.
- Utility of the representations are not obvious: there are fairly powerful representation learning methods that does not even require a diffusion model. For example, in Figure 1 and Figure 9, the classifier accuracy on CIFAR10 is lower than 80% whereas contrastive learning methods could easily achieve over 90% accuracy with linear classifiers. While the observations are interesting, it is not clear why we want to use diffusion representations for such downstream applications (why not do time-dependent contrastive learning?). Also, there are already works that try to learn diffusion-like methods with powerful representations [1]. All in all, the encoder learned here does not seem very useful empirically.
- Lack of relevance towards existing diffusion models: probably the most interesting part is Figure 5 (c/f), where it suggests that digit identity is highly correlated with large timesteps. However, it is unclear that these findings can be generalized to "standard" diffusion models. 

[1] D2C: Diffusion-Denoising Models for Few-shot Conditional Generation, NeurIPS 21.
clarity,_quality,_novelty_and_reproducibility: - Clarity (High). The paper is written clearly with many details provided.
- Quality (High). The writing of the paper is of high quality.
- Novelty (Mid-to-low). The core method is from an earlier papers, and the experimental evaluations does not seem to reveal conclusions that are too surprising. 
- Originality (High). The work in itself is original as far as I understand. 
summary_of_the_review: The paper performs an empirical study over diffusion-based representation learning. While the experiments are extensive, there are not really a lot of technical depth, nor critically useful empirical applications that can be learned from this paper. Even with the entire trajectory of representations, the downstream performance simply does not beat older methods such as contrastive learning. I am also not entirely convinced by this focus on "infinite-size" representations in the title and in parts of the paper -- coming up with "infinite-size" representations is easy (simply add redundancy would work), but such a representation has to be useful for downstream tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thank you for your submission to ICLR.  The reviewers and I are in agreement that although there are some interesting aspects to the proposed work, ultimately the paper seemed rather incremental relatively to the Abstreiter et al., 2021 paper: specifically the main contribution of this paper seems to be that the representation learning of this past work can perform better if a mix of information along the entire trajectories is used instead of just the final state.  This is an interesting insight, but seems to be more suitable to a technical note or part of a larger whole paper (for example, if similar approaches could be applied to other representation learning settings), rather than an independent paper by itself.  Thus, the consensus of the reviewers was that this paper wasn't suitable for ICLR in its current form.
justification_for_why_not_higher_score:  The paper ultimately appears rather incremental, both in the quality of the results and methodologically from the previous Abstreiter et al., 2021 paper.
justification_for_why_not_lower_score:  NA
Decision: Reject