Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors leverage the sparse manifold transform (SMT) and provide an insight into its merits on unsupervised learning. The authors show how SMT can be applied to the manifold disentanglement problem. As is pointed out, SMT as a white-box method can be an alternative to black box and self-supervised models building on the principles of parsimony namely sparsity and low-rank.
strength_and_weaknesses: Sparse manifold transform is an approach proposed for signal representation in [20]. SMT is a white box method, meaning that it consists of interpretable steps i.e., sparse coding and low-dimensional spectral embedding. The authors build on SMT showing its efficiency in the manifold disentanglement problem. Moreover, they provide experimental results on MNIST, CIFAR10 and CIFAR100 showing promising performance of SMT as compared to SOTA SSL methods e.g. SimCLR and VICReg. 
clarity,_quality,_novelty_and_reproducibility: The writing could be improved in several parts of the paper. For instance, the authors do not clearly state the contributions of the paper I introduction. Moreover, a large part of sections 1 and 2 focuses on the ideas of SMT which is method introduced in [20]. This makes sense to a certain extend for making the paper  self-contained. However, the authors should also make clear the key contributions of their work. Also, subsection ''SMT representation for MNIST and CIFAR" should be at the empirical results' section.
summary_of_the_review: The authors leverage the sparse manifold transform showing its potential as a white-box unsupervised learning method. The main contribution of the paper the use of SMT in manifold disentanglement problem and its probabilistic point of view interpretation for co-occurence modeling. My main comments are the followings:

1) The overall presentation of ideas could be improved making clear the contributions of the current paper (see above).
2) The experimental results show promising performance of the methods on MNIST and CIFAR datasets. Do the authors believe that a similar performance could be obtained in more complicated and high-dimensional datasets such as IMAGEnet? Specifically, are there any ideas on how to efficiently learn the sparsifying transform in such cases? Also, how practical is the implementation of compositional models that the authors hint on the manuscript?
3) An  advantage of SMT as compared to alternative SSL methods is its white-box nature. What is the trade-off between the "simplicity" of the method and  modifications needed for making it efficient  in more complicated datasets such as IMAGEnet? 


------------------------------
Post-rebuttal Update:
I want to thank the authors for their time and effort in responding to reviewers' comments and addressing their concerns. At this time, I will keep my scope the same.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the application of the Sparse Manifold Transform (SMT) to image classification problems. The SMT is a shallow transform that first sparsely represents the input data/image in a high-dimensional feature space (via a non-linear mapping) and then linearly embeds these sparse representations in a low-dimensional space. The authors work out the details of how to apply the SMT to image data and evaluate the method on MNIST and CIFAR-10/100, showing good results. In addition, some of the design choices are ablated.
strength_and_weaknesses: I think the overall direction of the paper is interesting and exploring ways to construct simple, interpretable models performing competitively with highly-overparameterized, very large models is a worthwhile endeavor. In more detail, I see the following strengths and weaknesses.

*Strengths:*
- The method is quite simple and produces surprisingly good results given its simplicity.
- The paper makes a good effort in describing the SMT and giving an intuitive understanding using the manifold disentanglement problem (Figure 2).
- The paper presents a number of interesting ablations.

*Weaknesses:*
- As acknowledged by the authors KNN, the reliance of their method on KNN limits the scalability of the method. This could be discussed in more detail: What does this mean in terms of compute and memory requirements? Also, have the authors considered training a small model (linear classifier, MLP) on top of the SMT representation? This could lead to more interesting compute/memory vs accuracy tradeoffs. Also what would be the effect of increasing resolution? The datasets considered by the authors have tiny images.
 -The paper could make a better case for interpretability. Instead of just looking at patches with similar embeddings, what are the nearest neighbors for misclassified examples? Which patches led to misclassification? Is there anecdotal evidence of the SMT allowing us to understand incorrect classifications?
- There are a number of prior works investigating engineered, principled transforms as a substitute for deep end-to-end learned black-box methods. These usually evaluate on MNIST and CIFAR-10/100, so comparing performance and methods would be possible. For example the scattering transform [a] and kernel methods [b].


[a] Oyallon, Edouard, and Stéphane Mallat. "Deep roto-translation scattering for object classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.

[b] Shankar, Vaishaal, et al. "Neural kernels without tangents." International Conference on Machine Learning. PMLR, 2020.

clarity,_quality,_novelty_and_reproducibility: *Clarity:* I feel that the paper uses quite a few buzzwords without defining their precise meaning (or at least to me these meanings were pretty unclear) in the paragraph “The sparse manifold transform”. For example on p3 “Similarity can be considered as a first-order derivative.”: A derivative of what w.r.t to what? Other examples: What is a “path in patch space”? What is a feature space “where locality and decomposition are respected”? In particular the I’m not sure if I understand what decomposition means in the context of image patches.

*Quality:* Please see strengths and weaknesses above.

*Novelty:* I could not find any references that apply the SMT to the image classification problems considered in this paper. However, the SMT is proposed in prior work, so the novelty is somewhat limited.

*Reproducibility:* Most of the details are explained well. However, I could not find details about the mentioned softmax-KNN classifier (e.g. how many neighbors are used?). Furthermore, as far as I can tell the fact that the EM algorithm is used to learn the 1-sparse feature representation is only stated in the appendix, and it would be useful to make this clear earlier. Finally, when applying GloVe to MNIST, how were the images tokenized?


*Typos:*
- p2: “representation” instead of “re-presentation”
- p4: “d-dimensional vector” instead of “d-dimensional vectors”
- p7: “colorjitter” instead of “colorgitter”

summary_of_the_review: The paper makes an interesting attempt at applying the SMT to image classification and shows quite strong results. The overall direction of the paper seems relevant and interesting to me. 

The clarity of the paper could be improved, scaling aspects should be discussed in more detail, and the method should be compared to more related work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a method to learn an embedding from sparse coding and dictionary learning through the sparse manifold transform. The authors show that this “white-box model” performs well compared to Deep Learning methods on classical image datasets like MNIST and CIFAR.
strength_and_weaknesses: **Strengths**

The method proposed in the paper, which is an extension of SMT, allows to embed data from sparse coding and dictionary learning. The model is interesting because it is simple and seems to work well compared to SOTA Deep Learning methods on classical datasets. This might be of interest for people working on self-supervised learning and dictionary learning on various types of signals.

**Weaknesses**

- The contributions are not always clearly highlighted. I don’t understand whether Part 2 contains new ideas concerning SMT or not. In particular, the authors claim to “revisit the formulation of SMT” in the introduction. Could this be more specific ?

- The authors use 1-sparse sparse coding to learn the dictionary D and the representation codes in the experiments.  As described in the paper, this corresponds to performing a k-means algorithm. Figure 6 in appendix highlights that the number of atoms has to be extremely large (at least 8000 atoms of dimension 32) to achieve optimal performance, and I wonder if this is due to the 1-sparse constraint. Why focusing only on this case? This seems very restrictive, especially for complex data. It would be interesting to increase the number of non zero coefficients (even to consider relaxations of the $\ell_0$ constraint for efficiency purposes as it seems to be an issue for CIFAR datasets, see Mairal et al. 2009) to see whether this allows to reduce the dimension of the dictionary without impacting the performance in terms of knn accuracy and computation time (same kind of experiments as in Figure 6 in appendix). This might also help doing better than randomized dictionaries in CIFAR experiments. In my opinion, this study would improve the quality of the paper.

- There is no discussion about the impact of the number of samples on performance. Would it be possible (and interesting from the authors point of view) to integrate figures providing the knn accuracy with respect to the number of images to see whether SMT performs well with small datasets compared to baseline methods?

Minor remarks:
- In Table 2, the authors claim that “the MNIST patch space can be viewed as a 3-manifold”. I don’t understand how this can be inferred from the results shown in the table. Could the authors elaborate on that point ?
- There is not enough discussion about the impact of the context range, especially in the experiments on CIFAR, and it seems to be an important hyper-parameter of the model. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and provide interesting explanations, examples and references from the literature. There are a few typos and misspellings in Part 3.

Up to my knowledge, this work presents new insights on the way SMT can be used for embedding. 

As there is no source code, it would be nice to add some details on training procedures and hyper-parameters, especially for deep learning methods used as baselines. The paper may not be reproducible as is.
summary_of_the_review: Even though I think that the 1-sparse case is restrictive and that the paper lacks a bit of experimental grounding and details, the idea seems new and interesting in the context of SSL and dictionary learning, and is worth being highlighted.

------------------------------------

The authors added relevant details and experiments to the paper, and responded to my requests for clarification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a barebones version of unsupervised representation learning framework based on the sparse manifold transform (SMT), which according to the authors, helps in understanding SOTA self-supervised representation learning (SSL) methods (such as VICReg). SMT has two steps: (1) sparse coding to map into a high-dimensional space and (2) manifold learning: a low-dimensional embedding that takes neighborhood/co-occurrence structure into account. Both sparsity (with respect to a dictionary) and local neighborhoods/co-occurrences serve as important features of natural signals like images, and even words (the sparsity is trivial).  While SMT is not a contribution of this paper, they show that SMT produces representations that capture the essence of unsupervised representation learning (SMT is a minimal model in this sense that it is a minimal model that takes both sparsity and neighborhood structure into account). The authors make connections with SOTA by pointing out that a deep neural encoder takes the place of sparse coding and the SMT optimization is very similar to that of VICReg. In principle, even SMT could be stacked in order to produce hierarchical representations that are necessary for larger natural images. Experiments on small-sized images in CIFAR illustrate that even SMT can produce representation that rival the SOTA. 

strength_and_weaknesses: Strengths:

1. The paper studies an important problem which is to provide some understanding of SOTA SSL methods, and proposes to use SMT which is a simple model using a single-level dictionary learning (which is well understood) and a single linear spectral embedding (which is also well understood in manifold learning literature).

2. The paper is written well with ample explanations, connections with plenty of related work and illustration on a toy dataset. 

3. Experiments on MNIST and CIFAR show results that to me are surprising. SMT performs comparably with SOTA SSL methods (when some data augmentations are turned off). 

Weaknesses:

1. I don't quite understand why SMT cannot work with all data augmentations. Perhaps the authors can explain this point better.

2. Beyond empirical analysis, I am not seeing how this minimal setup is helping explain SOTA SSL methods. If I look at SimCLR, for example, the features are derived at the image level using augmentations as positives and other images as negatives. I don't see that happening with SMT. So, I would appreciate a clearer explanation as to how having this minimal SMT model helps understand SimCLR or VICReg.

3. I think related to (2), maybe a longer discussion on how to connect SMT and VICReg optimization and loss functions more concretely can help. I am not very familiar with VICReg, so I think this would definitely help me. 

4. Besides SimCLR and VICReg, there are several other SSL models such as Masked Autoencoders, does SMT also help in understanding these models?

5. Also, are the authors saying that if there is a way to easily stack more layers of SMT in order to learn hierarchical representations, the gap between SMT and VICReg (for example) would not exist?
clarity,_quality,_novelty_and_reproducibility: The paper is clear in terms of its claims and the experimental results back up the claims. I believe the experiments are reproducible. The main novelty of the paper is building a minimal model for unsupervised representation learning using SMT (which is an existing method). Compared to the SOTA SSL methods, they show that this method performs well. Furthermore, this minimal model is to be seen as a simplified version of VICReg, thus leading to better understanding of methods like VICReg which I believe is interesting and novel. However, for the last point, I believe the authors provide a more elaborate explanation. Overall, this is a good paper that I am inclined to accept. 
summary_of_the_review: I think the paper uses SMT which is a simple and easy-to-understand model in a novel setting in order to gain understanding of SSL methods, which is an interesting and important research direction. Experimental results on the small image datasets are convincing and do show that SMT is indeed a strong model in spite of being simple, which is great. However, I have noted some weaknesses that I hope the authors can respond to. Overall, this is a nice paper with interesting ideas and results and I am happy to increase my score based on the authors' response. 

UPDATE AFTER AUTHOR RESPONSE:

I greatly appreciate the authors taking the time to significantly edit and improve the paper. Appendix K in particular helps me understand the connection between SMT and VICReg a lot better, and the loss functions indeed are similar. The minimal model does help illustrate that the ideas of SMT are useful in learning good SSL features. How these ideas fully translate to deep models trained with SGD with data augmentations is not clear, but this seems to be a good step in that direction. Perhaps if it is possible to train shallow (a few layers (?)) VICReg and showing the learned features are similar to SMT features in some sense, the connection would be even stronger. Overall, I am increasing my rating and recommending acceptance of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase. The response helped resolve many important concerns (e.g., clarification of the contributions, add study of linear probing, comparisons with manually-engineered transforms, evaluation in the low-sample regime).

The reviewers and meta reviewer all acknowledge that the submission proposes a simple (yet effective) and relevant approach to better understand fundamental components of SOTA SSL methods. The submission features as well a rich related work discussion and numerous insightful ablation studies (further improved post rebuttal).

All in all, the submission is recommended for acceptance.
justification_for_why_not_higher_score: * Still some gap to be bridged with SOTA approaches
justification_for_why_not_lower_score: * Unanimous support of the reviewers (and confidently so)
* Provide valuable insights to better understand SOTA SSL approaches
* Simple!
* Insightful discussions in the light of the related literature
* Likely to inform future research
Decision: Accept: notable-top-25%