Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors consider "joint embedding" self-supervised learning (SSL) formulations applied to a kernel learning setting.  That is, finding / learning a linear mapping in the feature space (induced by some kernel) that minimizes commonly used SSL learning objectives, such as for contrastive learning.

The authors prove properties about this mapping function and the solution to such problems, and derive closed form optimal solutions to specific formulations (objectives), as well as a semi definite programming formulation to solve a general objective over batches.

The authors experimentally explore the SSL kernel induced kernels - comparing the induced kernel from the SSL formulation (which leverages an adjacency matrix as part of fitting the kernel space mapping / the SSL objective) with the base kernel (RBF kernels used with NTK in the appendix).  They compare qualitatively by plotting the similar pairs under each kernel on the spiral toy dataset, and compare classification accuracy and for varying hyper parameters / settings. 
strength_and_weaknesses: Strengths:
1. It is an interesting idea and direction to consider contrastive / SSL representation learning approaches from the perspective of kernel learning.

2. Several useful properties are proven about the SSL kernel formulation, and solutions derived (including closed-form) for SSL formulations / objectives - which could be useful to researchers and practitioners.

3. Analysis and experiments were formed to better understand ad explain the SSL induced kernels.

Weaknesses:
1.  The paper feels a bit incomplete - the properties developed are not unexpected, and the intro argues for understanding of common modern SSL representation learning methods.  However, the paper really just presents a kernel algorithm for SSL and there does not really seem to be a connection to deep learning based SSL learning methods.

Essentially it's not really clear what is the real goal or objective of the paper.  If it's to propose a new SSL representation learning algorithm using kernel representations, then it needs more experiment evaluation to demonsrate its benefits over existing approaches.  If instead it's to reveal insights about current SSL representation learning approaches and losses used - this was also not accomplished.  As such, it doesn't feel complete.

2. Experiments seem lacking.  Comparison with more methods and more datasets would be better.  As mentioned above, if the point is a new kernel SSL algorithm - more thorough comparison with other kernel algorithms and with existing representation learning algorithms on multiple datasets and tasks should be done - not just a simple classification task on EMNIST and MNIST comparing only to using the standard SVM classifier with RBF kernel.   If the point is to understand modern SSL algorithms, experiments using those algorithms and showing how the proposed method helps understand their results should be included.

Additionally the classification results do not seem very convincing for the benefit of using the proposed approach - in particular if I understand it correctly, Figure 3 show that the accuracy by simply using augmented training data as opposed to the induced kernel based on the augmentations consistently yields higher test accuracy.


3. The results / resulting formulations and properties are not very surprising / seem somewhat expected, including the method itself, and seem similar and closely related to much prior work on learning on graphs or manifolds (using kernels).  It's not completely clear to me how this brings much novelty beyond past kernel approaches to learning across a graph / manifold - as these use the graph laplacian / manifold assumption in a similar way.  I think more discussion and comparison is warranted.  Baselines for comparing prediction results for a relationship-informed kernel learning representation approach should include such laplacian / manifold informed versions of the corresponding estimators, like graph / manifold regularized SVM.

Examples of such work:
- "Kernels and Regularization on Graphs" Smola and Kondor.
- "Learning on Graph with Laplacian Regularization" Ando and Zhang
- "Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples" Belkin et al.


4. Minor comments: 
- In the introduction, one type of self supervised learning missing is the other major class of SSL not mentioned - predicting held-out parts of the input data (as in transformer models).
- Figure 1 does not seem very helpful to me and doesn't seem to explain anything. 


clarity,_quality,_novelty_and_reproducibility: The paper is clear and good quality development of the approach and formulations, but the message of the paper is not very clear and seems like the intro and motivation do not match the body and experiments of the paper.  I think the idea of considering modern SSL loss using kernel representations is novel - so that is original - but the particular idea and formulation seems straight forward and inline with past work on kernel learning on graphs and manifolds.

Reproducibility as high as code is provided.
summary_of_the_review: While the development of the properties and formulations, and closed form solutions for this approach to learning and induced SSL kernel could be useful, overall I feel more like the paper is incomplete and lacks a coherent purpose and objective to achieve, and so falls short.  I think it could benefit from more experiments and comparisons, and elucidating the connection with current SSL approaches and what it reveals about them (as motivated in the introduction).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies self-supervised learning (SSL) with kernels. The authors prove the representer theorem for SSL and provide explicit expressions of the optimal induced kernel for some SSL methods using contrastive or non-contrastive loss functions. The prediction accuracy on downstream tasks is analyzed based on the complexity quantity related to kernels. In numerical experiments, the authors investigate hyper-parameter tuning and the relation between data augmentation and generalization property. 
strength_and_weaknesses: Strength
- The paper considers SSL with a linear map on an RKHS. Due to the restriction of the problem setting, the authors provide simple theoretical results that promote an intuitive understanding of SSL. The problem falls into a simple optimization problem. An explicit form of the solution is presented for non-contrastive/contrastive loss. 

Weaknesses
- Nowadays, SSL is widely used for learning tasks with large data sets. Considering such a situation, the scope of this paper is relatively narrow. This paper focuses on SSL with kernel features. Though Proposition 3.1 deals with a general class of loss functions, the theoretical analysis is provided only for VICReg loss and spectral contrastive loss.
- In analyzing the generalization performance of downstream tasks, the complexity quantity s_N(K) is considered. It is unclear how the complexity quantity relates to the concept of alignment and uniformity [1], which are regarded as important features in SSL.[1] Wang, and Isola, Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML,2020.
- In numerical experiments, feature extraction with RBF kernel is reported. However, most readers will be interested in feature extraction using deep neural networks (DNNs). The authors showed the SSL with neural tangent kernel (NTK) regime in the appendix. The authors should conduct intensive numerical studies using DNN and investigate how much their theoretical findings for the NTK regime explain numerical results. Though the authors discussed it in section 5, the numerical results in the present paper are not very informative in understanding the usefulness of SSL in practice. 

clarity,_quality,_novelty_and_reproducibility: The paper is not hard to follow. Since the applicability of theoretical results seems relatively narrow, I think that the paper does not have an impact that much. 
summary_of_the_review: Though the theoretical results in the paper are rigid, the scope of the paper seems narrow.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Modern methods in SSL form representations based on known or constructed relationships between samples and are particularly effective in this task. Here, this paper aim to extend this framework to incorporate algorithms based on kernel methods, where the embeddings are constructed from linear maps acting on the kernel feature space. 
strength_and_weaknesses: Strengths:
1、The theoretical derivation is rigorous;
2、The contribution description is relatively clear;
3、The writing process is clear.

Weaknesses:
1. Please describe the motivation of the proposed method in detail in the abstract;
2. For the organization of other parts of this article, it should be directly specific to the chapter, rather than described in sequence;
3. In the experimental part, the equipment type used and some hyper-parameter settings (the specific usage of the data set) should be described in detail;
4. The experimental part only proves the effectiveness of the proposed method, and does not compare with other advanced algorithms;
5. The design concept of the paper is complicated and cumbersome, please prove its reproducibility;
6. The last sentence of the paragraph above equation (14) lacks punctuation;
7. Tables are also a language. Don't just mention that the results of your proposed method are better than others, but try to explain why.

clarity,_quality,_novelty_and_reproducibility: The description of this paper is clear, the derivation process is detailed, and it is novel, but the complex process and design make me worry about its reproducibility.
summary_of_the_review: 1、For a contrastive and non-contrastive loss, this paper provide closed form solutions when the algorithm is trained over a single batch of data.
2、This paper show that a version of the representer theorem in kernel methods can be used to formulate kernelized SSL tasks as optimization problems.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper takes a Kernel learning approach to Self Supervised Learning. It provides a novel representation and the associated 
dot-product provides a new kernel. It provides some analysis of this kernel on downstream applications.
The main weakness seems to be that in light of the substantial body of work on kernel methods for graphs spanning two decades the contributions seem to be modest. Experimental evaluation also seems to be not too comprehensive.
It is thus clear that the paper needs to be improved to appear in a premier venue.

 
justification_for_why_not_higher_score: The concerns of not having substantive contributions vis-vis existing work is not effectively addressed
Decision: Reject