On the Universality of Self-Supervised Representation Learning

Authors: Wenwen Qiang, Jingyao Wang, Lingyu Si, Changwen Zheng
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-Supervised Learning, Representation Learning, Unsupervised Learning

Abstract:
In this paper, we investigate the characteristics that define a good representation or model. We propose that such a representation or model should possess universality, characterized by: (i) discriminability: performing well on training samples; (ii) generalization: performing well on unseen datasets; and (iii) transferability: performing well on unseen tasks with distribution shifts. Despite its importance, current self-supervised learning (SSL) methods lack explicit modeling of universality, and theoretical analysis remains underexplored. To address these issues, we aim to explore and incorporate universality into SSL. Specifically, we first revisit SSL from a task perspective and find that each mini-batch can be viewed as a multi-class classification task. We then propose that a universal SSL model should achieve: (i) learning universality by minimizing loss across all training samples, and (ii) evaluation universality by learning causally invariant representations that generalize well to unseen datasets and tasks. To quantify this, we introduce a 
$\sigma$-measurement that assesses the gap between the performance of SSL model and optimal task-specific models. Furthermore, to model universality, we propose the GeSSL framework. It first learns task-specific models by minimizing SSL loss, then incorporates future updates to enhance discriminability, and finally integrates these models to learn from multiple mini-batch tasks. Theoretical and empirical evidence supports the effectiveness of GeSSL.

Decision: reject
Reviewer scores: 6, 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729757694520

thread_type:
official_review

<thread_object>
creator: reviewer_apsp

<content>
summary:
This paper advances self-supervised learning (SSL) by defining "Universality," which includes discriminability, generalizability, and transferability. This paper introducs GeSSL, a framework that models universality by enhancing discriminability with Kullback-Leibler divergence, promoting generalizability through causal feature extraction across tasks, and implementing a bi-level optimization for transferability.

strengths:
This paper introduces a comprehensive framework for incorporating universality into self-supervised learning (SSL) by defining it through discriminability, generalization, and transferability. The σ-measurement quantifies performance gaps, strengthening the theoretical foundation of SSL methods. The GeSSL framework effectively integrates task-specific learning and focuses on causally invariant representations, enhancing adaptability and performance across diverse tasks. Overall, the paper addresses critical gaps in SSL methodologies and presents a novel approach supported by theoretical and empirical evidence.

weaknesses:
1.The self-supervised learning (SSL) paradigm requires algorithms to exhibit good alignment and uniformity. However, the concepts of discriminability and alignment mentioned in the paper seem to overlap. Similarly, generalizability and transferability appear to be closely related to uniformity, as improved uniformity is essential to prevent collapse and ensure good generalizability and transferability of the model.

2.In line 230, the paper mentions M mini-batches. If these M mini-batches are assumed to be independently and identically distributed (i.i.d.), it raises concerns about how diversity between different mini-batches is guaranteed. If the diversity is insufficient, the assumption that supports the model’s effectiveness may not hold.

3.Meta-learning typically requires longer run times, raising questions about the relationship between the running time of this algorithm and that of the original methods. For instance, it would be beneficial to compare the time taken to run one epoch of SimCLR with the time required when incorporating the proposed algorithm. This comparison could provide insights into the practical implications of adopting the new method.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730451931050

thread_type:
official_review

<thread_object>
creator: reviewer_79r3

<content>
summary:
This paper focuses on a news representation quality evaluation of self-supervised learning (SSL) frameworks. In particular, the authors first introduce a definition for SSL universality, which combines three available factors, including discriminability, generalizability, and transferability, together with a σ-measurement to quantify it. They next propose GeSSL, a novel framework that models universality by unifying a self-motivated target for discriminability, a multi-batch collaborative update mechanism for generalizability, and a task-based bi-level learning paradigm for transferability. Experimental results on benchmark datasets demonstrate the superior performance of GeSSL compared to the original baselines.

strengths:
The paper is well-written and easy to follow. Its idea of introducing a definition of measurable universality for SSL settings and a novel SSL framework (GeSSL) is novel and interesting. The experimental results on several benchmark datasets consistently show the benefit of using GeSSL compared to the mentioned self-supervised baselines.

weaknesses:
My major concerns are about the actual novelty and the theoretical part of the paper. In particular:
- The definition of universality (Def. 3.1) is not mathematically well-defined since it contains several unclear terms, such as "competitive performance", "quickly", "few samples", and "comparable performance". I suggest the authors provide precise mathematical definitions or quantitative thresholds for these terms. For example, what specific metrics or benchmarks would constitute "competitive performance"? What time or iteration limit defines "quickly"? This would help make the definition more rigorous.
- The definition of a "$\sigma$-measurement" (Def. 3.2) is not well-defined since we do not know the term $f_{\Phi_l}^{*}$. Also, it is not unclear how the KL term in Eq. (1) is computed. Could the authors explicitly define $f_{\Phi_l}^{*}$ and provide more details on how the KL divergence is calculated in practice, including any approximations or estimation techniques used?
- The main theoretical contribution of the paper (Theorem 4.1) is not very meaningful. In particular, assuming that the proof is true (I did not have enough time to carefully verify it), the inequality $\tilde f_\theta - f_\theta \leq 0$ does not imply that $\tilde f_\theta$ is a good approximation of $f_\theta$.  I suggest the authors provide additional theoretical results or empirical evidence to demonstrate the practical significance of this inequality. For example, what does this inequality imply about the performance or convergence of the algorithm in practice? Also, it would be better to provide the upper bound of $\|\tilde f_\theta - f_\theta\|$.
- As mentioned in the paper (lines 193-294), GeSSL seems to be a computationally expensive and not scalable method since it requires the computation of the second-order derivatives (Hessian) w.r.t the (high-dimensional) parameters. Could the authors provide a more detailed analysis of the computational complexity of their method, perhaps including comparisons with existing approaches? Also, can the authors discuss potential approximations or optimizations that could make the method more scalable for high-dimensional problems?

questions:
Please see my comments about the weaknesses of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576871471

thread_type:
official_review

<thread_object>
creator: reviewer_e5gk

<content>
summary:
The paper aims to formalize the notion of "goodness of a representation". The authors introduce the definition comprising three simple characteristics of a representation to be good: (i) discriminability, (ii) generalization and (iii) transferability. Furthermore, the authors develop the methodology that instantiates this definition. This methodology is compatible with existing SSL approaches. The authors evaluate the proposed approach on a variety of tasks and in conjunction with a few SSL methods.

strengths:
* The proposed methodology is compatible with many existing SSL methods and leads to their improvements.

weaknesses:
* The presentation of the paper does not convincingly demonstrates the novelty behind the proposed definitions for the representations to be "good". Specifically, the stated principles are already the ultimate goal of the entire representation learning community, thus, it is not clear what specifically novel authors introduce in their narration.

* Given that the proposed methodology is applied to existing SSL methods, the results demonstrate only minor quantitative improvements upon the corresponding SSL method.

questions:
* I suggest authors to restructure their tables and display the results of their method next to the corresponding used SSL method to simplify parsing the results.

* Given that authors propose new definitions and argue that the existing approaches do not meet these definitions, I would expect some experimental evidence towards to support this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598755360

thread_type:
official_review

<thread_object>
creator: reviewer_ye1v

<content>
summary:
The authors propose that a robust representation or model should exhibit universality, defined by three core attributes: (i) discriminability—performing well on training samples; (ii) generalization—achieving strong performance on unseen datasets; and (iii) transferability—adapting effectively to new tasks with distribution shifts. To explicitly capture universality within the self-supervised learning (SSL) process, the authors introduce a novel framework, GeSSL. GeSSL employs KL divergence to distill knowledge from future states into the current state, thus lowering training loss; it promotes generalizability by extracting causal features through multi-task learning (multiple mini-batches); and it incorporates a bi-level optimization mechanism to structure SSL learning in a meta-learning format, enhancing transferability.

strengths:
The paper is easy to follow, though the abstract is somewhat overwhelming.

The authors conduct comprehensive experiments across a diverse range of applications and datasets, effectively demonstrating the proposed method's effectiveness.

weaknesses:
1. Some important information, such as the classifier $\pi$, the parameter M and reference, is not well-described in the paper. Please see the questions section for more details. Clarifying these aspects would greatly enhance the reader's understanding of the paper.

&nbsp;

2. Section 2: Treating each mini-batch in the SSL training phase as a multi-class classification task is well-established in the SSL literature. For example, contrastive loss, widely used in frameworks like SimCLR and MoCo, functions as an instance discriminator [1,2,3]. The authors should acknowledge this in the text.


&nbsp;

3. I think as the authors state MEASUREMENT OF UNIVERSALITY is one of main contribution, F.4 should be appear as the result on the main text.

&nbsp;

[1] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).

[2] Hjelm, R. Devon, et al. "Learning deep representations by mutual information estimation and maximization." arXiv preprint arXiv:1808.06670 (2018).

Tian, Yonglong, et al. "What makes for good views for contrastive learning?." Advances in neural information processing systems 33 (2020): 6827-6839.

questions:
**Q1:** Line 149-153: "According to Theorem 9 in Ahuja et al. (2020), if a model achieves good performance across multiple different tasks, it can be considered to have learned causal representations. Moreover, Schölkopf et al. (2021) and Ahuja et al. (2023) conclude that causality in representations is a sufficient condition for generalizability. Thus, a generalizable representation should be causally invariant across multiple tasks, enabling the model to achieve very low loss on these tasks using the same representation"

The authors refer to Theorem 9 in Ahuja et al. (2020), but I was unable to find it in that paper. Furthermore, as I understand it in the context of causal representation, multitasking for causal representation implies using a shared representation, z=f(x), to perform multiple tasks on the same representation. However, in GeSSL, multitasking is applied across different mini-batches (i.e., different samples x), which do not yield the same representation. This approach may still allow the model to rely on spurious features for good performance. Could the authors clarify this?

&nbsp;

**Q2:** The authors repeatedly mention employing the classifier $\pi$ to output class probabilities for the mini-batch, but I couldn't find any part of the paper describing how this is obtained. 

Additionally, if π is a classifier per mini-batch, it would need to be retrained at every iteration. On the other hand, if $\pi$ is a classifier for the entire dataset, does that imply prior knowledge of the task before training?

&nbsp;

**Q3:** (σ-measurement): When referring to π in the context of $\sigma$-measurement, does π denote tasks like LIN, SEMI, or AP for image-based examples? I’m unclear on how the model is evaluated using $\sigma$-measurement in the ablation study F.4: Universality of Existing SSL Methods. Could the authors clarify? 

Additionally, since the authors identify the measurement of universality as a primary contribution, F.4 should appear as a main result in the core text.

&nbsp;

**Q4:** GeSSL processes M mini-batches simultaneously during training:
-I couldn’t find any information on the size of M or any ablation study regarding M.

- Since SSL performance is highly dependent on batch size, it would be beneficial if the authors compared GeSSL with a baseline that uses M× the original batch size.

In Table 20, the model analysis includes parameter size, training time, and performance, showing that GeSSL achieves better results than the baseline with less training time (in hours). However, GeSSL’s optimization is more intensive, utilizing ($\lambda=10$) and a second-order gradient-based approach. Does this mean GeSSL requires fewer epochs to converge? If so, Table 9 shows some baselines achieving better performance in 400 epochs than GeSSL does in 1000 epochs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730772784328

thread_type:
official_review

<thread_object>
creator: reviewer_etz8

<content>
summary:
This paper introduce a new self-supervised learning (SSL) framework to improve the universality by explicitly modeling discriminability, generalizability, and transferability. The method combines self-motivated target learning, multi-batch collaborative updates, and task-based bi-level optimization to better balance model performance across different tasks and domains. The approach is validated on multiple scenarios (unsupervised learning, semi-supervised learning, transfer learning, few-shot learning) showing consistent improvements over existing SSL methods.

strengths:
1. Different from previous works that mainly focus on discriminability, the authors propose to use a self-motivated target and bi-level optimization to explicitly model universality properties, leading to more robust and transferable models.


2. The authors have conducted comprehensive experiments across multiple learning paradigms and datasets. Thorough ablation studies demonstrate the necessity of each component (self-motivated target, multi-batch updates, bi-level optimization). The theoretical analysis provides performance guarantees through the $\sigma$-measurement and proves convergence properties.

3. The proposed method can be applied to enhance various existing SSL methods. The extensive empirical results demonstrate consistent improvements when integrating GeSSL with different baseline approaches like SimCLR, MoCo, BYOL etc., showing the broad applicability of the proposed method.

weaknesses:
1. It remains unclear to me why using KL divergence alone is sufficient for measuring distribution differences in the self-motivated target. The authors argue that KL divergence better reflects the uncertainty between distributions compared to other metrics like MSE or cross-entropy, but don't fully explain why this specific choice leads to optimal universality properties compared to alternatives like Wasserstein distance.

2. My another concern is how the method balances computational efficiency with model performance. While the authors demonstrate performance improvements, the bi-level optimization introduces additional memory overhead and computational complexity. The optimal trade-off between efficiency and universality lacks thorough analysis, especially for resource-constrained scenarios.

questions:
1. Why choose KL divergence over alternatives like Wasserstein distance? Were there any empirical observations that led to this choice?

2. In Section 3.3, you mention K is typically set to 1 due to batch size and training complexity constraints. Could you elaborate on how sensitive the model performance is to different K values?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732782210757

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Novelty

comment:
Dear PC, SAC, AC, and Reviewers:

**We sincerely would like to express our great gratitude to you for the time and effort in this work.**

**First of all**, we are honored that this work has been recognized by the reviewers and received the Rating of (6,6,6,3,5), where the comments affirmed that our work is Novel and Innovative, Theoretically Robust, and Empirically Effective. Meanwhile, we have fine-tuned this paper according to the reviewers' valuable suggestions.

**Secondly**, after rebuttal and discussion, we are pleased that most concerns were addressed, with Reviewer 79r3 deciding to improve the score. For now, the only concern from Reviewer e5gk and Reviewer apsp (2/5 who give the negative rating),  lies in the differences between this work and previous SSL methods.  Considering that they did not respond to our further replies, we would like to discuss this issue again here to achieve consensus and clarify misunderstandings.

---

Reviewer e5gk stated:

> "I argue that reaching the properties is already an ultimate goal of community, thus, in essence modern evaluation protocols consider diverse downstream datasets within a given downstream task and diverse set of tasks for transfer." 

Reviewer apsp stated:

> “ the authors **interpret** what constitutes a good representation/model through the lens of universality, specifically from the perspectives of discriminability, generalizability, and transferability. However, a model with good alignment and uniformity is, to some extent, already a good representation model.”

---

In this regard, we would like to clarify:

---

Both the mentioned references previous SSL works can be summarized as **do what leads to a good representation**. These works evaluate the effectiveness of **do what** through experiments, such as discriminative tasks, generalization tasks, and transferability tasks, to demonstrate its utility. However, they do not address the question of **what constitutes a good representation**. Specifically, while they validate the effectiveness of *"do what"*, **they fail to explain how the representation is transformed through this process**. 

**Reviewer asap** says that a model with good alignment and uniformity is, to some extent, already a good representation model. We agree with this. But, a question is that **why good alignment and uniformity can lead to a good representation model**? For example, SimCLR has demonstrated through experiments that contrastive loss achieves strong performance across various tasks, but it does not explain why the loss is effective. [1] theoretically shows that contrastive loss can be interpreted as achieving alignment and uniformity in the feature space, but it does not explain why they are key to learning good representations. [2] proves that contrastive loss forms an upper bound of supervised loss, suggesting that minimizing contrastive loss can effectively minimize supervised loss and thus imply the learning of good representations. However, [1] assumes that positive and negative samples are perfectly selected, whereas the success of SimCLR and the extensions are not based on this assumption. Therefore, [1] does not sufficiently explain why SimCLR achieves good representations. This is a remaining problem for SSL. 

[1] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

[2] A Theoretical Analysis of Contrastive Unsupervised Representation Learning

In contrast, our work directly focuses on defining and interpreting *a good representation* (**which is affirmed by all the reviewers**). **We formalize the concepts of feature discriminability, generalizability, and transferability as key characteristics of *a good representation* and propose corresponding learning objectives**. The most significant value of this approach lies in enabling the interpretation of representations based on these formally defined characteristics, i.e., **feature interpretability**. From this perspective, our contribution is substantial and meaningful. 

If Reviewer e5gk and apsp insist that such formalization is meaningless, then by the same reasoning, numerous fields like formal logic, formal verification, interpretable ML, etc., would also lose their value. They fundamentally aim to formalize understanding into learnable objectives.

Regarding experiments, our method achieves an average improvement of over 4% on various tasks, which is a significant advancement in SSL. We have conducted extensive experiments which all demonstrate the effectiveness. The Friedman test shows that, for significance levels of ${\alpha _F} = 0.1$, ${\alpha _F} = 0.05$, and ${\alpha _F} = 0.025$, our method significantly exceeds the critical values of the F-distribution. Thus, the experimental results substantiate the validity of our work.

---

Therefore, we insist that our work is solid.

Lastly, we would like to deeply thank you again for your time and effort on this work.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper characterizes a good representation in self-supervised representation learning should posess universality which consists of discriminability, generalization, and transferability. The proposeed GeSSL frameowrk explicitly models universality by using a self-motivated target for discriminability with KL divergence, a multi-batch collaborative update to aim for generalizable features, and task-based bi-level optimization to enhance transferability. The experiments show the improvement when added to previous self-supervised learning methods. Theoretical analysis is provided.

The main concern expressed by reviewers (including Reviewer e5gk) is overlap with existing self-supervised learning concepts. There were a lot of discussions during the rebuttal period regarding this point, but some reviewers remained unconvinced. Reviewer e5gk also expressed concerns about the methodological novelty, regarding the similarity with meta learning approaches such as MAML-like methods. Although a direct response was not provided regarding this point, Appendix H addresses this point to some extent (and the strong empirical results also support the proposed method.) Another reviewer pointed out that the rigor of the definitions was lacking, but the reviewer was satisfied with the rebuttal and raised the score to 6.

Overall, we had 3 borderline-positive scores (6, 6, 6) and two negative score (5, 3). We would like to recommend rejection at this time, since the paper needs to further differentiate and position the work better within the existing literature. We hope the authors can include a lot of the discussions during the rebuttal phase into future versions of the paper.