Supervised Dimension Contrastive Learning

Authors: Jaemyung Yu, SungHyun Baek, Jongsuk Kim, Junmo Kim
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Supervised Representation Learning, Dimension Contrastive Learning

Abstract:
Self-supervised learning has emerged as an effective pre-training strategy for representation learning using large-scale unlabeled data. However, models pre-trained with self-supervised learning still require supervised fine-tuning to achieve optimal task-specific performance. Due to the lack of label utilization, it is difficult to accurately distinguish between positive and hard negative samples. Supervised contrastive learning methods address the limitation by leveraging labels, but they focus on global representations, leading to limited feature diversity and high cross-correlation between representation dimensions. To address these challenges, we propose Supervised Dimension Contrastive Learning, a novel approach that combines supervision with dimension-wise contrastive learning. Inspired by redundancy reduction techniques like Barlow Twins, this approach reduces cross-correlation between embedding dimensions while enhancing class discriminability. The aggregate function combines the embedding dimensions to generate predicted class variables, which are optimized to correlate with their corresponding class labels. Orthogonal regularization is applied to ensure the full utilization of all dimensions by enforcing full-rankness in the aggregate function. We evaluate our method on both in-domain supervised classification tasks and out-of-domain transfer learning tasks, demonstrating its superior performance compared to traditional supervised learning, supervised contrastive learning, and self-supervised learning methods. Our results show that the proposed method effectively reduces inter-dimensional correlation and enhances class discriminability, proving its generalizability across various downstream tasks.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730396952788

thread_type:
official_review

<thread_object>
creator: reviewer_cbsm

<content>
summary:
Motivated by Barlow Twins, this paper proposes a method to constrain the feature diversity in supervised contrastive learning. Experimental results show the effectiveness of the proposed method.

strengths:
From experimental section, we can see that the proposed method performance well in OOD scenario. This is interesting and validates to some extent that feature diversity does correlate with the generalizability of feature representations.

weaknesses:
1. The novelty of this paper is really limited. This article does not explain the motivation for CLASS CORRELATION and FULL RANK AGGREGATION to be proposed. At the same time, no intuitive insight or theoretical analysis is given as to why CLASS CORRELATION and FULL RANK AGGREGATION promote supervised comparative learning performance.

2. The related work section is also quite weak. The authors have not done a good job of highlighting the developmental lineage of self-supervised learning, supervised comparative learning, and the connections and differences between the different algorithms. And also did not highlight the advantages and footholds of the method proposed in this paper.

questions:
1. Why can CLASS CORRELATION encourage the embedding dimensions to be not only augmentation-invariant and decorrelated but also discriminative for the supervised classification task? As we can see, CLASS CORRELATION is only a metric of Correlational Relationship.

2. Please give the theoretical analysis or related references to the statement: To guarantee that all embedding dimensions contribute to class separability, every embedding dimension should play a role in generating the predicted class variables through the aggregate function. It can be achieved by enforcing aggregate function is full-rank.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557484361

thread_type:
official_review

<thread_object>
creator: reviewer_utbt

<content>
summary:
This paper enhances Barlow Twins by using the class-correlation loss and orthogonal regularization loss. The class correlation loss uses supervision information, and the orthogonal regularization loss enforces the aggregate function to be full-rank. These additional losses positively affect the overall performance. Through numerical experiments on in-domain classification and out-domain transfer learning tasks, the proposed SupDCL demonstrated superior performance.

strengths:
- The proposed method demonstrated superior performance on in-domain classification and out-domain transfer learning tasks.

weaknesses:
- The class correlation and orthogonal regularization losses have not been investigated in Barlow Twins, but such losses are often investigated in other methods or areas.
- According to Table 4, most of the performance gain of the proposed method compared with Barlow Twins comes from supervision.

questions:
- What is an intuition that the proposed method improved performance on out-domain transfer learning tasks? 
- Based on Figure 1, the proposed method inherits the superior out-domain performance from Barlow Twins. The other regularization or loss terms are likely to obtain improvement similar to the proposed method from Barlow twins.
- Are there any theoretical insights about the performance of the proposed method?
- Can we use the proposed method without the class correlation loss as a self-supervised learning method? If so, does this outperform the existing methods?
- Regarding Table 4, is there any discussion of the performance of Barlow Twins plus the orthogonal regularization loss?
- In Table 3(b), the existing methods increase performance as the projector dimensionality increases, but the proposed method decreases after 128. Is there any interpretation for that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700139027

thread_type:
official_review

<thread_object>
creator: reviewer_t162

<content>
summary:
The paper introduces Supervised Dimension Contrastive Learning (SupDCL), which combines supervision with dimension-wise contrastive learning. It targets limitations in supervised contrastive learning (high cross-correlation and limited feature diversity) by reducing redundancy in embedding dimensions while enhancing class discriminability.

strengths:
- The integration of dimension contrastive learning with supervision to enhance feature diversity makes sense. SupDCL’s approach to reducing cross-correlation among dimensions to improve generalization, particularly in transfer learning tasks, is a well-justified.
- The evaluation covers both in-domain and out-domain tasks on different datasets, providing a clear comparison of SupDCL against multiple baselines, including self-supervised and supervised learning methods.
- Extensive ablation studies that analyze each component of SupDCL’s loss function.
- The paper acknowledges the limitations of SupDCL, specifically its reliance on fixed-form class labels. It provides suggestions for handling these limitations through future work.

weaknesses:
- The paper introduces orthogonal regularization and decorrelation strategies to improve feature diversity, but it lacks a theoretical foundation to show why these particular choices are necessary. While the empirical results show improved performance, a lack of formal justification leaves the effectiveness of these strategies unclear.
- While SupDCL outperforms SupCon in out-doman tasks, it underperforms SupCon in in-doman tasks.
- SupDCL borrows heavily from existing redundancy reduction techniques (Barlow Twins) without significantly advancing the state of contrastive learning. The contribution of introducing supervision to dimension contrastive learning is incremental and could be viewed as a straightforward application of existing techniques.

questions:
- Why is feature discriminative inportant for supervised contrastive learning? Figure 1 and experiments show that SupDCL underperforms SupCon in in-doman tasks.
- What is $L_{inv}$ and $L_{decorr}$ is Eq (7)? They are not explained.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730771478507

thread_type:
official_review

<thread_object>
creator: reviewer_cmuw

<content>
summary:
This paper argues that representations learned by supervised contrastive learning methods have limited feature diversity and high cross-correlation between dimensions. The authors propose an approach to learn representations that are both augmentation-invariant and dimension-wise decorrelated, while also being discriminative for classification. They achieve this by extending Barlow Twins, a dimension contrastive learning model, to supervised dimension contrastive learning. Specifically, they introduce a discriminativeness loss instead of the cross-entropy loss, consisting of a class correlation loss and an orthogonal regularization loss. Experimental results show that the proposed SupDCL performs well on both in-domain supervised classification tasks and out-of-domain transfer learning tasks.

strengths:
The idea of this paper is to combine SupCon (supervised contrastive learning) with Barlow Twins (decorrelated features). The specific approach involves introducing a discriminativeness loss to leverage the supervised labels. The motivation and approach are presented very clearly. The authors also provide experiments, analysis, and evaluation metrics to demonstrate the effectiveness of the proposed method.

weaknesses:
I believe the experiments could be strengthened, and there are some minor points that would enhance the clarity of the paper. These are outlined in more detail in the Questions section.

questions:
1. This work aims to extend Barlow Twins to a supervised version. A straightforward extension model could be a combination of Barlow Twins and SupCon, where, instead of generating two augmentations $X^A$ and $X^B$ from image $X$, $X^B$ is selected from another image that shares the same class as image $X$. In Table 2, I would like to see the results of this model.

2. Another straightforward extension is to use a cross-entropy loss function instead of the proposed discriminativeness loss function to leverage the supervised labels. While you have compared this model in Table 3, comparing the results in Table 1 and Table 2 would also be worthwhile.

3. In Table 3, you mention the representation space and embedding space; however, in the methods section and Figure 2, there is no reference to the representation space. You have defined the embedding space, $Z$, as an $M-dimensional$ vector, but it is unclear which vector you refer to as the representation vector.

4. Similarly, Section 4.2 does not clarify what the representation $Y$, a $D-dimensional$ vector, refers to.

5. In lines 371-374, when $t_d = 0.1$, Barlow Twins achieves 1320, which seems to be the lowest count rather than the highest as described. Is there a mistake here?

6. Visualization: Could you visualize the embedding space or representation space (with clearer references if possible) of different methods, along with the different dimensions of those representations? This would help to better understand how the proposed method achieves improved decorrelation and discriminative representations.


Minors:
1. It would be better to mark all the best numbers in all tables in bold.

2. In Table 2, add arrows to indicate whether larger or smaller values are better, and mark the best value in bold.

3. Figure 1 can be split into two separate figures, as the third sub-figure is not on the same scale as the other two.

4. In lines 353-358, 645 (SupDCL) vs. 532 (SupCon) is described as outperforming, while 645 (SupDCL) vs. 904 (Barlow Twins) is described as comparable?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732805474037

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarification of Our Framework

comment:
Dear Reviewers,

We thank the reviewers for their valuable feedback, which guided us to strengthen the **theoretical and empirical justification** of our proposed dimension-based framework. Initially, we focused on the motivations behind our approach. Now, we theoretically and empirically demonstrate that all three components addresses specific challenges. Specifically, we decompose the mutual information between class labels and learned representations into two components (refer to Revised G2): **diversity** and **discriminativeness**, and together, they form a **comprehensive framework** for robust generalization. All justifications have been revised and reflected in the updated version of the paper.

$I(C;Y)$

$=\underbrace{H(Y)}_{\text{diversity}}$ “the total information capacity of the learned representation $Y$”

$-\underbrace{(H(\hat{C}|C)+H(Y_\text{null}))}_{\text{discriminativeness}}$ “the uncertainty and information loss related to the aggregation process”


1. Dimension Decorrelation - Maximizing $H(Y)$:
    - Reduces inter-dimension correlation using Barlow Twins (BT), theoretically shown to **increase information capacity of representation** (Revised G2 of the global response and Appendix A.2).
    - Empirically, it improves generalization performance, as evidenced by the relationship between dimension correlation and whole-domain accuracy (Figure 1) and the effectiveness of BT+CE over CE alone (Table B).


2. Class Correlation - Minimizing $H(\hat{C}|C)$:
    - A novel method for robust distributional alignment, theoretically shown to **be invariant to covariate shift**, and minimize distance between predicted and true class distributions (Revised G2 of the global response and Appendix A.3.1).
    - Empirically, class correlation improves in-domain and out-domain performance, consistently outperforming CE across all settings (Table B).


3. Full-rank Aggregation - Minimizing $H(Y_\text{null})$:
    - Preserves class-relevant information of representation, theoretically shown to **minimize information capacity of null space** (Revised G2 of the global response and Appendix A.3.2).
    - Empirically, comparisons such as BT+CE vs. BT+CE+Orthogonal and SupDCL w/o Orthogonal vs. SupDCL confirm its significant contribution to both in-domain and out-domain performance (Table B).

Combining these components, our method achieves **state-of-the-art whole-domain generalization**, outperforming all supervised and self-supervised baselines across in-domain and out-domain tasks (Table C).


**Table C**. In-domain classification and out-domain transfer learning. Linear evaluation performance comparison on 10 downstream datasets, for ResNet-50 pre-trained on ImageNet-1K.

| **Method** | **In-domain (IN1K)** | **Average out-domain (10 datasets)** | **Whole-domain (all 11 datasets weighted equally)** | **Whole-domain (50% in-domain, 50% averaged out-domain)** |
| --- | --- | --- | --- | --- |
| Supervised | 76.1 | 74.5 | 74.6 | 75.3 |
| *Self-supervised Representation Learning:* |  |  |  |  |
| SimCLR | 69.1 | 73.0 | 72.6 | 71.0 |
| Barlow Twins | 73.2 | 79.0 | 78.5 | 76.1 |
| SwAV | 75.3 | 79.0 | 78.7 | 77.2 |
| MoCo v3 | 71.1 | 79.4 | 78.7 | 75.3 |
| DINO | 75.3 | 80.3 | 79.9 | 77.8 |
| *Supervised Representation Learning:* |  |  |  |  |
| SupCon | 77.9 | 77.8 | 77.8 | 77.9 |
| PaCo | 78.7 | 69.4 | 70.2 | 74.1 |
| GPaCo | **79.5** | 68.8 | 69.8 | 74.2 |
| **SupDCL-1024 (Ours)** | 78.2 | 78.9 | 78.9 | 78.6 |
| **SupDCL (Ours)** | 77.5 | **80.8** | **80.5** | **79.2** |


**Table B.** Comparison of SupDCL and straightforward extensions of Barlow Twins with cross-entropy loss

| **Method** | **In-domain (CIFAR-100)** | **Average Out-domain** |
| --- | --- | --- |
| Barlow Twins | 52.6% | 39.1% |
| Barlow Twins + CE | 72.6% | 39.6% |
| Barlow Twins + CE with Orthogonalization | 73.8% | 40.8% |
| SupDCL w/o Orthogonalization | 73.8% | 41.0% |
| SupDCL (Ours) | **74.2%** | **42.0%** |
</content>
</thread_object>

</thread>


Metareview: