Title: Identifiable Contrastive Learning with Automatic Feature Importance Discovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-supervised Learning

Contrastive Learning

Identifiability

Representation Learning
Abstract: Existing contrastive learning methods rely on pairwise sample contrast $z_x^\top z_{x'}$ to learn data representations, but the learned features often lack clear interpretability from a human perspective. Theoretically, it lacks feature identifiability and different initialization may lead to totally different features. In this paper, we study a new method named tri-factor contrastive learning (triCL) that involves a 3-factor contrast in the form of $z_x^\top S z_{x'}$, where $S=\text{diag}(s_1,\dots,s_k)$ is a learnable diagonal matrix that automatically captures the importance of each feature. We show that by this simple extension, triCL can not only obtain identifiable features that eliminate randomness but also obtain more interpretable features that are ordered according to the importance matrix $S$. We show that features with high importance have nice interpretability by capturing common classwise features, and obtain superior performance when evaluated for image retrieval using a few features. The proposed triCL objective is general and can be applied to different contrastive learning methods like SimCLR and CLIP. We believe that it is a better alternative to existing 2-factor contrastive learning by improving its identifiability and interpretability with minimal overhead. Code is available at https://github.com/PKU-ML/Tri-factor-Contrastive-Learning.

Identifiable Contrastive Learning with Automatic Feature Importance Discovery

Abstract

Existing contrastive learning methods rely on pairwise sample contrast z ⊤ x z x ′ to learn data representations, but the learned features often lack clear interpretability from a human perspective. Theoretically, it lacks feature identifiability and different initialization may lead to totally different features. In this paper, we study a new method named tri-factor contrastive learning (triCL) that involves a 3-factor contrast in the form of z ⊤ x Sz x ′ , where S = diag( s 1 , . . . , s k ) is a learnable diagonal matrix that automatically captures the importance of each feature. We show that by this simple extension, triCL can not only obtain identifiable features that eliminate randomness but also obtain more interpretable features that are ordered according to the importance matrix S . We show that features with high importance have nice interpretability by capturing common classwise features, and obtain superior performance when evaluated for image retrieval using a few features. The proposed triCL objective is general and can be applied to different contrastive learning methods like SimCLR and CLIP. We believe that it is a better alternative to existing 2-factor contrastive learning by improving its identifiability and interpretability with minimal overhead. Code is available at https://github.com/PKU-ML/ Tri-factor-Contrastive-Learning .

1 Introduction

As a representative self-supervised paradigm, contrastive learning obtains meaningful representations and achieves state-of-the-art performance in various tasks by maximizing the feature similarity z ⊤ x z + x between samples augmented from the same images while minimizing the similarity z ⊤ x z -x between independent samples [4, 20, 15, 5, 18]. Besides the empirical success, recent works also discuss the theoretical properties and the generalization performance of contrastive learning [32, 30, 18].

However, there still exist many properties of contrastive learning that are not guaranteed. In this paper, we focus on a significant one: the feature identifiability. Feature identifiability in the representation learning refers to the property there exists a single, global optimal solution to the learning objective. Consequently, the learned representations can be reproducible regardless of the initialization and the optimizing procedure is useful. As a well-studied topic, identifiability is a desirable property for various tasks, including but not limited to, transfer learning [10], fair classification [26] and causal inference [24]. The previous works propose that contrastive learning obtains the linear feature identifiability while lacking exact feature identifiability, i.e., the optimal solutions obtain a freedom of linear transformations [29]. As a result, the different features are coupled, which hurts the interpretability and performance of learned representations.

(a) 10 most important dimensions

(b) 10 least important dimensions

In this paper, we propose a new contrastive learning model: tri-factor contrastive learning (triCL), which introduces a 3-factor contrastive loss, i.e., we replace z ⊤ x z x ′ with z ⊤ x Sz x ′ when calculating the similarity between two features, where S is a learnable diagonal matrix called importance matrix. We theoretically prove that triCL absorbs the freedom of linear transformations and enables exact feature identifiability . Besides, we observe that triCL shows other satisfying properties. For example, the generalization performance of triCL is theoretically guaranteed. What is more, we find that the diagonal values of the importance matrix S in triCL indicate the degrees of feature importance. In Figure 1, we visualize the samples that have the largest values in the most and least important dimensions ordered by the importance matrix. We find the samples activated in the more important dimensions are more semantically similar, which verifies the order of feature importance in triCL is quite close to the ground truth. Theoretically, we prove that the dimensions related to the larger values in the importance matrix make more contributions to decreasing the triCL loss. With the automatic discovery of feature importance in triCL, the downstream task conducted on the representations can be accelerated by selecting the important features and throwing the meaningless features.

As triCL is a quite simple and general extension, we apply it to different contrastive learning frameworks, such as SimCLR [4] and CLIP [28]. Empirically, we first verify the identifiability of triCL and further evaluate the performance of triCL on real-world datasets including CIFAR-10, CIFAR-100, and ImageNet-100. In particular, with the automatic discovery of important features, triCL demonstrates significantly better performance on the downstream tasks using a few feature dimensions. We summarize our contributions as follows:

We propose tri-factor contrastive learning (triCL), the first contrastive learning algorithm which enables exact feature identifiability. Additionally, we extend triCL to different contrastive learning methods, such as spectral contrastive learning, SimCLR and CLIP.

Besides the feature identifiability, we analyze several theoretical properties of triCL. Specifically, we construct the generalization guarantee for triCL and provide theoretical evidence that triCL can automatically discover the feature importance.

Empirically, we verify that triCL enables the exact feature identifiability and triCL can discover the feature importance on the synthetic and real-world datasets. Moreover, we investigate whether triCL obtains superior performance in downstream tasks with selected representations.

2 Related Work

Self-supervised Learning. Recently, to get rid of the expensive cost of the labeled data, selfsupervised learning has risen to be a promising paradigm for learning meaningful representations by designing various pretraining tasks, including context-based tasks [14], contrastive learning [20] and masked image modeling [21]. Among them, contrastive learning is a popular algorithm that achieves impressive success and largely closes the gap between self-supervised learning and supervised learning [27, 4, 20, 33]. The idea of contrastive learning is quite simple, i.e., pulling the semantically similar samples (positive samples) together while pushing the dissimilar samples (negative samples) away in the feature space. To better achieve this objective, recent works propose different variants of contrastive learning, such as introducing different training objectives [36, 18, 35], different structures [4, 6, 16, 39], different sampling processes [11, 25, 38, 7] and different empirical tricks [20, 3].

Theoretical Understandings of Contrastive Learning. Despite the empirical success of contrastive learning, the theoretical understanding of it is still limited. [30] establish the first theoretical guarantee on the downstream classification performance of contrastive learning by connecting the InfoNCE loss and Cross Entropy loss. As there exists some unpractical assumptions in the theoretical analysis in [30], recent works improve the theoretical framework and propose new bounds [1, 2, 34]. Moreover, [18] analyzes the downstream performance of contrastive learning from a graph perspective and constructs the connection between contrastive learning and spectral decomposition. Recently, researchers have taken the inductive bias in contrastive learning into the theoretical framework and shown the influence of different network architectures [17, 31]. Except for the downstream classification performance of contrastive learning, some works focus on other properties of representations learned by contrastive learning. [23] discuss the feature diversity by analyzing the dimensional collapse in contrastive learning. [29] prove that the contrastive models are identifiable up to linear transformations under certain assumptions.

3 Preliminary

Contrastive Pretraining Process. Webegin by introducing the basic notations of contrastive learning. The set of all natural data is denoted as D u = { ¯ x i } N u i =1 with distribution P u , and each natural data ¯ x ∈ D u has a ground-truth label y (¯ x ) . An augmented sample x is generated by transforming a natural sample ¯ x with the augmentations distributed with A ( ·| ¯ x ) . The set of all the augmented samples is denoted as D = { x i } N i =1 . We assume both sets of natural samples and augmented samples to be finite but exponentially large to avoid non-essential nuances in the theoretical analysis and it can be easily extended to the case where they are infinite [18]. During the pretraining process, we first draw a natural sample ¯ x ∼ P u , and independently generate two augmented samples x ∼ A ( ·| ¯ x ) , x + ∼ A ( ·| ¯ x ) to construct a positive pair ( x, x + ) . For the negative samples, we independently draw another natural sample ¯ x -∼ P u and generate x -∼ A ( ·| ¯ x -) . With positive and negative pairs, we learn the encoder f : R d → R k with the contrastive loss. For the ease of our analysis, we take the spectral contrastive loss [18] as an example:

We denote z x = f ( x ) as the features encoded by the encoder. By optimizing the spectral loss, the features of positive pairs ( z ⊤ x z x + ) are pulled together while the negative pairs ( z ⊤ x z x -) are pushed apart.

Augmentation Graph. A useful theoretical framework to describe the properties of contrastive learning is to model the learning process from the augmentation graph perspective [18]. The augmentation graph is defined over the set of augmented samples D , with its adjacent matrix denoted by A . In the augmentation graph, each node corresponds to an augmented sample, and the weight of the edge connecting two nodes x and x + is equal to the probability that they are selected as a positive pair, i.e., A xx + = E ¯ x ∼P u [ A ( x | ¯ x ) A ( x + | ¯ x )] . And we denote ¯ A as the normalized adjacent matrix of the augmentation graph, i.e., ¯ A = D -1 / 2 AD -1 / 2 , where D is a diagonal matrix and D xx = ∑ x ′ ∈D A xx ′ . To analyze the properties of ¯ A , we denote ¯ A = U Σ V ⊤ is the singular value decomposition (SVD) of the normalized adjacent matrix ¯ A , where U ∈ R N × N , V ∈ R N × N are unitary matrices, and Σ = diag( σ 1 , . . . , σ N ) contains descending singular values σ 1 ≥ . . . σ N ≥ 0 .

4 Exact Feature Identifiability with Tri-factor Contrastive Learning

In this section, we propose a new representation learning paradigm called tri-factor contrastive learning (triCL) that enables exact feature identifiability in contrastive learning. In Section 4.1, we prove that contrastive learning obtains linear identifiability, i.e., the freedom in the optimal solutions are linear transformations. In Section 4.2, we introduce the learning process of triCL and theoretically verify it enables the exact feature identifiability.

4.1 Feature Identifiability of Contrastive Learning

When using a pretrained encoder for downstream tasks, it is useful if the learned features are reproducible, in the sense that when the neural network learns the representation function on the same data distribution multiple times, the resulting features should be approximately the same. For example, reproducibility can enhance the interpretability and the robustness of learned representations [22]. One rigorous way to ensure reproducibility is to select a model whose representation function is identifiable in function space [29]. To explore the feature identifiability of contrastive learning, we first characterize its general solution.

Lemma 4.1 ([18]) . Let ¯ A = U Σ V ⊤ be the SVD decomposition of the normalized adjacent matrix ¯ A . Assume the neural networks are expressive enough for any features. The spectral contrastive loss (Eq. 1) attains its optimum when ∀ x ∈ D ,

where U x takes the x -th row of U , U k denotes the submatrices containing the first k columns of U , and R ∈ R k × k is an arbitrary unitary matrix.

From Lemma 4.1, we know that the optimal representations are not unique, due to the freedom of affine transformations. This is also regarded as a relaxed notion of feature identifiability, named linear feature identifiability L ∼ defined below [29].

Definition 4.2 (Linear feature identifiability) . Let L ∼ be a pairwise relation in the encoder function space F = { f : X → R k } defined as:

where A is an invertible k × k matrix.

It is apparent that the optimal encoder f (Eq. 2) obtained from the spectral contrastive loss (Eq. 1) is linearly identifiable. Nevertheless, there are still some ambiguities w.r.t. linear transformations in the model. Although the freedom of linear transformations can be absorbed on the linear probing task [18], these representations may show varied results in many downstream tasks e.g., in the k-NN evaluation process. So we wonder whether we could achieve the exact feature identifiability. We first further define two kinds of more accurate feature identifiabilities below.

Definition 4.3 (Sign feature identifiability) . Let S ∼ be a pairwise relation in the encoder function space F = { f : X → R k } defined as:

where f j ( x ) is the j -th dimension of f ( x ) .

Definition 4.4 (exact feature identifiability) . Let ∼ be a pairwise relation in the encoder function space F = { f : X → R k } defined as:

4.2 Tri-factor Contrastive Learning with Exact Feature Identifiability

Motivated by the trifactorization technique in matrix decomposition problems [9] and the equivalence between the spectral contrastive loss and the matrix decomposition objective [18], we consider adding a learnable diagonal matrix when calculating the feature similarity in the contrastive loss to absorb

the freedom of linear transformations. To be specific, we introduce a contrastive learning model that enables exact feature identifiability, named tri-factor contrastive learning (triCL) , which adopts a tri-term contrastive loss:

where we name S = diag( s 1 , . . . , s k ) as the importance matrix and it is a diagonal matrix with k non-negative learnable parameters satisfying s 1 ≥ · · · ≥ s k ≥ 0 3 . Additionally, the encoder f is constrained to be decorrelated, i.e.,

̸

for an encoder f : R d → R k . One way to ensure feature decorrelation is the following penalty loss,

leading to a combined triCL objective,

Similar feature decorrelation objectives have been proposed in non-contrastive visual learning methods with slightly different forms, e.g., Barlow Twins [36].

Since triCL automatically learns feature importance S during training, it admits a straightforward feature selection approach. Specifically, if we need to select m out of k feature dimensions for downstream tasks ( e.g., in-time image retrieval), we can sort the feature dimensions according to their importance s i 's (after training), and simply use the top m features as the most important ones. Without loss of generality, we assume s 1 ≥ · · · ≥ s k , and the top m features are denoted as f ( m ) .

Identifiability of TriCL. In the following theorem, we show that by incorporating the diagonal importance matrix S that regularizes features along each dimension, triCL can resolve the linear ambiguity of contrastive learning and become sign-identifiable.

̸

̸

Theorem 4.5. Assume the normalized adjacent matrix ¯ A has distinct largest k singular values ( ∀ i, j ∈ [ k ] , σ i = σ j when i = j ) and the neural networks are expressive enough, the tri-factor contrastive learning (triCL, Eq. 9) attains its optimum when ∀ x ∈ D , j ∈ [ k ]

which states that the tri-factor contrastive learning enables the sign feature identifiability .

As shown in Theorem 4.5, the only difference remaining in the solutions (Eq. 10) is the sign. To remove this ambiguity, for dimension j , we randomly select a natural sample ¯ x ∼ P u , encode it with the optimal solution f ⋆ of triCL, and observe the sign of its j -th dimension f ⋆ j (¯ x ) . If f ⋆ j (¯ x ) = 0 , we draw another sample and repeat the process until we obtain a non-zero feature f ⋆ j (¯ x ) . We then store the sample as an original point x 0 j and adjust the sign of different learned representations as follows:

¯ f ⋆ j ( x ) = ( -1) ✶ ( f ⋆ j ( x 0 j ) > 0) · f ⋆ j ( x ) , ∀ x ∈ D , j ∈ [ k ] (11) By removing the freedom of sign, the solution becomes unique and triCL enables the exact feature identifiability:

Corollary 4.6. Set ¯ f ⋆ as the final learned encoder of tri-factor contrastive learning, and then triCL obtains the exact feature identifiability.

5 Theoretical Properties of Tri-factor Contrastive Learning

Besides the feature identifiability, we analyze other theoretical properties of triCL in this section. Specifically, in Section 5.1, we provide the generalization guarantee of triCL and we present another advantage of triCL: triCL can automatically discover the importance of different features. In Section 5.2, we extend triCL to other contrastive learning frameworks.

5.1 Downstream Generalization of Tri-factor Contrastive Learning

̸

Theorem 5.1. We denote α as the probability that the natural samples and augmented views have different labels, i.e., α = E ¯ x ∼P u E x ∼A ( ·| ¯ x ) ✶ [ y (¯ x ) = y ( x )] . Let f ⋆ SCL and ( f ⋆ triCL , S ⋆ ) be the optimal solutions of SCL and triCL, respectively. Then, SCL and triCL have the same downstream classification error when using all features

In the last section, we propose a new contrastive model (triCL) that enables exact feature identifiability. In the next step, we aim to theoretically discuss the downstream performance of the identifiable representations learned by triCL. For the ease of our theoretical analysis, we first focus on a common downstream task: Linear Probing. Specifically, we denote the linear probing error as the classification error of the optimal linear classifier on the pretrained representations, i.e., E ( f ) = min g E ¯ x ∼P u ✶ [ g ( f (¯ x )) = y (¯ x )] . By analyzing the optimal solutions of triCL when using the full or partial features, we obtain the following theorem characterizing their downstream performance.

̸

where σ i is the i -th largest eigenvalue of ¯ A , and c 1 , c 2 are constants. When using only m ≤ k features of the optimal features, triCL with the top m features admits the following error bound

Instead, without importance information, we can only randomly select m SCL features (denoted as f ( m ) SCL ), which has the following error bound (taking expectation over all random choices)

Comparing the two upper bounds, we can easily conclude that

Thus, triCL admits a smaller error when using a subset of features for downstream classification.

Theorem 5.1 shows that triCL is particularly helpful for downstream tasks when we select a subset of features according to the learned feature importance. When using all features, the two methods converge to the same downstream error bound, which also aligns with our observations in practice.

5.2 Extensions to Other Contrastive Learning Frameworks

In the above sections, we propose tri-factor contrastive learning (triCL) which enables the exact feature identifiability and obtains an ordered representation while achieving the guaranteed downstream performance. In the next step, we extend the triCL to a unified contrastive learning paradigm that can be applied in different contrastive learning frameworks.

Extension to Other SSL Methods. As replacing the 2-factor contrast f ( x ) ⊤ f ( x ′ ) with the 3-factor contrast f ( x ) ⊤ Sf ( x ′ ) is a simple operation and not constrained to the special form of spectral contrastive loss, we extend triCL to other contrastive frameworks. We first take another representative contrastive learning method SimCLR [4] as an example. Comparing the InfoNCE loss in SimCLR and the spectral contrastive loss, we find they are quite similar and the only difference is that they push away the negative pairs with different loss functions ( l 2 loss v.s. logsumexp loss). So we propose the tri-InfoNCE loss by changing the terms of negative samples in triCL (Eq. 9) to a tri-logsumexp term, i.e.,

Besides the InfoNCE loss used in SimCLR, we present more types of tri-factor contrastive loss in Appendix C. In summary, triCL can be applied in most of the contrastive frameworks by replacing the f ( x ) ⊤ f ( x ′ ) term with a tri-term f ( x ) ⊤ Sf ( x ′ ) when calculating the similarity, where the importance matrix S is a diagonal matrix capturing the feature importance.

Extension to the Multi-modal Domain. Different from the symmetric contrastive learning frameworks like SCL [18] and SimCLR [4], the multi-modal contrastive frameworks like CLIP [28] have asymmetric networks to encode asymmetric image-text pairs. So we extend the tri-factor contrastive loss to a unified asymmetric form:

where f A , f B are two different encoders with the same output dimension k . Uniformly, we denote that the positive pairs ( x a , x b ) ∼ P O and the negative samples x -a , x -b are independently drawn from P A , P B . With different concrete definitions of P O , P A , P B , different representation learning paradigms can be analyzed together. For single-modal contrastive learning, the symmetric process in Section 3 is a special case of the asymmetric objective. For multi-modal learning, we denote P A , P B as the distributions of different domain data and P O as the joint distribution of semantically related pairs (e.g., semantically similar image-text pairs in CLIP). We theoretically prove that the asymmetric tri-factor contrastive learning still obtains the exact feature identifiability and guaranteed generalization performance in Appendix A.

6 Experiments

In this section, we provide empirical evidence to support the effectiveness of tri-factor contrastive learning. In section 6.1, we empirically verify that the tri-factor contrastive learning can enable the exact feature identifiability on the synthetic dataset. In section 6.2, we empirically analyze the properties of the importance matrix and demonstrate the advantages of automatic feature importance discovery on various tasks. Furthermore, in section 6.3, we empirically compare the generalization performance of tri-factor contrastive learning with different baselines including SimCLR [4] and spectral contrastive learning (SCL) [18] on CIFAR-10, CIFAR-100 and ImageNet-100.

6.1 The Verification of the Identifiability on the Synthetic Dataset

Due to the randomness of optimization algorithms like SGD, we usually can not obtain the optimal encoder of the contrastive loss. So we consider verifying the feature identifiability on the synthetic dataset. To be specific, we first construct a random matrix ¯ A with size 5000 × 3000 to simulate the augmentation graph. Then we compare two different objectives: ∥ ¯ A -FG ⊤ ∥ 2 F , ∥ ¯ A -FSG ⊤ ∥ 2 F , where F is a matrix with size 5000 × 256 , G is a matrix with size 3000 × 256 and S is a diagonal with size 256 × 256 . With the analysis on [18], these two objectives are respectively equivalent to the spectral contrastive loss and tri-factor contrastive loss. According to the Eckart-Young Theorem [12], we obtain the optimal solutions of them with the SVD algorithms. For two objectives, we respectively obtain 10 optimal solutions and we calculate the average Euclidean distance between 10 solutions. We investigate that the optimal solutions of the trifactorization objective are equal (average distance is 0) while there exist significant differences between optimal solutions of bifactorization (average distance is 101.7891 and the variance of distance is 17.2798), which empirically verifies the optimal solutions of contrastive learning obtain the freedom while triCL can remove it. More details can be found in Appendix B.

6.2 The Automatic Discovery of Feature Importance

Except for the visualization example in Figure 1, we further quantitatively explore the properties of ordered representations learned by triCL.

The Distribution of the Importance Matrix. We first observe the distribution of the diagonal values in the learned importance matrix S . Specifically, we pretrain the ResNet-18 on CIFAR-10, CIFAR-100 and ImageNet-100 [8] by triCL. Then we normalize the importance matrices and ensure the sum of the diagonal values is 1. In Figure 2(a), we present the diagonal values of the importance

0

50

100

150

200

250

Dimension

0.00325

0.00350

0.00375

0.00400

0.00425

0.00450

0.00475

Normalized Diagonal Value

CIFAR10

CIFAR100

ImageNet100

(a) The distributions of diagonal values on the importance matrix learned on different datasets.

0

CIFAR10

CIFAR100

ImageNet100

50

100

150

200

250

Dimension

matrices trained on different datasets and we find they are highly related to the properties of the datasets. For instance, CIFAR-100 and ImageNet-100, which possess a greater diversity of features, exhibit more active diagonal values in the top 50 dimensions compared to CIFAR-10. While for the smallest diagonal values, the different importance matrices are quite similar as most of them represent meaningless noise.

The K-NN Accuracy on Selected Dimensions. With the ResNet-18 and the projector pretrained on CIFAR-10, CIFAR-100 and ImageNet-100, we first sort the dimensions according to the importance matrix S . Then we conduct the k-NN evaluation for every 10 dimensions. As shown in Figure 2(b), the k-NN accuracy decreases slowly at first as these dimensions are related to the ground-truth labels and they share a similar significance. Then the k-NN accuracy drops sharply and the least important dimensions almost make no contributions to clustering the semantically similar samples. Note that the k-NN accuracy drops more quickly on CIFAR-10, which is consistent with the fact it has fewer types of objects.

Linear Evaluation on Selected Dimensions. We first train the ResNet-18 with triCL and spectral contrastive learning (SCL) on CIFAR-10. Then we select 20 dimensions from the learned representations (containing both the backbone and the projector). For triCL, we sort the dimensions according to the descending order of the importance matrix and select the largest 20 dimensions (1-20 dimensions), middle 20 dimensions (119-138 dimensions), and smallest 20 dimensions (237-256 dimensions). And for SCL, we randomly choose 20 dimensions. Then we train a linear classifier following the frozen representations with the default settings of linear probing. As shown in Figure 3(a), the linear accuracy decreases in the descending order of dimensions in triCL and the linear accuracy of the largest 20 dimensions of triCL is significantly higher than random 20 dimensions of SCL, which verifies that triCL can discover the most important semantic features related to the ground-truth labels.

Image Retrieval. We conduct the image retrieval on ImageNet-100. For each sample, we first encode it with the pretrained networks and then select dimensions from the features. For the methods learned by triCL, we sort the dimensions according to the values in the importance matrix S and select the largest dimensions. For SCL, we randomly choose the dimensions. Then we find 100 images that have the largest cosine similarity with the query image and calculate the mean average precision (mAP) that returned images belong to the same class as the query ones. As shown in Figure 3(b), we observe the 50 dimensions of triCL show the comparable performance as the complete representation while the performance of SCL continues increasing with more dimensions, which further verifies that triCL can find the most important features that are useful in downstream tasks.

Out-of-Distribution Generalization. Besides the in-domain downstream tasks, we also examine whether the importance matrix can sort the feature importance accurately with out-of-domain shifts. To be specific, we prertain the ResNet-18 with SCL and triCL on ImageNet-100 and then conduct k-NN Accuracy

50

40

30

20

10

0

1-20

(triCL)

119-138

(triCL)

237-256

(triCL)

random

(SCL)

0

10

20

30

40

50

60

70

80

Selected Dimensions

50

100

150

200

250

Number of Dimensions

0.0

0.1

0.2

0.3

0.4

0.5

0.6

mAP

most important dimensions in triCL

least important dimensions in triCL

random dimensions in SCL

1-40

(triCL)

237-276

(triCL)

473-512

(triCL)

random

(SCL)

0

5

10

15

20

25

Selected Dimensions

linear probing with 40 dimensions of learned representations on the out-of-domain dataset stylized ImageNet-100 [13]. During the downstream evaluation process, we sort the dimensions according to the descending order of the importance matrix and select the largest 40 dimensions (1-40 dimensions), middle 40 dimensions (237-276 dimensions), and smallest 40 dimensions (473-512 dimensions) of the features learned by triCL. And for SCL, we randomly select 40 dimensions. As shown in Figure 3(c), we observe that the transfer accuracy of the most important features learned by triCL shows significantly superior performance, which shows the robustness and effectiveness of the importance matrix.

6.3 Transfer Learning on Benchmark Datasets

Setups. During the pretraining process, we utilize ResNet-18 [19] as the backbone and train the models on CIFAR-10, CIFAR-100 and ImageNet-100 [8]. We pretrain the model for 200 epochs On CIFAR-10, CIFAR-100, and for 400 epochs on ImageNet-100. We select two self-supervised methods as our baselines and apply tri-factor contrastive learning on them, including spectral contrastive learning (SCL) [18], SimCLR [4]. When implementing the tri-factor contrastive learning, we follow the default settings of the baseline methods. During the evaluation process, we consider two transfer learning tasks: linear evaluation and finetuning. During the linear evaluation, we train a classifier following the frozen backbone pretrained by different methods for 50 epochs. During the finetuning process, we train the whole network (including the backbone and the classifier) for 30 epochs.

Results. As shown in Table 1, we find that triCL shows comparable performance as the original contrastive learning methods in transfer learning on different real-world datasets, which is consistent with our theoretical analysis. Meanwhile, we observe that triCL shows significant improvements on some tasks, e.g., triCL improves the finetune accuracy of SCL by 1.7% on CIFAR-100. Compared to the original contrastive learning methods, the representations learned by triCL keep comparable transferability with stronger identifiability and interpretability.

7 Conclusion

In this paper, we propose a new self-supervised paradigm: tri-factor contrastive learning (triCL) which replaces the traditional 2-factor contrast with a 3-factor form z ⊤ x Sz ′ x when calculating the similarity in contrastive learning, where S is a learnable diagonal matrix named importance matrix. With the importance matrix S , triCL enables the exact feature identifiability. Meanwhile, the diagonal values in the importance matrix reflect the importance of different features learned by triCL, which means triCL obtains the ordered representations. Moreover, we theoretically prove that the generalization performance of triCL is guaranteed. Empirically, we verify that triCL achieves the feature identifiability, automatically discovers the feature importance and achieves the comparable transferability as current contrastive learning methods.

References

Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106.09943 , 2021.

Han Bao, Yoshihiro Nagano, and Kento Nozawa. Sharp learning bounds for contrastive unsupervised representation learning. arXiv preprint arXiv:2110.02501 , 2021.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , 2021.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 , 2021.

Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Chris Ding, Tao Li, Wei Peng, and Haesun Park. Orthogonal nonnegative matrix t-factorizations for clustering. In SIGKDD , 2006.

Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010.14407 , 2020.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV , 2021.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, 1936.

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 , 2018.

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR , 2018.

Jean-Bastien Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: a new approach to self-supervised learning. In NeurIPS , 2020.

Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. In ICLR , 2023.

Jeff Z HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. arXiv preprint arXiv:2211.14699 , 2022.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. In NeurIPS , 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 , 2021.

Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In NeurIPS , 2016.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR , 2022.

Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In ICML , 2016.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS , 2020.

Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In NeurIPS , 2019.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In ICML , 2021.

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML , 2019.

Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In ICML , 2022.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020.

Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS , 2021.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR , 2022.

Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR , 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In ICML , 2021.

Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS , 2022.

Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML , 2023.

Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In ICLR , 2023.

A Proofs

A.1 Proof of Theorem 4.5

Proof. We first rewrite L tri as a matrix decomposition objective

According to the Eckart-Young Theorem [12], the optimal solutions F ⋆ , S ⋆ satisfy

where Σ ∈ R k × k is a diagonal matrix with the k -largest eigenvalues of ¯ A and U ∈ R N × k contains the corresponding eigenvectors of the k -largest eigenvalues. When the regularizer L Dec is minimized, F ⋆ satisfy ( F ⋆ ) ⊤ F ⋆ = I . In the next step, we prove the uniqueness of the optimal solution.

We denote H = F ⋆ Σ( F ⋆ ) ⊤ . As ( F ⋆ ) ⊤ F ⋆ = I , we obtain HH ⊤ = F ⋆ S ⋆ ( S ⋆ ) ⊤ ( F ⋆ ) ⊤ . If ζ , σ are a pair of eigenvector and eigenvalue of HH ⊤ , we have

So the eigenvalues of HH ⊤ are the eigenvalues of S ⋆ ( S ⋆ ) ⊤ . As the positive eigenvalues of HH ⊤ are uniquely determined and S ⋆ has a descending order, S ⋆ is also determined and S ⋆ = ∑ .

We note that HH ⊤ = F ⋆ S ⋆ ( S ⋆ ) ⊤ ( F ⋆ ) ⊤ , i.e., HH ⊤ F ⋆ = F ⋆ S ⋆ ( S ⋆ ) ⊤ , which means that the k columns of F ⋆ are the eigenvectors of HH ⊤ and the corresponding eigenvalues are σ 1 · · · σ k . As HH ⊤ only has k different non-negative eigenvalues σ 1 , · · · , σ k , the eigenspace of each eigenvalue is one-dimensional. When we consider the real number space, any two eigenvectors ζ i , ζ ′ i of the same eigenvalue σ i satisfy ζ i = cζ ′ i . As ( F ⋆ ) ⊤ F ⋆ = I , we obtain c = ± 1 . As f ( x ) = 1 √ D xx F x , we obtain

A.2 Proof of Theorem 5.1

Proof. The following analysis mainly follows the proofs in [37].

We denote d x as the x -th row of D . And we denote F m as the matrix composed of encoder features on selected dimenstions, i.e., ( F m ) x = √ d x f ( m ) ( x ) . Recall that A x,x + = A ( x, x + ) = E ¯ x ∼P u [ A ( x | ¯ x ) A ( x + | ¯ x )] and ¯ A is the normalized form of A , i.e., ¯ A x,x + = A ( x,x + ) √ d x · d x + . Then we reformulate the downstream error,

where C x,j = √ ( d i ) ✶ y x = j . Then we consider the relationship between the downstream error and the augmentation graph, we element-wise consider the matrix ( D 1 / 2 Y -¯ AC ) ,

So when j = y x ,

̸

̸

̸

̸

̸

̸

With that, we obtain ∥ D 1 / 2 Y -¯ AC ∥ = ∑ x 2 β 2 x d x . As we assume that E ¯ x ∼P u ( A ( x | ¯ x ) ✶ y x =¯ y ) = α , so

̸

̸

When j = y x ,

We define β x = ∑ x + A ( x, x + ) ✶ y x + = y x , and we have

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

Then we have

Then we obtain

̸

In the next step, we analyze the prediction error. We denote ¯ y as the ground-truth label of original data ¯ x . We first define an ensembled linear predictor p ′ f . For an original sample, the predictor ensembles the results of all different views and chooses the label predicted the most. With the definition, ¯ y = p ′ f (¯ x ) only happens when more than half of the views predict wrong labels. So

̸

̸

Then we plug the optimal solutions of SCL (Lemma 4.1) in to F m , and we obtain

When we plug the optimal solutions of triCL (Theorem 4.5) into F m , we obtain:

A.3 Feature Identifiability of Asymmetric Tri-Factor Contrastive Learning

We first extend the augmentation graph to an asymmetric form. The asymmetric augmentation graph is defined over the set of all samples with its adjacent matrix denoted by P O . In the augmentation graph, each node corresponds to a sample, and the weight of the edge connecting two nodes x A and x B is equal to the probability that they are selected as a positive pair, i.e., ( P O ) x a ,x b = P O ( x a , x b ) . And we denote ¯ P O as the normalized adjacent matrix of the augmentation graph, i.e., ( ¯ P O ) x a ,x b = P O ( x a ,x b ) 2 P A ( x a ) P B ( x b ) .

̸

̸

Similar to the symmetric form, we then rewrite L tri as a matrix decomposition objective

According to the Eckart-Young Theorem [12], the optimal solutions F ⋆ A , S ⋆ , F ⋆ B satisfy

where Σ ∈ R k × k is a diagonal matrix with the k -largest eigenvalues of ¯ P O and U ∈ R N A × k contains the corresponding eigenvectors of the k -largest eigenvalues. When the regularizer L Dec is minimized, F ⋆ A and F ⋆ B satisfy ( F ⋆ A ) ⊤ F ⋆ A = I , ( F ⋆ B ) ⊤ F ⋆ B = I . In the next step, we prove the uniqueness of the optimal solution.

We denote H = F ⋆ A Σ F ⋆ B , and we obtain HH ⊤ = F ⋆ A S ⋆ ( S ⋆ ) ⊤ ( F ⋆ A ) ⊤ . If ζ , σ are a pair of eigenvector and eigenvalue of HH ⊤ , we have

So the eigenvalues of HH ⊤ are the eigenvalues of S ⋆ ( S ⋆ ) ⊤ . As the positive eigenvalues of HH ⊤ are uniquely determined and S ⋆ has an increasing order, S ⋆ is also determined and S ⋆ = ∑ .

We note that HH ⊤ = F ⋆ A S ⋆ ( S ⋆ ) ⊤ ( F ⋆ A ) ⊤ , i.e., HH ⊤ F ⋆ A = F ⋆ A S ⋆ ( S ⋆ ) ⊤ , which means that the k columns of F ⋆ A are the eigenvectors of HH ⊤ and the corresponding eigenvalues are σ 1 · · · σ k . As HH ⊤ only has k different non-negative eigenvalues σ 1 , · · · , σ k , the eigenspace of each eigenvalue is one-dimensional. When we consider the real number space, any two eigenvectors ζ i , ζ ′ i of the same eigenvalue σ i satisfy ζ i = cζ ′ i . As ( F ⋆ A ) ⊤ F ⋆ A = I , we obtain c = +1 . Then we eliminate the ambiguity of the sign following Eq 11 and F ⋆ A is unique. Similarly, F ⋆ B is also unique. So the optimal solution of L triCLIP is unique.

B Experimental Details

B.1 Experiment Details of Section 6.1

We first generate a random matrix A with size 5000 × 3000 , and make sure that it does not contain multiple eigenvectors (which is easy to satisfy). For the matrix factorization problem ∥ A -FG ⊤ ∥ 2 F , we apply off-the-shelf algorithms and repeat this process ten times. We then calculate the mean and variance of the l 2 pairwise distance between the obtained solutions of F . For the trifactorization objective ∥ A -FSG ⊤ ∥ 2 F , we use SVD to obtain an initial solution and apply the sign identification procedure to determine the sign of each eigenvector. Similarly, we also repeat this process ten times and calculate the mean and variance of the l 2 pairwise distance between different solutions.

B.2 Experiment Details of Section 6.2

Pretraining Setups. For different evaluation tasks (k-NN, linear evaluation, image retrieval), we use the same pretrained models. We adopt ResNet-18 as the backbone. For CIFAR-10 and CIFAR-100, the projector is a two-layer MLP with hidden dimension 2048 and output dimension 256. And for ImageNet-100, the projector is a two-layer MLP with hidden dimension 4096 and output dimension 512. We pretrain the models with batch size 256 and weight decay 0.0001. For CIFAR-10 and CIFAR-100, we pretrain the models for 200 epochs. While for ImageNet-100, we pretrain the models for 400 epochs. We use the cosine anneal learning rate scheduler and set the initial learning rate to 0.4 on CIFAR-10, CIFAR-100, and 0.3 on ImageNet-100.

As the importance matrix is learned on the projection layer, we conduct the downstream tasks on the features encoded by the complete networks (containing both the backbones and the projectors).

The Distribution of the Importance Matrix. When observing the distribution of feature importance discovered by the importance matrix S , we first apply the softplus activation functions on the diagonal values of S and sort different rows of S by the descending order of corresponding diagonal values in S . We denote the non-negative ordered diagonal values of S as ( s 1 , · · · , s k ) . When we present the distribution of them in Figure 2(a), we normalize the diagonal values and obtain

The K-NN Accuracy on Selected Dimensions. For k-NN evaluation on 10 selected dimensions, we do not finetune the models. We sort the dimensions of f ( x ) by the descending order of corresponding diagonal values in the importance matrix. The k-NN is conducted on the standard split of CIFAR-10, CIFAR-100 and ImageNet-100 and the predicted label of samples is decided by the 10 nearest neighbors.

Linear Evaluation on Selected Dimensions. We train the linear classifier on 20 dimensions of the frozen networks for 30 epochs during the linear evaluation. We set batch size to 256 and weight decay to 0.0001. For triCL, we sort the dimensions by descending order of the importance matrix. And for SCL, we randomly choose 20 dimensions.

C More Extensions of Tri-factor Contrastive Learning

In this section, we apply tri-factor contrastive learning to another representative contrastive learning objective: the non-contrastive loss [15, 5].

Besides contrastive learning, non-contrastive learning is another popular self-supervised framework that throws the negative samples in contrastive learning and learns the meaningful representations only by aligning the positive pairs. Taking the state-of-the-art algorithm BYOL [15] as an example, they use an MSE loss:

where g ( x ) and f ( x ) are two different networks to avoid the feature collapse. Then we consider adapting the tri-term loss to the non-contrastive learning, i.e.,

It is noticed that BYOL utilizes the stop-gradient technique on the target network f and it is updated by exponential moving average. So we only calculate the feature decorrelation loss on the online network g .

Peer reviews:
comments: SUMMARY:
In this paper, the authors proposed a new contrastive learning scheme (tri-contrastive learning) with a learnable diagonal matrix, which can capture the feature importance. To show the significance of the proposed method, the authors conducted both theoretical and experimental analysis, where some interesting findings are observed. However, it is not clear how the proposed method advances existing contrastive learning-based methods, and experimental results are not very convincing. The detailed comments are below.

STRENGTHS:
The idea is simple and interesting, with reasonable theoretical analysis based on eigenspace.

WEAKNESSES:
1. The idea of analyzing feature importance for contrastive learning is not new [a]. While the proposed method is different from [a], the submission lacks analysis how the proposed method outperforms [a], especially based on the generalization capability the submission focused on.

[a] Accurate and Robust Feature Importance Estimation under Distribution Shifts, aaai21

2. The visualization example in fig 1 is not clear. Noted that the contrastive learning is performing on the feature space, it is not clear to me based on the paper how the feature space linked to the image space. 

3. The generalization bound is not very clear, the upper bound depends on the term \alpha, which relies on data augmentation. How different augmentations inclunce the term \alpha. In what condition that the sample and its augmented view could have different labels?

4. The improvement on three benchmark datasets are very marginal (almost no improvement), and the results can’t justify the generalization capability claimed in this paper, convergence analysis is also not presented.

QUESTIONS:
Please see above.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new contrastive triCL method, which extends the existing pair-wise contrastive framework. The core motivation is the inability of CL methods to deliver identifiable representations. To tackle this, the paper proposes factorizing the pair-wise similarity measure x^T@y into a bilinear x^T@S@y form, where learnable matrix S is a diagonal feature importance matrix; and theoretically proves the identifiability property. The paper also demonstrates possible automated feature importance discovery by analyzing the learned feature importance matrix.\n\nSTRENGTHS:\n1. The paper addresses an important problem of identifiability, which is often overlooked in contrastive learning.\n\n2. The proposed method is elegant in its simplicity. Also, due to its simplicity, it does not introduce any major computational overheads.\n\n3. Feature importance discovery can be an important component to speed up the inference (by excluding features of low importance)\n\nWEAKNESSES:\n1. In my opinion, the term tri-contrastive (triCL) does incorrectly reflect the essence of the method. The proposed method still is based on pair-wise contrast in a nutshell, although the contrast is in the factorized form. \n\n2. It seems that BarlowTwins tackles almost the same problem as the paper states in L33-34. On the technical side, the L_dec term Eq (8) seems to be a special case of the Barlow twins' objective. In that sense, the L_dec term in triCL can be considered not as just a regularization, but as a main source of the supervisory signal.\n\n3. The ablation is needed to verify the importance of the L_dec term in the triCL objective. I would be interested to see the results for L_dec, L_tri, and L_dec + L_tri objectives separately.\n\n4. Experimental evaluation is limited to relatively small-scale datasets and only two baseline contrastive learning methods.\n\nUPD: The rebuttal addressed most of my concerns.\n\nQUESTIONS:\n1. Why is the softmax in S? Wouldn't the softmax limit the flexibility of each feature dimension, when computing x^T@S@y similarity? Have authors tried other normalization approaches, e.g. sigmoid for each diagonal element of S?\n\n2. Does identifiability of representation always boils down to reproducibility within the scope of this work?\n\nLIMITATIONS:\nI suggest adding a paragraph on limitations, which would highlight the scope of application of the proposed approach (or is the proposed method expected to work better in all circumstances?)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
Traditional CL calculates the similarities between two samples while this paper proposes to insert a learnable diagonal matrix between two feature vectors multiplication and name it triCL. The authors claim and show the proposed method leads to the exact interpretability of learned features. Some theoretical analysis is provided.

STRENGTHS:
The paper is well written and the proposed idea is simple and may be useful for explaining the CL.

WEAKNESSES:
1. Lack of novelty: The proposed method is essentially a bilinear function, and it can be rewritten the same as conventional CL, i.e., $f_1^TSf_2 = (\tilde{S}f_1)^T(\tilde{S}f_2)$ where $S=\tilde{S}^T\tilde{S}$ since S is a nonnegative diagonal matrix, i.e., a kernel matrix. From this point, the new parameter $S$ can be integrated into the neural network training, without explicit consideration. $S$ may be useful for the interpretation of CL, but at this point, I do not see how different using $S$ is from using traditional features with some statistical pro-processing such as norm. At least, I do not see the necessity of the proposed method in terms of interpretability.

2. Weak experiments: From Table 1, the improvement is marginal and even worse than the baseline. This is probably because introducing bilinear forms makes the training easier stuck in local minima. Also, these results verify my thoughts about bilinear form in learning above. I suggest the authors to investigate more in training, e.g., plotting training curves and doing some comparisons.

QUESTIONS:
see my comments above.

LIMITATIONS:
no discussion on limitations is provided, and i suggest the authors to add this part
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

comments: SUMMARY:
The paper proposes a new contrastive learning objective that learns an additional diagonal matrix which forces the features to exhibit different importance. Theoretical results are provided to show that the new algorithm achieves similar downstream linear probe accuracy but also better interpretability / identifiability. Empirically, the authors test the new objective on several standard benchmark datasets and show that learned features exhibit better k-nn accuracy when a subset of dimensions are used, and the downstream linear probe accuracy can also be improved in certain cases.

STRENGTHS:
- The motivation is clear and the paper is written well.
- It's nice that the learned representations can have a clear subset of dimensions that captures most of the semantic information. This can be especially useful in many retrieval use cases where the dimensionality is the key bottleneck for scaling up the size of data. With this modified representation, people can easily select a subset of dimensions to construct less informative but more computational efficient representations for downstream use cases.
- The proposed method is justified with corresponding theory.

WEAKNESSES:
- Only small datasets are used. Could include larger ones like ImageNet.
- Theorem 5.1 seems to be exactly following that in [HaoChen et al], so I'm not sure there's enough novelty in this specific theorem.

QUESTIONS:
Is there a easy way to project the full feature from SCL or SimCLR to a 20-dimensional subspace and still match the same k-nn performance of the top 20 dimensions from tri-SCL? I wonder whether doing some PCA can achieve that.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07