Identifiable Contrastive Learning with Automatic Feature Importance Discovery
Authors: Qi Zhang
Yifei Wang
Yisen Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-supervised Learning
Contrastive Learning
Identifiability
Representation Learning

Abstract: Existing contrastive learning methods rely on pairwise sample contrast $z_x^\top z_{x'}$ to learn data representations, but the learned features often lack clear interpretability from a human perspective. Theoretically, it lacks feature identifiability and different initialization may lead to totally different features. In this paper, we study a new method named tri-factor contrastive learning (triCL) that involves a 3-factor contrast in the form of $z_x^\top S z_{x'}$, where $S=\text{diag}(s_1,\dots,s_k)$ is a learnable diagonal matrix that automatically captures the importance of each feature. We show that by this simple extension, triCL can not only obtain identifiable features that eliminate randomness but also obtain more interpretable features that are ordered according to the importance matrix $S$. We show that features with high importance have nice interpretability by capturing common classwise features, and obtain superior performance when evaluated for image retrieval using a few features. The proposed triCL objective is general and can be applied to different contrastive learning methods like SimCLR and CLIP. We believe that it is a better alternative to existing 2-factor contrastive learning by improving its identifiability and interpretability with minimal overhead. Code is available at https://github.com/PKU-ML/Tri-factor-Contrastive-Learning.

Identifiable Contrastive Learning with Automatic Feature Importance Discovery
Qi Zhang! Yifei Wang** Yisen Wangl
National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology. Peking University
2 School of Mathematical Sciences, Peking University Institute for Artificial Intelligence, Peking University
Abstract
Existing contrastive learning methods rely on pairwise sample contrast x 2x² to learn data representations, but the learned features often lack clear interpretability from human perspective. Theoretically. it lacks feature identifiability and different initialization may lead totally different features. In this paper, study a new method named tri-factor contrastive learning (triCI.) that involves a 3-factor contrast in the form of z SSg', where S diag( -1, 8&) is learnable diagonal matrix that automatically captures the importance of each feature. We show that by this simple extension, triCL can not only obtain identifiable features that eliminate randomness but also obtain more interpretable features that are ordered according the importance matrix We show that features with high importance have nice interpretability by capturing common classwise features, and obtain superior performance when evaluated for image retrieval using a few features. The proposed triCL objective is general and can be applied to different contrastive learning methods like SimCLR and CLIP. We believe that it is better alternative to existing 2-factor contrastive learning by improving its identifiability and interpretability with minimal overhead. Code available at https: /tte:///ttttbt cww/PKU-UL Sri-factor--Cntraastee Learning.
Introduction
As a representative self-supervised paradigm, contrastive learning obtains meaningful representations and achieves vate-of-the-ar performance in various tasks by maximizing the feature similarity 2 = between samples augmented from the same images while minimizing the similarity x zニ between independent samples [4, 20, 15, 5, 18]. Besides the empirical success, recent works also discuss the theoretical properties and the generalization performance of contrastive learning [32, 30, 18].
guaranteed. In this paper, we focus on significant one: the feature identifiability. Feature identifiability in the representation learning refers the property there exists single, global optimal solution the learning objective. Consequently, the learned representations can be reproducible regardless of the initialization and the optimizing procedure useful. As well-studied topic, identifiability is a desirable property for various tasks, including but not limited to, transfer learning [10], fair classification [26] and causal inference [24]. The previous works propose that contrastive learning obtains the linear feature identifiability while lacking exact feature identifiability, i.c.. the optimal solutions obtain a freedom of linear transformations [29]. As a result, the different features are coupled, which hurts the interpretability and performance of learned representations.
However, there still exist many properties of contrastive learning that
Equual Contribution.
Corresponding Author: Yisen Wang (yisen. wang @pku.edu.cn).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) 10 most important dimensions
(b) 10 Ieast important dimensions
Figure 1: The visualization of samples on ImageNet- that have the largest values in 10 selected dimensions of representations learned by tri-factor contrastive learning (each row represents dimension).
In this paper, we propose a new contrastive learning model: tri-factor contrastive learning (triCI., which introduces 3-factor contrastive loss, i.e., we replace 2 2n. with 2._ Se., when calculating the similarity between two features, where is learnable diagonal matrix called importance matrix. We theoretically prove that triCL absorbs the freedom of linear transformations and enables exact feature identifiability. Besides, we observe that triCL shows other satisfying properties. For example, the generalization performance of triCL is theoretically guaranteed. What is more, we find that the diagonal values of the importance matrix S in triCL indicate the degrees of feature importance. In Figure 1, we visualize the samples that have the largest values in the most and least important dimensions ordered by the importance matrix. We find the samples activated the more important dimensions more semantically similar, which verifies the order of feature importance in triCI. is quite close to the ground truth. Theoretically, we prove that the dimensions related to the larger values in the importance matrix make more contributions decreasing the triCL loss. With the automatic discovery of feature importance in triCL, the downstream task conducted on the representations can be accelerated by selecting the important features and throwing the meaningless features.
As triCL a quite simple and general extension, we apply to different contrastive learning frameworks, such SimCLR [4] and CLIP [28]. Empirically, we first verify the identifiability of triCL and further evaluate the performance of triCL on real-world datasets including CIFAR- 10, CIFAR- 100, and ImageNet- 100. In particular, with the automatic discovery of important features, triCL demonstrates significantly better performance on the downstream tasks using a few feature dimensions. We summarize our contributions as follows:
We propose tri-factor contrastive learning (triCL), the first contrastive learning algorithm which enables exact feature identifiability. Additionally, we extend triCL to different contrastive learning methods, such as spectral contrastive learning, SimCLR and CLIP
Besides the feature identifiability, we analyze several theoretical properties of triCL. Specifi- cally, we construct the generalization guarantee for triCL and provide theoretical evidence that triCL can automatically discover the feature importance.
Empirically, we verify that triCL enables the exact feature identifiability and triCL can discover the feature importance the synthetic and real-world datasets. Moreover, we investigate whether triCI. obtains superior performance downstream tasks with selected representations.

---

2 Related Work
Self-supervised Learning. Recently, get rid of the expensive cost of the labeled data, self- supervised learning has risen to be a promising paradigm for learning meaningful representations by designing various pretraining tasks, including context-based tasks [14], contrastive learning [20] and masked image modeling [21]. Among them, contrastive learning is a popular algorithm that achieves impressive success and largely closes the gap between self-supervised learning and supervised learning [27, 4, 20, 33]. The idea of contrastive learning is quite simple, i.e. pulling the semantically similar samples (positive samples) together while pushing the dissimilar samples (negative samples) away in the feature space. To better achieve this objective, recent works propose different variants of contrastive learning, such as introducing different training objectives [36, 18, 35], different structures [4, 6, 16, 39], different sampling processes [11, 25, 38, 7] and different empirical tricks [20, 3].
Theoretical Understandings of Contrastive Learning. Despite the empirical success of contrastive learning. the theoretical understanding of it is still limited. [30] establish the first theoretical guarantee on the downstream classification performance of contrastive learning by connecting the InfoNCE loss and Cross Entropy loss. As there exists some unpractical assumptions in the theoretical analysis in [30], recent works improve the theoretical framework and propose new bounds [1, 2, 34j. More- over, [18] analyzes the downstream performance of contrastive learning from graph perspective and constructs the connection between contrastive learning and spectral decomposition. Recently, researchers have taken the inductive bias in contrastive learning into the theoretical framework and shown the influence of different network architectures [17, 31]. Except for the downstream classifica- tion performance of contrastive learning, some works focus on other properties of representations learned by contrastive learning- [23] discuss the feature diversity by analyzing the dimensional collapse in contrastive learning. [29] prove that the contrastive models are identifiable up to linear transformations under certain assumptions.
3 Preliminary
Contrastive Pretraining Process. We begin by introducing the basic notations of contrastive learning. The set of all natural data is denoted as DD, = { with distribution Pu, and each natural data T E DD,, has ground-truth label y()). An augmented sample x is generated by transforming natural sample with the augmentations distributed with A(- The set of all the augmented samples is denoted as D J. We assume both sets of natural samples and augmented samples to be finite but exponentially large to avoid non-essential nuances in the theoretical analysis and it can be easily extended to the case where they are infinite [18]. During the pretraining process, we first draw a natural sample 전 ~ Pu. and independently generate two augmented samples x ~ A((-))) independently draw another natural sample ă ~ Pu and generate x ~ A(. (i ). With positive and negative pairs, we learn the encoder f : .Rd -> IR* with the contrastive loss. For the ease of our analysis, we take the spectral contrastive loss [18] an example:
x+ A(-1z) to construct a positive pair (I, x For the negative samples,
(sCu(f) 2 f()) ((t))(+++EEE-(( ((xT))
We denote 2n f as the features encoded by the encoder. By optimizing the spectral loss, the features of positive pairs (2_ zz) ) are pulled together while the negative pairs (z 2g ) are pushed apart.
Augmentation Graph. A useful theoretical framework to describe the properties of contrastive learning is to model the learning process from the augmentation graph perspective [18]. The augmentation graph defined over the set of augmented samples DD with its adjacent matrix denoted by A. In the augmentation graph, each node corresponds to an augmented sample, and the weight of the edge connecting two nodes r and x equal to the probability that they are selected as a positive pair, i.c.,Artt = Es-P, ((x]()))((C matrix of the augmentation graph, i..., A = D
[z]] And we denote A as the normalized adjacent
1/2, where D is a diagonal matrix and Duz = E Azx' To analyze the properties of A, we denote A UVV T is the singular value decomposition (SVD) of the normalized adjacent matrix Å, where U E RNN, V E N are unitary matrices, and Σ- (iag, contains descending singular values 01 2 ...ON 2 0.
3

---

Exact Feature Identifiability with Tri-factor Contrastive Learning
In this section, we propose a new representation learning paradigm called tri-factor contrastive learning (trICL) that enables exact feature identifiability in contrastive learning. In Section 4. we prove that contrastive learning obtains linear identifiability, i.e. the freedom in the optimal solutions are linear transformations. In Section 4.2, we introduce the learning process of triCL and theoretically verify enables the exact feature identifiability.
4.1 Feature Identifiability of Contrastive Learning
When using pretrained encoder for downstream tasks, it is useful if the learned features are reproducible, in the sense that when the neural network learns the representation function on the same data distribution multiple times, the resulting features should be approximately the same. For example, reproducibility can enhance the interpretability and the robustness of learned representations [22]. One rigorous way to ensure reproducibility is to select model whose representation function is identifiable in function space [29]. To explore the feature identifiability of contrastive learning, we first characterize its general solution.
Lemma 4.1 ([18)) Let À UEVT be the SVD decomposition of the normalized adjacent matrix Å. Assume the neural networks are expressive enough for any features. The spectral contrastive loss (Eq. 1) attains its optimum when V E DD
(((x)
1 VDz..
1 d
(2)
where Uz takes the x-th row U, , U* denotes the submatrices containing the first k columns of U, and R E lgkkk is an arbitrary unitary matrix.
From Lemma 4.1, we know that the optimal representations are not unique, due to the freedom of affine transformations. This is also regarded as a relaxed notion of feature identifiability, named
linear feature identifiability A defined below [29].
Definition 4.2 (Linear feature identifiability). Let be a pairwise relation the encoder function space F = {ʃ X + IR*) defined as:
I' M A
where is an invertible & x matrix.
It is apparent that the optimal encoder f (Eq. 2) obtained from the spectral contrastive loss (Eq. 1) is linearly identifiable. Nevertheless, there are still some ambiguities w.r.f. linear transformations in the model. Although the freedom of linear transformations can be absorbed the linear probing task [18], these representations may show varied results in many downstream tasks e.g., in the k-NN evaluation process. So wonder whether could achieve the exact feature identifiability. We first further define two kinds of more accurate feature identifiabilities below.
Definition 4.3 (Sign feature identifiability). Let E be pairwise relation in the encoder function space F = {f + RR) defined as:
f
where 15(x) is the j-th dimension of ((x Definition 4.4 (exact feature identifiability). Let be pairwise relation in the encoder function
space F = { X + } defined as:
(5)
4.2 Tri-factor Contrastive Learning with Exact Feature Identifiability
Motivated by the trifactorization technique in matrix decomposition problems [9] and the equivalence
between the spectral contrastive loss and the matrix decomposition objective [18], we consider adding
learnable diagonal matrix when calculating the feature similarity in the contrastive loss to absorb

---

the freedom of linear transformations. To be specific, we introduce contrastive learning model that enables exact feature identifiability, named tri-factor contrastive learning (triCL), which adopts tri-term contrastive loss:
Lii (1,5) -2 2,+ f(x) Sf(xt) E. (f(x))((((()))
(6)
where we name S = diag(s1. (1............ ,8)) 8k) non-negative learnable parameters satisfying 81 2 2 sk 2 0 Additionally, the encoder f is constrained to be decorrelated, i.e.
the importance matrix and it is diagonal matrix with k:
(1, (0, ifi i.j [k].
if
Ex (x f())
for an encoder f: Rdd TR. One way to ensure feature decorrelation is the following penalty loss,
dee (ʃ) - -I
-11 -I
(8)
leading to a combined triCL objective,
[Hs Lu)(f, + Edec
99)
Similar feature decorrelation objectives have been proposed in non-contrastive visual learning methods with slightly different forms, e.g., Barlow Twins [36].
Since triCL automatically learns feature importance S during training. it admits straightforward feature selection approach. Specifically, if we need to select m out of feature dimensions for downstream tasks (e.g., in-time image retrieval), we can sort the feature dimensions according to their importance s; ss (after training), and simply use the top m features as the most important ones. Without loss of generality, we assume 81 2 2 8g, and the top features are denoted f f(m),
Identifiability of TriCL. the following theorem, we show that by incorporating the diagonal importance matrix S that regularizes features along each dimension, triCL can resolve the linear ambiguity of contrastive learning and become sign-identifiable.
Theorem 4.5. Assume the normalized adjacent matrix Á has distinct largest k singular values i.j E [k). ai # oj when * j) and the neural networks are expressive enough, the tri-factor contrastive learning (IriCI, Eq. 9) attains its optimum when V x. E DD [k]
VD +00)=+/_[t),..>> =iingo.s"i
(10)
which states that the tri-factor contrastive learning enables the sign feature identifiability.
As shown Theorem 4.5, the only difference remaining in the solutions (Eq. 10) is the sign. To remove this ambiguity, for dimension j. randomly select natural sample F ~ Pus encode it with the optimal solution f* of triCL, and observe the sign of its j-th dimension f If (z) = O, we draw another sample and repeat the process until we obtain a non-zero feature (z). We then store the sample original point zaj and adjust the sign of different learned representations follows:
(x) = (-1)1 (ff(()))))))) (x), Vr E D,j E [k]
(11)
By removing the freedom of sign, the solution becomes unique and triCL enables the exact feature identifiability:
Corollary 4.6. Set J* as the final learned encoder of tri-factor contrastive learning, and then triCL obtains the exact feature identifiabilly.
Theoretical Properties of Tri-factor Contrastive Learning
Besides the feature identifiability, analyze other theoretical properties of triCL in this section. Specifically, in Section 5. we provide the generalization guarantee of triCL and we present another advantage of triCL: triCL can automatically discover the importance of different features. In Section 5.2, we extend triCL to other contrastive learning frameworks.
"In practice we enforce the non non-negative conditions by applying the softplus activation functions on the diagonal values of S. We only enforce the monotonicity at the end of training by of S and different dimensions of [xx ) by the descending order of corresponding diagonal values in S.
simply sorting different rows
5

---

5.1 Downstream Generalization of Tri-factor Contrastive Learning
In the last section, we propose new contrastive model (triCL) that enables exact feature iden- tifiability. In the next step, we aim to theoretically discuss the downstream performance of the identifiable representations learned by triCL. For the ease of our theoretical analysis, we first fo- cus a common downstream task: Linear Probing. Specifically, we denote the linear probing error as the classification error of the optimal linear classifier on the pretrained representations, i.c., E(f) min, LE>P, (f # (z)]. By analyzing the optimal solutions of triCL when using the full or partial features, we obtain the following theorem characterizing their downstream performance.
Theorem 5.1. We denote a as the probability that the natural samples and augmented views have different labels, i.e., optimal solutions of SCL and triCL, respectively. Then, SCL and triCL have the same downstream classification ervor when using all features
Le, ((y()) + y(*)}. Let fecs. and (frric,, S*) be the
E(f5cL) E((FmoL)) Sa
Σ i=k+1
of + C2 a,
(12)
where oi the i-th largest eigenvalue of A, and C1, C2 are constants. When using only m S features of the optimal features, triCL with the top m features admits the following error bound
Sa Σ a= $=m+1
(13)
Instead, without importance information, we can only randomly select m SCLffatures (denoted as (m)) which has the following error bound (taking expectation over all random choices)
(a-
k
N
E(fSEL) Sa
m k
of +
U(fs(L) (1)
(14)
f==+1
Comparing the two upper bounds, we can easily conclude that
U(S())) - U(((()))) m(k m)
m
k
0
of 2 0.
(15)
m
m
{=m+1
Thus, triCL admits a smaller error when using a subset of features for downstream classification.
Theorem 5.1 shows that triCL is particularly helpful for downstream tasks when we select a subset of features according to the learned feature importance. When using all features, the two methods converge to the same downstream error bound, which also aligns with our observations in practice.
5.2 Extensions to Other Contrastive Learning Frameworks
In the above sections, we propose tri-factor contrastive learning (triCL) which enables the exact feature identifiability and obtains an ordered representation while achieving the guaranteed downstream performance. In the next step, we extend the triCL a unified contrastive learning paradigm that can be applied in different contrastive learning frameworks.
Extension to Other SSL Methods. As replacing the 2-factor contrast f f((')) with the f-factor contrast f(x) Sf(')) is a simple operation and not constrained to the special form of spectral contrastive loss, we extend triCL to other contrastive frameworks. We first take another representative contrastive learning method SimCLR [4] as example. Comparing the InfoNCE loss in SimCLR and the spectral contrastive loss, we find they are quite similar and the only difference is that they push away the negative pairs with different loss functions (22 loss v.s. logsumezp loss). So propose the Iri-InfoNCE loss by changing the terms of negative samples in triCI. (Eq. 9) to a tri-logsumexp term, i...,
exp ((())(((xtt))
LUNNCE(f) = -Eg. log E,- exp(s(x) Sf(x)) + [[..((
(16)

---

Besides the InfoNCE loss used in SimCLR, we present more types of tri-factor contrastive loss in Appendix C. summary, triCL can be applied in most the contrastive frameworks by replacing the f(x)) f(x') term with tri-term (ar) (f(²') when calculating the similarity, where the importance matrix S is diagonal matrix capturing the feature importance.
Extension to the Multi-modal Domain. Different from the symmetric contrastive learning frame- works like SCL [18] and SimCLR [4], the multi-modal contrastive frameworks like CLIP [28] have asymmetric networks to encode asymmetric image-texi pairs. So we extend the tri-factor contrastive loss unified asymmetric form:
2E,,, a. (1 T (Sn((i)) BBa (Ja(z)) ([((5))
(17)
+ II (((=)))a( -1] ([Be). r|p.
where fA, fB are two different encoders with the same output dimension k. Uniformly, we denote that the positive pairs (Ia, ~ Po and the negative samples Fg 27 are independently drawn from PA. Pg. With different concrete definitions of Po. PA. Pg, different representation learning paradigms can be analyzed together. For single-moda modal contrastive learning, the symmetric process in Section 3 is special case of the asymmetric objective. For multi- modal learning, we denote PA, Pn as the distributions of different domain data and Po as the joint distribution of semantically related pairs (e.g.. semantically similar image-text pairs in CLIP). We theoretically prove that the asymmetrie tri-factor contrastive learning still obtains the exact feature identifiability and guaranteed generalization performance in Appendix A.
6 Experiments
In this section, we provide empirical evidence to support the effectiveness of tri-factor contrastive learning. In section 6.1, we empirically verify that the tri-factor contrastive learning can enable the exact feature identifiability on the synthetic dataset. In section 6.2, we empirically analyze the properties of the importance matrix and demonstrate the advantages of automatic feature importance discovery on various tasks. Furthermore, in section 6.3, we empirically compare the generalization performance of tri-factor contrastive learning with different baselines including SimCLR [4] and spectral contrastive learning (SCL) [18] on CIFAR-10, CIFAR- 100 and ImageNet-100.
6.1 The Verification of the Identifiability on the Synthetic Dataset
Due to the randomness of optimization algorithms like SGD, we usually can not obtain the optimal encoder of the contrastive loss. So we consider verifying the feature identifiability on the synthetic dataset. To be specific, we first construct random matrix with size 5000 x 3000 simulate the augmentation graph. Then we compare two different objectives: || - FG || | - FSG" 1/?- where matrix with size 5000 x 256, G is a matrix with size 3000 x 256 and S is diagonal with size 256 x 256. With the analysis on [18], these two objectives are respectively equivalent to the spectral contrastive loss and tri-factor contrastive loss. According to the Eckart- Young Theorem [12], we obtain the optimal solutions of them with the SVD algorithms. For two objectives, respectively obtain 10 optimal solutions and we calculate the average Euclidean distance between 10 solutions. We investigate that the optimal solutions of the trifactorization objective are equal (average distance is 0) while there exist significant differences between optimal solutions of bifactorization (average distance is 101.7891 and the variance of distance is 17.2798), which empirically verifies the optimal solutions of contrastive learning obtain the freedom while triCL can remove it. More details can be found in Appendix B.
6.2
The Automatic Discovery of Feature Importance
Except for the visualization example in Figure 1, we further quantitatively explore the properties of ordered representations learned by triCL.
The Distribution the Importance Matrix. We first observe the distribution of the diagonal values in the learned importance matrix S. Specifically, we pretrain the ResNet-18 18 on CIFAR- 10, CIFAR-100 100 and ImageNet- 100 [8] by triCL. Then we normalize the importance matrices and ensure the sum of the diagonal values is I. In Figure 2(a), we present the diagonal values of the importance
7

---

CIFAR1O
Normalized Diegonal Value
0,00450
CIFAR100
ImageNett10
0.00425
40
K:NN Accuracy
0.00400
30
0.00375
20
10
CIFARID
0.00325
ImageNel100
100
200
Dimension
(a) The distributions of diagonal values the im-(b) K-NN accuracy adjacent 10 dimensions portance matrix Icarned different datasets.
dered the importance matrix learned on CIFAR- CIFAR- and ImageNet-100.
Figure 2: The distributions of the discovered feature importance and the verifications on whether the order of feature importance indicated by the importance matrix is accurate.
matrices trained on different datasets and we find they are highly related to the propertics of the datasets. For instance, CIFAR-100 and ImageNet- 100, which possess greater diversity of features, exhibit more active diagonal values in the top 50 dimensions compared to CIFAR-10. While for the smallest diagonal values, the different importance matrices are quite similar most of them represent meaningless noise.
The K-NN Accuracy on Selected Dimensions. With the ResNet-18 and the projector pretrained on CIFAR-10, CIFAR-100 and ImageNet- 100, we first sort the dimensions according to the importance matrix S. Then we conduct the k-NN evaluation for every 10 dimensions. As shown in Figure 2(b), labels
the k-NN accuracy decreases slowly at first as these dimensions are related to the ground and they share similar significance. Then the k-NN accuracy drops sharply and the least important dimensions almost make no contributions to clustering the semantically similar samples. Note that the k-NN accuracy drops more quickly on CIFAR-10, which is consistent with the fact it has fewer types of objects.
Linear Evaluation on Selected Dimensions. We first train the ResNet-18 18 with triCL and spectral contrastive learning (SCL) on CIFAR-10. Then we select dimensions from the learned repre- sentations (containing both the backbone and the projector). For triCL, we sort the dimensions according to the descending order of the importance matrix and select the largest 20 dimensions (1-20 dimensions), middle 20 dimensions (119-133 138 dimensions), and smallest 20 dimensions (237-256 dimensions). And for SCL, we randomly choose 20 dimensions. Then we train a linear classifier following the frozen representations with the default settings of linear probing. As shown in Figure 3()), the linear accuracy decreases in the descending order dimensions in triCL and the linear accuracy of the largest 20 dimensions of triCL is significantly higher than random 20 dimensions of SCL, which verifies that triCL can discover the most important semantic features related to the ground-truth labels.
We conduct the image retrieval on ImageNet- 100. each sample, we first encode it with the pretrained networks and then select dimensions from the features. For the methods learned by triCL, we sort the dimensions according to the values in the importance matrix S and select the largest dimensions. For SCL, randomly choose the dimensions. Then we find 100 images that have the largest cosine similarity with the query image and calculate the mean average precision (mAP) that returned images belong to the same class the query ones. As shown in Figure 3(b), we observe the 50 dimensions of triCL show the comparable performance as the complete representation while the performance of SCL continues increasing with more dimensions, which further verifies that triCL can find the most important features that are useful in downstream tasks. Out-of-Distribution Distribution Generalization. Besides the in-domain downstream tasks, also examine whether the importance matrix can sort the feature importance accurately with oft-of-domain shifts. To be specific, we prertain the ResNet-18 with SCL and triCL on ImageNet- 100 and then conduct 100
Image Retrieval.
Out-of-

---

70
.ross inoators dinenegne trG.
importa direnjees
1.50
122-3.6
223-255
30
1.30
350
203
200
2-40
227-277
(a) Linear evaluation results of se-
(b) The mean average precision in the image retrieval on the selected dimension of triCI. and SCL. on
(c) Transfer accuracy of selected dimensions of triCL and SCL stylized ImageNet- 100.
lected dimensions of triCL and
SCL CIFAR-10.
ImageNet-100.
Figure 3: With the automatic discovery of feature importance, Tri-factor contrastive learning (triCI.) obtains superior performance than spectral contrastive learning (SCL) on downstream tasks that use selected representations.
Table 1: The linear probing acuuracy and finetune accuracy of ResNet-18 pretrained by tri-factor contrastive learning and self-supervised baselines on CIFAR-10, CIFAR-100, ImageNet-100. TriCL averagely obtains a comparable performance as the original contrastive learning. Moreover, triCL achieves significant improvements on some tasks, such as finetuning on CIFAR-100.
Linear Accuracy
Finetune Accuracy CIFAR100
CIFAR10
CIFAR100
ImageNet100
CIFARIO
ImageNet100
SCL
88.4
60.3
72.3
92.3
66.1
76.4
tri-SCL 88.3
60.8
71.2
92.7
67.8
76.3
SimCLR
87.9
60.2
73.5
92.2
72.5
75.9
tri-SimCI.R
87.9
59.8
74.1
92.3
73.3
75.6
linear probing with 40 dimensions of learned representations on the out-of-domain dataset stylized ImageNer-100) [13]. During the downstream evaluation process, the descending order of the importance matrix and select the largest 40 dimensions (1-40 dimensions). middle 40 dimensions (237-276 dimensions), and smallest 40 dimensions (473-512 dimensions) of the features learned by triCL. And for SCL, we randomly select 40 dimensions. As shown in Figure 3(c), we observe that the transfer accuracy of the most important features learned by triCL shows significantly superior performance, which shows the robustness and effectiveness of the importance matrix.
the dimensions according to
6.3 Transfer Learning on Benchmark Datasets
Setups. During the pretraining process, we utilize ResNet- 18 [19] as the backbone and train the models on CIFAR- 10, CIFAR-100 100 and ImageNet-100 [8]. We pretrain the model for 200 epochs CIFAR-10, CIFAR-100, and for 400 epochs on ImageNet-100. We select two self-supervised methods as our baselines and apply tri-factor contrastive learning on them, including spectral contrastive learning (SCL) [18], SimCLR [4]. When implementing the tri-factor contrastive learning, we follow the default settings of the baseline methods. During the evaluation process, we consider two transfer learning tasks: lincar evaluation and finetuning. During the linear evaluation, following the frozen backbone pretrained by different methods for 50 epochs. During the finetuning process, train the whole network (including the backbone and the classifier) for 30 epochs.
train classifier
Results. As shown in Table 1, we find that triCL shows comparable performance as the original contrastive learning methods in transfer learning different real-world datasets, which is consistent with our theoretical analysis. Meanwhile, we observe that triCL shows significant improvements on some tasks, e.g., triCL. improves the finetune accuracy of SCL by 1.7% on CIFAR-100. Compared to the original contrastive learning methods, the representations learned by triCI. keep comparable transferability with stronger identifiability and interpretability.

---

7 Conclusion
In this paper, we propose a new self-supervised paradigm: tri tri-factor contrastive learning (triCL) which replaces the traditional 2-factor contrast with 3-factor form z Sz when calculating the similarity in contrastive learning. where S is learnable diagonal matrix named importance matrix. With the importance matrix S. triCL enables the exact feature identifiability. Meanwhile, the diagonal values in the importance matrix reflect the importance of different features learned by triCL, which means triCL obtains the ordered representations. Moreover, we theoretically prove that the generalization performance of triCL is guaranteed. Empirically, we verify that triCL achieves the feature identifiability, automatically discovers the feature importance and achieves the comparable transferability as current contrastive learning methods.
Acknowledgements
Yisen Wang was supported by National Key R&D Program of China (2022/D116034)) National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB0S), and Beijing Nova Program (20230484344).
References
[1] Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning, arXiv preprint arXiv:2106.0990 2021.
[2] Han Bao, Yoshihiro Nagano, and Kento Nozawa. Sharp learning bounds for contrastive unsupervised representation learning. arXiv preprint arXiv:2110.02501 2021.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CVPR, 2021.
[6] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
[7] Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[9] Chris Ding, Tao Li, Wei Peng, and Haesun Park. Orthogonal nonnegative matrix t-factorizations for clustering. In SIGKDD, 2006.
[10] Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint (tt:::2000.1440000 2020.
[11] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisser- man. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
[12] Carl Eckart and Gale Young. The approximation of matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.
[13] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12233 2018.
10

---

[14] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018
[15] Bran-Bastien Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: a new approach self-supervised learning. In NewrIPS, 2020.
[16] Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: contrastive learning perspective on oversmoothing and beyond. In ICLR, 2023.
[17] Jeff Z HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. arXiv preprint arXiv:2211. 14699, 2022.
[18] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurlPS, 2021.
[19] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
[22] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In NeurlPS, 2016.
[23] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
[24] Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In ICML, 2016.
[25] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS, 2020.
[26] Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. In NeurIPS, 2019.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 037448. 2018
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[29] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representa- tions. In ICML, 2021.
[30] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
[31] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy, Understanding contrastive learning requires incorporating inductive biases. In ICML, 2022.
[32] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[33] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NewrlPS, 2021.
11

---

[34] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
[35] Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang, Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
[36] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
[37] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS, 2022.
[38] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modie contrastive learning. In ICML, 2023.
[39] Zhijian Zhuo Yifei Wang, Jinwen Ma, and Yisen Wang. Towards unified theoretical under- standing non-contrastive learning via rank differential mechanism. In ICLR, 2023.
12

---

A Proofs
A.1 Proof of Theorem 4.5
Proof. We first rewrite Liri as a matrix decomposition objective
Lirlf) = -2 +(x))( Sf(x*) + E,E,- (f(=))
A
=E
=
De'd + DreDE's (f(z))f(z)) 2A f(x) Sf((() + const (18)
= -IAA -FFFFF 112
According to the Eckart- Sckart-Young Theorem [12], the optimal solutions F*, F satisfy
(F*) U
where Σ Rx is diagonal matrix with the k-largest eigenvalues of À and U NR contains the corresponding eigenvectors of the k-largest eigenvalues. When the regularizer Lee is minimized, F* satisfy (F*) F* = In the next step, we prove the uniqueness of the optimal solution.
We denote H = F*F(FF)TT... As (F=) F T, we obtain HHT = -'S If f, are a pair of eigenvector and eigenvalue of HH we have
HH Ç F* (F))
(19)
S*(S")
= - ((F')
So the eigenvalues of H the eigenvalues of (***)) As the positive eigenvalues of H are uniquely determined and S* has a descending order, S* is also determined and S* =
We that H 5 (S*) columns of F* are the ei eigenvectors of H and the corresponding eigenvalues 01 ... Ok. As H only has & different non-negative eigenvalues 01, ... .k the eigenspace of each eigenvalue one-dimensional. When consider the real number space, any two eigenvectors G, C of the same eigenvalue o; satisfy C cct. As (F*) F I, we obtain +1. As f(x) obtain
i.e., HH F ~ S* (S*) which means that the
vD. Fu,
17(z) + VDzx (U*), S = diag(o).......
(20)
A.2 Proof of Theorem 5.1
Proof. The following analysis mainly follows the proofs in [37].
We denote dz as the æ-th row of D. And we denote Fm. as the matrix composed encoder features on selected dimenstions, i.e. (Fm)a = (d (x). Recall that Ag,st = A(x, x+) =
Ez-p, A(xx|=)A(x+(a)) and is the normalized form of A, i.e. Ae Ala.t vd.d,t
Then we
reformulate the downstream error,
Eg,ylly Wy (m)) (x)1 -E de lyve W, W,((m)()))1
(0.Nz)
=1ID2/2y - 11
=1IDi/2y -AC + AC - F.. 11
where Cz,₁ (d.) Then consider the relationship between the downstream enror and the augmentation graph, we element-wise consider the matrix (D1/2y- AC),
A
(DI/2))=3 (do) (AC)a
Σ
Vdn vdst / (dot) [
(21)
13

---

So when j = Ve.
A0)es (do) Σ x(x,x*)
L 1ot
Vdz
A(x;
-V
vdl
A(I,It)
A(x, x
(22)
dd
Vd
A(E, x+)
1 bstts
dd
A(I, x+))
Σ
ddz
Ual #iee
z
When * ya
A(x, x+
AAD2a = [(d=)) v==1
Nat=s
ddz
A(x,
=0
1 1,+=j
(23)
ddz
A(x, xt"
dz
N, m3-
x
We define 3r = E A(E V,+ tv., and we have
A(x, x+)
A(x,
Net #U F2 2
11
Vd
ddz
J7 a
A(Z,
A(Z,x*)
Net A2=)²
)2
+
=1) )
dds
Vdz
j&un x-
A(x,It)
A(x,
)22 +
Σ 1u,+=
(24)
ddz
Net $Vo!
de
A(x, x+)
A(I,I**)
Vdn
Het
#U
)2
+
Vit
Lug )2
2,2
dnz
With that, we obtain | - AC = Σ
28
As we assume that ExuP, (A = a, so de
11
A(z,2t) (oeetlyy-tt 1y
Σ (((o)))))
(T) Lu
2,z
< Σ ((d (F) + 1911
(25)
2,x- 2E (A
2a.
14

---

Then we have
2
|D -CI1
Σ
dz
x
2Bx
(dz - 2 Σ A(x, xt Be
2 Σ A(x, 2 11)
(definition of 3₂)
$ 4a.
(Equation (25))
Then obtain
Eg,glly - W, ( (x)]1
-11D1/2- - AC + - Fm W
$ 22I|AC - 1 8a
(I BI 2||4|] + 12
(A FT +UUT)C W/ 111 + 8a
2 Fm F. Pm P.
11) + 8a
x 4(I((A - F.. 11
|F m C W,)1l) + 80x
(I - B1I² S 2(1 + +BI1)))
< 4(I((A - F.. 11 + |m | -) 8a
(B)B S ||
$ 4(1((A - FT. )| 8a:
(I| 1).
(26)
In the next step, we analyze the prediction error. We denote y as the ground-truth label of original data z. We first define an ensembled linear predictor z/y- For an original sample, the predictor ensembles the results of all different views and chooses the label predicted the most. With the definition, y # (Z) only happens when more than half of the views predict wrong labels. So
Pr(j + (z())) S 2 Pr(i # ps()))
W, I 41E (e), ( (mm)
< 8(En, Wm (1)
Eg P x1(3)
< 8(E.. - (x)(x)) + 2x.)
- (2 - F + 64a + 16a.
Then we plug the optimal solutions of SCL (Lemma 4.1) in to Fm and we obtain
N
{((F(())) 32((1-
of) + 80a.
(27)
i=++1
When we plug the optimal solutions of triCL (Theorem 4.5) into F., we obtain:
N
E(()(()) S 32
Σ
of + 80a.
(28)
i=m+1
A.3 Feature Identifiability of Asymmetric Tri-Factor Contrastive Learning
We first extend the augmentation graph to an asymmetric form. The asymmetric augmentation graph defined over the set of all samples with its adjacent matrix denoted by Po. In the augmentation graph, each node corresponds to sample, and the weight of the edge connecting two nodes vA and on is equal the probability that they are selected positive pair, i.e.(PP))...zz Po 2b)- And we denote Po the normalized adjacent matrix of the augmentation graph, i.e., (Po)....0 2s)) PA(e_)Pe(Rs)
15

---

Similar to the symmetric form, we then rewrite Lii as a matrix decomposition objective
BulfA.fB..5 = -2E on (f + +B, in (a(==))
Pol xx)²
+ (A(xa) (na))(( (fa(za) (Ja(ta)"ST(((z)) Pa (I (A
Ra.2%
2Po a Ibb (a Sfar))) + const
= =IIPo - FASFB 112
According to the Eckart- Young Theorem [12], the optimal solutions F7, S*, Fn satisty
(F - (
where 2 E x is a diagonal matrix with the k-largest cigenvalues Po and U Na contains the corresponding eigenvectors of the k-largest eigenvalues. When the regularizer Dee is minimized, F and 1 satisfy (FT) F7 = (F)) Fn = In the next step, we prove the uniqueness of the optimal solution.
We denote H F. and we obtain HHT (T If S, a are pair of eigenvector and eigenvalue HH we have
HH = (B)
(= S* (( = ((F))()))
(29)
So the eigenvalues of H are the eigenvalues of 5* T* As the positive eigenvalues of HHT uniquely determined and 5* has increasing order, S is also determined and S* =
We note that HH = (S* columns of FĀ are the cigenvectors of HH and the corresponding eigenvalues are 01 -.. As HHT only has & different non-negative eigenvalues 1, ... ke the eigenspace of each eigenvalue one-dimensional. When consider the real number space, any two eigenvectors C,, C of the same eigenvalue σ; satisfy S, cC,, As (FA) FĀ I, we obtain +1. Then eliminate the ambiguity of the sign following Eq 11 and A is unique. Similarly, FB is also unique. So the optimal solution of LuiCLIP is unique.
i.e., HH F (S*) which means that the k
B Experimental Details
B.1 Experiment Details of Section 6.1
We first generate a random matrix with size 5000 x 3000, and make sure that does not contain multiple eigenvectors (which is easy satisfy). For the matrix factorization problem ||4 FG' 1|. we apply off-the-shelf algorithms and repeat this process ten times. We then calculate the mean and variance of the l2 pairwise distance between the obtained solutions of F. For the trifactorization objective ||A- FSG I|?. we use SVD to obtain an initial solution and apply the sign identification procedure determine the sign of each eigenvector. Similarly, also repeat this process times and calculate the mean and variance of the la pairwise distance between different solutions.
B.2 Experiment Details of Section 6.2
Pretraining Setups. For different evaluation tasks (k-NN, linear evaluation, image retrieval), use the same pretrained models. We adopt ResNet-18 18 as the backbone. For CIFAR-10 10 and CIFAR- 100, the projector is two-layer MLP with hidden dimension 2048 and output dimension 256. And for ImageNet-100 100, the projector is two-layer MLP with hidden dimension 4096 and output dimension models with batch size 256 and weight decay 0.0001. For CIFAR-10 10 and
512. We pretrain CIFAR-100, we pretrain the models for 200 epochs. While for ImageNet- 100, we pretrain the models for 400 epochs. We use the cosine anneal learning rate scheduler and set the initial learning rate to 0.4 on CIFAR-10, CIFAR- 100, and 0.3 on ImageNet-100.
16

---

As the importance matrix is learned on the projection layer, we conduct the downstream tasks on the features encoded by the complete networks (containing both the backbones and the projectors).
The Distribution of the Importance Matrix. When observing the distribution of feature importance discovered by the importance matrix S, first apply the softplus activation functions the diagonal values of S and sort different rows of S by the descending order of corresponding diagonal values in S. We denote the non-negative ordered diagonal values of S as ($1.-.., sk). When we present the distribution of them in Figure 2(a), we normalize the diagonal values and obtain
k
(81/
8,
The K-NN Accuracy on Selected Dimensions. For k-NN evaluation on 10 selected dimensions, we do not finetune the models. We sort the dimensions of (a) by the descending order of corresponding diagonal values in the importance matrix. The k-NN is conducted the standard split of CIFAR-10, CIFAR- 100 and ImageNet- and the predicted label of samples is decided by the 10 nearest neighbors.
Linear Evaluation on Selected Dimensions. We train the linear classifier on 20 dimensions of the frozen networks for 30 epochs during the linear evaluation. We batch size to 256 and weight decay to 0.0001. For triCL., we sort the dimensions by descending order of the importance matrix. And for SCL, randomly choose 20 dimensions.
C More Extensions of Tri-factor Contrastive Learning
In this section, we apply tri-factor contrastive learning to another representative contrastive learning objective: the con-contrastive loss [15, 5].
Besides contrastive learning, non-contrastive learning is another popular self-supervised framework that throws the negative samples in contrastive learning and learns the meaningful representations only by aligning the positive pairs. Taking the state-of-the-ar algorithm BYOL [15) as an example, they use MSE loss:
g(x) f(xt) [g(())l2 [lf(z))
CMSE(f.g 2-
(30)
where (xx) and f() are two different networks avoid the feature collapse. Then we consider adapting the tri-term loss to the non-contrastive learning, i.e.,
T Sf(x*) (1)()) 1 [If(x ([.)(())(=)) -1| I1|2
Cuass(f..g 9) 2- E Em..rt
(31)
It is noticed that BYOL utilizes the stop- grodggra technique on the target network f and it is updated by exponential moving average. So we only calculate the feature decorrelation loss on the online network g.
17


Reviews:
[{"review_id": "cG06pc3ccV", "reviewer": "Reviewer_eV59", "summary": "The paper proposes a new contrastive learning objective that learns an additional diagonal matrix which forces the features to exhibit different importance. Theoretical results are provided to show that the new algorithm achieves similar downstream linear probe accuracy but also better interpretability / identifiability. Empirically, the authors test the new objective on several standard benchmark datasets and show that learned features exhibit better k-nn accuracy when a subset of dimensions are used, and the downstream linear probe accuracy can also be improved in certain cases. ", "questions": "Is there a easy way to project the full feature from SCL or SimCLR to a 20-dimensional subspace and still match the same k-nn performance of the top 20 dimensions from tri-SCL? I wonder whether doing some PCA can achieve that.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear and the paper is written well.\n- It's nice that the learned representations can have a clear subset of dimensions that captures most of the semantic information. This can be especially useful in many retrieval use cases where the dimensionality is the key bottleneck for scaling up the size of data. With this modified representation, people can easily select a subset of dimensions to construct less informative but more computational efficient representations for downstream use cases.\n- The proposed method is justified with corresponding theory.", "weaknesses": "- Only small datasets are used. Could include larger ones like ImageNet.\n- Theorem 5.1 seems to be exactly following that in [HaoChen et al], so I'm not sure there's enough novelty in this specific theorem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer eV59 for appreciating our work on its theoretical and empirical insights. We will address the main weaknesses and questions you mentioned.\n\nQ1: More experiments on the large-scale dataset ImageNet.\n\nA1: We conduct additional experiments on the large-scale dataset ImageNet. To be specific, we pretrain ResNet-18 with SimCLR and tri-SimCLR for 100 epochs on ImageNet. Following the settings of the downstream tasks on ImageNet-100 in Section 5.2, we examine whether tri-SimCLR can obtain ordered representations on ImageNet. \n\n*Table A: The comparison between SimCLR and Tri-SimCLR in importance-relevant tasks on ImageNet.*\n\n|  | SimCLR (Random 40 dimensions) | Tri-SimCLR (40 dimensions with largest S values) |\n| --- | --- | --- |\n| Linear Accuracy | 28.9 | **34.1(+5.2)** |\n| Retrieval mAP | 15.4 | **22.7(+7.3)**|\n\nAs shown in the above table, in both linear evaluations and image retrieval with selected 40 dimensions, tri-SimCLR accurately orders the feature importance and achieves significant improvements compared to the original contrastive learning methods. The empirical results on ImageNet further verify that tri-contrastive learning enhances the interpretability of the learned representations on large-scale datasets. And we will add the results to the paper in future updates.\n\n---\n\nQ2: Theorem 5.1 seems to be exactly following that in [HaoChen et al], so I'm not sure there's enough novelty in this specific theorem.\n\nA2: We note that the main advantage of the proposed triCL is that we establish the feature identifiability and enhance interpretability that does not hold for canonical CL features. Meanwhile, for feature importance irrelevant downstream tasks, we prove in Theorem 5.1 that it has the same generalization guarantees as canonical CL following Haochen et al. It is worth mentioning that this theorem mainly serves as a sanity check that ensures its generalization ability, which is not the key contribution of this work. In more details, although following the same vene, our results generalize Haochen et al by showing that the conventional 2-factor contrastive loss is not the only choice, and a tri-contrastive loss can also attain an equal generalization bound. Thus, we believe that the theorem is a necessary part of the paper, but not the key contribution. \n\n---\n\nQ3: Is there an easy way to project the full feature from SCL or SimCLR to a 20-dimensional subspace and still match the same downstream performance of the top 20 dimensions from tri-SCL (e.g., doing some PCA)?\n\nA3: Indeed, it is an interesting idea to reduce the dimensions of pretrained features with PCA. To compare it with triCL, we conduct additional experiments by applying PCA to reduce the dimensions from 256 to 20. Then we conduct linear evaluations on the selected dimensions.\n\n*Table B: The linear accuracy of selected dimensions of triCL, SCL+PCA, and SCL on CIFAR-10.*\n\n|  | triCL | SCL (PCA features) | SCL (random) |\n| --- | --- | --- | --- |\n| Linear Accuracy  | **82.0** | 61.1 | 69.0 |\n\n As shown in the above table, we observe that the features selected by the importance matrix show much better linear performance (82.0 vs 61.1) than the features generated by PCA.  And we observe that PCA even performs worse than random selections, probably because the representations learned by SCL are nonlinear and PCA would fail to find the important dimensions. So we suspect that simple dimension reduction methods like PCA cannot find the most important pretrained features accurately, which further shows the advantages of triCL.\n\n---\n\nHope our elaborations and new results above could address your concerns. Please let us know if there is more to clarify."}, {"review_id": "rIUKtlVHRa", "reviewer": "Reviewer_tkty", "summary": "Traditional CL calculates the similarities between two samples while this paper proposes to insert a learnable diagonal matrix between two feature vectors multiplication and name it triCL. The authors claim and show the proposed method leads to the exact interpretability of learned features. Some theoretical analysis is provided.", "questions": "see my comments above.", "limitations": "no discussion on limitations is provided, and i suggest the authors to add this part", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and the proposed idea is simple and may be useful for explaining the CL.", "weaknesses": "1. Lack of novelty: The proposed method is essentially a bilinear function, and it can be rewritten the same as conventional CL, i.e., $f_1^TSf_2 = (\\tilde{S}f_1)^T(\\tilde{S}f_2)$ where $S=\\tilde{S}^T\\tilde{S}$ since S is a nonnegative diagonal matrix, i.e., a kernel matrix. From this point, the new parameter $S$ can be integrated into the neural network training, without explicit consideration. $S$ may be useful for the interpretation of CL, but at this point, I do not see how different using $S$ is from using traditional features with some statistical pro-processing such as norm. At least, I do not see the necessity of the proposed method in terms of interpretability.\n\n2. Weak experiments: From Table 1, the improvement is marginal and even worse than the baseline. This is probably because introducing bilinear forms makes the training easier stuck in local minima. Also, these results verify my thoughts about bilinear form in learning above. I suggest the authors to investigate more in training, e.g., plotting training curves and doing some comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer tkty for appreciating our work on the interpretability of contrastive learning. We understand your main concerns are on whether we can integrate the importance matrix into the neural network and rewrite tri-contrastive learning the same as original contrastive learning. We will elaborate on this part in detail and address the weaknesses you mentioned.\n\n---\n\nQ1: The proposed method can be rewritten the same as conventional CL, i.e., $f_1^\\top Sf_2 = (\\tilde{S}f_1)^\\top(\\tilde{S}f_2)$, where $S = \\tilde{S}^\\top S$ since S is a nonnegative diagonal matrix, i.e., a kernel matrix. And the new parameter can be integrated into the neural network training. \n\nA1: Indeed, the importance matrix can be integrated into the 2-branch architecture, but we note that **the triCL objective is essentially different from canonical 2-branch CL objectives and cannot be reduced**.\n\nTo see this, remind that the tri-contrastive objective consists of two losses, $L_{tri}(f, S) = 2E_{x,x^+}f(x)^\\top S f(x^+) +E_{x}E_{x^-}\\left(f(x)^\\top S f(x^-)\\right)^2$ and $L_{dec}(f) = \\left\\Vert E_{x}f(x)f(x)^\\top-I\\right\\Vert^2$. By integrating the importance matrix into the neural network ($f' = \\tilde{S}f$), the first term ($L_{tri}$) can be rewritten as the original contrastive learning, and the second term ($L_{dec}$) becomes $L_{dec}(f') = \\left\\Vert E_{x}f'(x)f'(x)^\\top-I\\right\\Vert^2 = \\left\\Vert E_{x}(\\tilde{S}f(x))(\\tilde{S}f(x))^\\top-I\\right\\Vert^2$. In this situation,  the optimal solutions $f^\\star, S^\\star$ are not unique and the feature identifiability will disappear.\n\nTo verify it, we use the following counter-example. We denote $f'^\\star=\\tilde{S}^\\star f^\\star$ as the optimal solutions of $L_{dec}(f')$ and $R$ as an **arbitrary** invertible matrix. Then $f'^\\star = \\tilde{S}^\\star R R^{-1} f^\\star$ are also optimal solutions (the learned encoder is $R^{-1}f^\\star$ and the learned importance matrix is $\\tilde{S}^\\star R$ ) . In this case, it is obvious that the values in the importance matrix will be easily perturbed by the variable $R$ , which means the importance matrix can not indicate the feature importance anymore.  Therefore, we cannot reduce triCL to a canonical CL objective using a single output feature for each branch.\n\nThe use of the name triCL is because this idea originates from and has a close connection to the matrix tri-factorization method with a 3-factor objective $\\Vert A-USV\\Vert^2$ such that $U,V$ are orthogonal matrices [a], and it is clear that this problem cannot be reduced to a 2-factor problem.\n\n[a] Orthogonal nonnegative matrix tri-factorizations for clustering. In *ACM SIGKDD* 2006.\n\n---\n\nQ2: From Table 1, the improvement is marginal and even worse than the baselines. And there needs more experiments to investigate the training process.\n\nA2: It is crucial to highlight that **the main advantages of our proposed methods are that triCL 1) enhances the identifiability and interpretability of features (improving performance on feature importance relevant tasks), and 2) keeps comparable performance on feature importance irrelevant tasks,** compared with conventional contrastive learning. \n\nFor **feature importance relevant tasks**, the experiments in Section 6.2 show that triCL achieves significantly superior k-NN accuracy, linear accuracy, and image retrieval precision on selected dimensions, which verifies that the importance matrix $S$ can automatically find the most important features.\n\nFor **feature importance irrelevant tasks**, the experiments in Section 6.3 show that triCL can achieve comparable performance as the original contrastive learning. As Theorem 5.1 shows that triCL has an equal generalization bound to the 2-factor spectral contrastive loss [a], the comparable performance is expected and consistent with the theoretical analysis.  \n\nBesides, following your suggestions, we conduct additional experiments to compare the training process of triCL and the original contrastive learning. To be specific, we compare the training loss of SimCLR and tri-SimCLR on ImageNet-100. **As shown in Figure A in rebuttal pdf, we observe that the two objectives have similar training curves and converge at a similar pace.** And there is no evidence that triCL makes the training easier stuck in local minima.\n\n[a] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.\n\n---\n\nQ3: Limitations.\n\nA3:  The primary advantages of our proposed tri-contrastive learning approach are the enhanced feature identifiability and interpretability with comparable performance in feature importance irrelevant tasks. This method proves to be particularly beneficial for tasks that require identifiable and explainable features, such as image retrieval tasks. However, when the downstream tasks are irrelevant to the interpretability of learned representations (e.g., linear probing and finetuning), triCL only achieves comparable performance as conventional contrastive learning. In the future, we will explore more tasks that need feature interpretability and expand the scope of the application of triCL."}, {"review_id": "HJhXWUjIno", "reviewer": "Reviewer_Zqsm", "summary": "The paper proposes a new contrastive triCL method, which extends the existing pair-wise contrastive framework. The core motivation is the inability of CL methods to deliver identifiable representations. To tackle this, the paper proposes factorizing the pair-wise similarity measure x^T@y into a bilinear x^T@S@y form, where learnable matrix S is a diagonal feature importance matrix; and theoretically proves the identifiability property. The paper also demonstrates possible automated feature importance discovery by analyzing the learned feature importance matrix.", "questions": "1. Why is the softmax in S? Wouldn't the softmax limit the flexibility of each feature dimension, when computing x^T@S@y similarity? Have authors tried other normalization approaches, e.g. sigmoid for each diagonal element of S?\n\n2. Does identifiability of representation always boils down to reproducibility within the scope of this work?", "limitations": "I suggest adding a paragraph on limitations, which would highlight the scope of application of the proposed approach (or is the proposed method expected to work better in all circumstances?)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important problem of identifiability, which is often overlooked in contrastive learning.\n\n2. The proposed method is elegant in its simplicity. Also, due to its simplicity, it does not introduce any major computational overheads.\n\n3. Feature importance discovery can be an important component to speed up the inference (by excluding features of low importance)", "weaknesses": "1. In my opinion, the term tri-contrastive (triCL) does incorrectly reflect the essence of the method. The proposed method still is based on pair-wise contrast in a nutshell, although the contrast is in the factorized form. \n\n2. It seems that BarlowTwins tackles almost the same problem as the paper states in L33-34. On the technical side, the L_dec term Eq (8) seems to be a special case of the Barlow twins' objective. In that sense, the L_dec term in triCL can be considered not as just a regularization, but as a main source of the supervisory signal.\n\n3. The ablation is needed to verify the importance of the L_dec term in the triCL objective. I would be interested to see the results for L_dec, L_tri, and L_dec + L_tri objectives separately.\n\n4. Experimental evaluation is limited to relatively small-scale datasets and only two baseline contrastive learning methods.\n\nUPD: The rebuttal addressed most of my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Zqsm for appreciating our work on its importance and simplicity. We will address the weaknesses and questions you mentioned.\n\n---\n\nQ1： The term tri-contrastive does incorrectly reflect the essence of the method. \n\nA1:  Here, the term “tri-contrastive” does not imply that we change the pair-wise contrast to three encoders. Instead, it refers to the proposed importance matrix $S$. We note that **our method is deeply connected to the matrix tri-factorization approach** [a] that transforms the matrix decomposition objective from $\\Vert A - UV \\Vert ^2$ to $\\Vert A - USV \\Vert^2$. Therefore, the name triCL indicates the technique we use here. Alternatively, the name “Contrastive Learning with Importance Discovery (CLID)” could reveal the main focus of this method more clearly. Please let us know if you have better options.\n\n[a] Orthogonal nonnegative matrix tri-factorizations for clustering. KDD 2006.\n\n---\n\nQ2: It seems that BarlowTwins tackles almost the same problem. \n\nA2: Although the BarlowTwins loss and the $L_{dec}$ term are similar, we note that **the triCL objective consists of two terms: $L_{tri}$ and $L_{dec}$**. It is important to highlight that we **cannot achieve feature identifiability only with $L_{dec}$**  (and thus Barlow Twins). \n\nSpecifically, we show a counter-example. If we only consider $L_{dec}$, i.e., $\\Vert \\mathbb{E}_{x}f(x)f(x)^\\top-I\\Vert^2,$ and denote $f^\\star$ as an optimal solution of it, $R$ as an **arbitrary** orthogonal matrix, then we note that $f^\\star R$  is also an optimal solution. \n\nConsequently, **only introducing $L_{dec}$**  **cannot enable the feature identifiability and the existence of $L_{tri}$ term makes triCL and BarlowTwins essentially different.**\n\n---\n\nQ3: The ablation on the $L_{dec}$  term in triCL.\n\nA3: We conduct additional experiments to verify the importance of both the $L_{tri}$ and $L_{dec}$ terms.  \n\n*Table A: The linear accuracy of selected features on CIFAR-10.*\n\n|  | $L_{tri}$  (20 dimensions with largest S values) | $L_{dec}$  (Random 20 dimensions) | $L_{dec}$ + $L_{tri}$ (20 dimensions with largest S values) |\n| --- | --- | --- | --- |\n| Linear Accuracy  | 76.5 | 70.5 | **82.0** |\n\nAs shown in the above table, both the $L_{dec}$ and $L_{tri}$ are important in importance-relevant tasks. \n\n---\n\nQ4: Experimental evaluation on other datasets and methods.\n\nA4: Following your suggestions, we conduct experiments on ImageNet and apply triCL to another baseline BarlowTwins. \n\n*Table B: The comparison in feature importance relevant tasks on ImageNet.*\n\n|  | SimCLR (Random 40 dimensions) | Tri-SimCLR (40 dimensions selected by S) |\n| --- | --- | --- |\n| Linear Accuracy | 28.9 | **34.1** |\n| Retrieval mAP | 15.4 | **22.7** |\n\n*Table C: The linear accuracy of selected features on CIFAR-10.*\n\n|  | BarlowTwins (Random 50 dimensions) | Tri-BarlowTwins (50 dimensions selected by S) |\n| --- | --- | --- |\n| Linear Accuracy | 55.9 | **63.2** |\n\nAs shown in the above tables,  triCL achieves significant improvements in the feature importance relevant tasks, which further verifies that triCL can be applied in different datasets and frameworks. \n\n---\n\nQ5: Why is the softmax in S? Wouldn't it limit the flexibility? other approaches?\n\nA5: Here, Softmax normalizes the values in $S$ to avoid the dramatic change of loss values for numerical stability. Meanwhile, it also implements a global receptive field over all importance values. Like the L2 normalization adopted in SimCLR, softmax sacrifices a bit of flexibility, but from the table below, we observe that it indeed performs the best among other choices (such as Sigmoid and ReLU). \n\n*Table D: The linear accuracy of selected features on CIFAR-10.*\n\n|  | SCL  (Random 20 dimensions) | Sigmoid  (20 dimensions with largest S values) | ReLU (20 dimensions with largest S values) | Softmax (20 dimensions with largest S values) |\n| --- | --- | --- | --- | --- |\n| Linear Accuracy  | 69.0 | 81.1 | 78.4 | **82.0** |\n\n---\n\nQ6: Does identifiability of representation always boils down to reproducibility within the scope of this work?\n\nA6: In fact, identifiability brings various advantages besides the reproducibility, like interpretability, robustness, and superiority in casual inference [b]. Specifically, we demonstrate two advantages of triCL.\n\n**Feature interpretability and disentanglement.**  Theorem 4.5 shows that with identifiability, the diagonal values of $S$ are the singular values of the augmentation graph, i.e., they are interpretable. Besides, as the identifiability eliminates the freedom of linear transformations in original CL, the features are decoupled.  For example, Figure 1(a) indicates that each dimension in triCL only captures the semantics of one class.\n\n**Better Feature Transferability.** As the identifiability can decouple causal features and non-causal features, the transferability may be improved when facing the distribution shift [c].  Empirically, the following table shows that tri-SimCLR achieves improvements in the transfer learning task. \n\n*Table E: The transfer accuracy from ImageNet to CIFAR-10.*\n\n|  | SimCLR | Tri-SimCLR |\n| --- | --- | --- |\n| Linear Accuracy | 45.6 | **46.5(+0.9)** |\n\n[b] Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose. In Environmental Modelling & Software, 2019.\n\n[c] Toward causal representation learning. In Proceedings of the IEEE. 2021.\n\n---\n\nQ7: Limitations.\n\nA7: The primary advantages of our proposed tri-contrastive learning approach are enhanced feature identifiability and interpretability. This method proves to be particularly beneficial for tasks that require identifiable and explainable features, such as image retrieval tasks. However, for importance-relevant tasks (e.g., linear probing and finetuning), triCL only achieves comparable performance as conventional contrastive learning."}, {"review_id": "INTF1kcZxz", "reviewer": "Reviewer_aafx", "summary": "In this paper, the authors proposed a new contrastive learning scheme (tri-contrastive learning) with a learnable diagonal matrix, which can capture the feature importance. To show the significance of the proposed method, the authors conducted both theoretical and experimental analysis, where some interesting findings are observed. However, it is not clear how the proposed method advances existing contrastive learning-based methods, and experimental results are not very convincing. The detailed comments are below. ", "questions": "Please see above. ", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea is simple and interesting, with reasonable theoretical analysis based on eigenspace. \n\n", "weaknesses": "1. The idea of analyzing feature importance for contrastive learning is not new [a]. While the proposed method is different from [a], the submission lacks analysis how the proposed method outperforms [a], especially based on the generalization capability the submission focused on.\n\n[a] Accurate and Robust Feature Importance Estimation under Distribution Shifts, aaai21\n\n2. The visualization example in fig 1 is not clear. Noted that the contrastive learning is performing on the feature space, it is not clear to me based on the paper how the feature space linked to the image space. \n\n3. The generalization bound is not very clear, the upper bound depends on the term \\alpha, which relies on data augmentation. How different augmentations inclunce the term \\alpha. In what condition that the sample and its augmented view could have different labels?\n\n4. The improvement on three benchmark datasets are very marginal (almost no improvement), and the results can’t justify the generalization capability claimed in this paper, convergence analysis is also not presented. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer aafx for appreciating our work on the novelty and simplicity. We will address the main weaknesses in the following.\n\n---\n\nQ1: The comparison between [a] and tri-contrastive learning.\n\nA1: Following your suggestions. we compare ProFILE proposed in [a] with triCL from different aspects. First, they are quite different in methodology:\n\n- **Different levels of feature importance.** We note that PRoFILE proposed by [a] estimates the importance of different dimensions of **input samples** while tri-contrastive learning obtains the importance order of **encoded features.**\n- **Different theoretical guarantees.** We note that PRoFILE designs an empirical objective to estimate the importance of different dimensions, and they do not theoretically analyze the generalization performance. In contrast, triCL establish the theoretical bounds in Section 5.1 and 5.2.\n\nEmpirically, we compare the generalization performance between triCL and PRoFILE.\n\n*Table A: Comparison between SCL, PRoFILE, and triCL in linear evaluation with selected dimensions on CIFAR-10.*    \n\n|  | SCL (Random 20 dimensions) | PRoFILE (20 dimensions with largest estimated importance) | triCL (20 dimensions with largest S values) |\n| --- | --- | --- | --- |\n| Linear Accuracy  | 69.0 | 74.3 | **82.0** |\n\nAs shown in the above Table, although PRoFILE shows better performance than the random selections, triCL still achieves significant improvements over PRoFILE (82.0 vs 74.3). In summary, triCL outperforms PRoFIle both theoretically and empirically.\n\n[a] Accurate and Robust Feature Importance Estimation under Distribution Shifts, AAAI21.\n\n---\n\nQ2: The visualization example in Fig 1 is not clear. Noted that the contrastive learning is performing on the feature space, it is not clear based on the paper how the feature space linked to the image space.\n\nA2: In Figure 1, we visualize the features learned by triCL by presenting them with the corresponding inputs before being encoded. We add Figure B in rebuttal pdf to better explain the visualization process. In practice, we first **encode the images** by the encoders pretrained with tri-contrastive loss (with importance matrix $S$) to get their features.  Then we respectively select the 10 most and least important dimensions in the feature according to the importance matrix $S$ of the encoder. We present **each selected dimension by each row filled by images that have the largest activation on this dimension**. As shown in Figure 1, the images whose features are activated in the dimensions associated with the highest values in the importance matrix consistently belong to the same class, while the samples activated in the dimensions with the lowest values in S appear to be mostly independent, which further confirms that the importance matrix effectively reflects the significance of different feature dimensions.\n\n---\n\nQ3: In what condition that the sample and its augmented view could have different labels? How different augmentations influence the term \\alpha? \n\nA3: **When the augmentation is too strong, (e.g., the scale of the crop is too large), the augmented views may lose the semantic information and the labels would be changed.** For example, as shown in Figure C in rebuttal pdf, for an image that contains a car and a person, when the augmentation strength is aggressive, the augmented view may lose the center objective (car) and focus on the person. In this case, it is obvious that the augmented views of the same images would have different labels. And in the extreme case, the augmented views may only focus on the background messages and lose all the semantic information of objectives. Based on the analysis and the examples above, we know that the probability that two augmented views of the same image have different labels is mainly decided by the choices of data augmentation. To be specific, **when we use more aggressive augmentations, there will exist more augmented views that have different labels, which implies a larger labeling error $\\alpha$.** So the generalization bound indicates that the triCL needs to choose appropriate augmentation strength and avoid too-aggressive augmentations to obtain better generalization performance.\n\n---\n\nQ4: The improvements on three benchmark datasets are very marginal, and the results can’t justify the generalization capability, convergence analysis is also not presented.\n\nA4: There may be some misunderstanding here. In fact, by combing Theorem 5.1 and the generalization guarantee of the 2-factor contrastive loss in [b], what we try to prove theoretically is that triCL and the original spectral contrastive learning share the **equal** generalization bound. **So the comparable downstream performance in feature importance irrelevant tasks between two contrastive objectives is expected and exactly verifies the theoretical analysis.** Meanwhile, the main advantages of our proposed methods are the feature identifiability and interpretability.  For example, the experiments in Section 6.2 show that triCL achieves significantly superior k-NN accuracy, linear accuracy, and image retrieval precision on selected dimensions.\n\nBesides, we conduct additional experiments to compare the training process of triCL and the original contrastive learning. To be specific, we compare the training loss of SimCLR and tri-SimCLR on ImageNet-100. As shown in Figure A in rebuttal.pdf, we observe that the two objectives have similar training curves, which implies that these two objectives converge at a similar pace.\n\n[b] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021."}]

Author rebuttal:
The Rebuttal PDF can be seen in the attached file, which contains

- Figure A: comparison of the training loss between SimCLR and tri-SimCLR;
- Figure B: an additional explanation of Figure 1 in the paper;
- Figure C: a visualization example of how the augmentation strength influences the labeling error $\alpha$ in Theorem 5.1.




Meta-review: The paper proposes a new contrastive triCL method, which extends the existing pair-wise contrastive framework. The core motivation is the inability of CL methods to deliver identifiable representations. Theoretical results are provided to show that the new algorithm achieves similar downstream linear probe accuracy but also better interpretability/identifiability. Empirically, the authors test the new objective on several standard benchmark datasets and show that learned features exhibit better k-nn accuracy when a subset of dimensions is used, and the downstream linear probe accuracy can also be improved in certain cases.

This paper is well-written and makes good contributions. After author-reviewer discussions and reviewer-ac discussions, the paper still received mixed scores with three positives and one negative. The main concern of Reviewer aafx is the generalization capability of feature importance. Please remember to carefully explain this point (as you replied to the reviewer) in the camera-ready version.
Decision: Accept (poster)