Non-negative Contrastive Learning
Authors: Yifei Wang
Qi Zhang
Yaoyu Guo
Yisen Wang
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Learning
Representation Learning
Self-supervised Learning
Deep Learning
Disentanglement
Interpretability
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge, as these features are often opaque to human understanding. In this paper, we propose Non-negative Contrastive Learning (NCL), a renaissance of Non-negative Matrix Factorization (NMF) aimed at deriving interpretable features. The power of NCL lies in its enforcement of non-negativity constraints on features, reminiscent of NMF's capability to extract features that align closely with sample clusters. NCL not only aligns mathematically well with an NMF objective but also preserves NMF's interpretability attributes, resulting in a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCL. Empirically, we show that these advantages enable NCL to outperform CL significantly on feature disentanglement, feature selection, as well as downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning as well. Code is available at https://github.com/PKU-ML/non_neg.

Published as a conference paper at ICLR 2024
NON-NEGATIVE CONTRASTIVE LEARNING
Yifei Wang 1M
Qi Zhang? Yaoyu Guo!
Yisen Wang2.3t
School of Mathematical Sciences, Peking University
2 National Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology. Peking University
3 Institute for Artificial Intelligence, Peking University
ABSTRACT
Deep representations have shown promising performance when transferred downstream tasks in a black-box manner. Yet, their inherent lack of interpretabil- ity remains significant challenge, these features are often opaque to human derstanding. In this paper, we propose Non-negative Contrastive Learning (NCL), renaissance of Non-negative Matrix Factorization (NMF) aimed deriving in- terpretable features. The power of NCL lies its enforcement of non-negativity constraints features, reminiscent of NMF's capability extract features that align closely with sample clusters. NCL not only aligns mathematically well with NMF objective but also preserves NMF's interpretability attributes, resulting a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCI.. Empirically, we show that these advantages enable NCL. to outperform CL. significantly on feature disentanglement, feature selection, well downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning well. Code is available at https:://////.. /tte::///ttttbb com www.////////-///eee __e
AOWWMAA TIN RAp/
15
OFWAACY
(a) CL
(b) NCL
(c) CL
NCL
Figure 1: Feature visualization semantic consistency (a-b) and sparsity (c-d) CIFAR-10. The first two demonstrate top-activated samples along each feature dimension, where those of CL (a) often have distinct semantics along each dimension (column) (e.g.. dears and airplanes), while those of NCL (b) have much better semantic consistency, indicating better feature disentanglement. Com- paring (c) and (d), it is easy that NCL features enjoy much better sparsity than CL features with only a few activated dimensions (< 10%) per sample.
INTRODUCTION
It is widely believed that the success of deep learning lies in its ability to learn meaningful represen- tations (Bengio al. 2013). In recent years, contrastive learning (CL) has further initiated huge interest in self-supervised learning (SSL) of representations and attained promising performance in various downstream tasks (Chen al. 2020 Wang al., 2021; Guo et al., 2023 Luo al. 2023) Nevertheless, the learned representations still lack natural interpretability. As shown in Figure la top activated examples along each feature dimension (column) belong to quite different classes, so we can hardly inspect the meaning of each feature. Although various interpretability tools have been
"qqual Contribution. Yifei Wang has graduated from Peking University, and currently a postdoc at MIT. "ooresponndig Author: Yisen Wang (yisen, wang @ww..duu.un............

---

Published as a conference paper at ICLR 2024
body dassical
Non-negative Matrix
classical ML
Matrix Factorization
Factorization
Enhancing explainability
mathematical equivalance Dhangerak)
non non-interpretable
mathematical equivalence [this
interpretoble
non-sparse
sparse
non-disentongled
disentangled
This Work
Non-negative Contrastive Learning
modern ML
Contrastive Learning
Simple reparameterization
f-00= a_00=o.((0)
Figure 2: Relationship between different learning paradigms discussed this work.
developed (Selvaraju al. 2017). we are wondering whether there way to directly learn inter- pretable features while maintaining the simplicity, generality, and superiority advantages of deep representation learning.
In this paper, we develop Non-negative Contrastive Learning (NCL) an interpretable alternative to canonical CL. Through a simple reparameterization, NCL can remarkably enhance the feature interpretability, sparsity, orthogonality, and disentanglement. As illustrated in Figure lb each fea- ture derived from NCL corresponds to a cluster of samples with similar semantics, offering intuitive insights for human inspection. NCL draws inspiration from the venerable Non-negative Matrix Factorization (NMF) technique. Notably, by imposing non-neganee non-negativity on features, NMF known derive components that are more naturally interpretable than its unrestricted counterpart, Matrix Factorization (MF) (Lee & Seung] 1999). Guided by this merit, we develop NCL as a non-negative counterpart for CL. NCL is indeed mathematically equivalent the NMF objective, analogous to the equivalence between CL and MF discovered in prior work (HaoChen et al. 2021] Zhang et al., theoretically
2023a (refer Figure for visual summary). Built upon this connection, justify these desirable properties of NCL in comparison to CL. In particular, we establish provable guarantees on the identifiability and downstream generalization of NCL, showing that NCL, in the ideal case, can even attain Bayes-optimal error. In practice, NCL can bring clear benefits to various tasks (feature selection, feature disentanglement, and downstream classífication). At last, we show that NCL can be extended and yield better performance for supervised learning well with the proposed Non-negative Cross Entropy (NCE) loss. We summarize our contributions as follows:
We introduce Non-negative Contrastive Learning (NCL), an evolved adaptation of NMF, to address the interpretability challenges posed by CL. Our findings demonstrate NCL's superiority in sparsity, orthogonality, and feature disentanglement.
We provide a comprehensive thcoretical analysis of NCL, which not only characterizes the optimal representations and justifies these desirable properties, but also establishes identi- fiability and generalization guarantees for NCL.
We show how the advantages of NCL in feature interpretability be applied to improv- ing various downstream tasks. We also discuss how to extend NCL beyond SSL with demonstration of its benefit in supervised learning.
2 BACKGROUND ON CONTRASTIVE LEARNING
The task of representation learning is to learn an encoder function f Rdd - R&k that extracts low-dimensional data representations z IR* (a.k.a features) from inputs i Rd.. To perform contrastive learning, we first draw pair of positive samples (z, 2e.2) by randomly augmenting the same natural sample z P()), and we denote the augmentation distribution draw negative samples x as independently augmented samples, following the marginal distribution P(x) Ag ().. The goal of contrastive learning is to align the features of positive samples while pushing negative samples apart. A well-known CL objective is the InfoNCE loss (Oord et al. 2018).
A(- We also
LxCE()) = -E (27 )M log
exp(f(x)T f(x4)) ))+++))+) ΣM exp(f(x) f(x7)"
exp(F(x)TT(T))
2

---

Published as a conference paper at ICLR 2024
where { are M negative samples independently drawn from P(x). HaoChen al. (2021)
propose the spectral contrastive loss (spectral loss) that is more amenable for the theoretical analysis.
(Sp(s) = -2 f(x) [xx+) + R -. (f(z)) T ())
Besides CL., it is also used for other types of SSL. (Zhang et al. 2022 2023a).
Equivalence Matrix Factorization. Notably, HaoChen et al. (2021) show that the spectral con- trastive loss (Eq. 2 can be equivalently written as the following matrix factorization objective:
LM = || - 1 II where Fz,: = /P(x)f(a)
(3)
Here, À D-1/2 AD-1/2 1/2 denotes the normalized version of the co-occurrence matrix A E N of all augmented samples z E X (assuming [X| = N for ease of exposition); Vr, x' E X, A == P(x,x') [x A(r'(z) More discussions on related work can be found in Appendix
2.1 LIMITATIONS IN REPRESENTATION SYMMETRY
According to the seminal work of Bengio et al. 2012 a good representation should extract ex- planatory factors that are sparse, disentangled, and with semantic meanings. However, upon close observation, CL-derived features remain predominantly non-explanatory and fall short of these cri- teria. For example, Figure Ia shows that different samples (like planes and dogs) may have similar activation along each feature dimension (a sign of non-disentanglementt Further, we also visualize the values of the learned features of SimCLR (Chen et al., 2020) in Figure Icl where we can see that almost all CL features have non-zero values, indicating absence of sparsity.
Remarkably, an important cause of CL's deficiency in these aspects is its rotation symmetry. Notice that CL objectives invariant w.r.f. the rotation group (O()), which means that for an optimal solution f")x,, any rotation matrix R also gives an optimal solution (z) Consequently, CL cannot guarantee feature disentanglement along specific axis since non-permutative rotation can easily corrupt them. The following theorem shows that any (unconstrained) CL objective depending only on pairwise distance will have rolation symmetry in its optimal solution, covering most (if not all) CL objectives. such as InfoNCE (Oord et al. (2188). hinge loss (Saunshi et al. 2019, NT-Xent (Chen al. 2020), spectral contrastive loss (HaoChen et al.) 202I)) etc.
Theorem 1. As long as the unconstrained objective L only relies on pairwise Euclidean similarity (or distance), e.g., f(x) [xx),, its solution (x)) suffers from rotation symmetry.
Proof. Observe that V v, x' E X, (Rf*
= (
To attain feature disentanglement, we need to break this representation symmetry and learn repre- sentations aligned with the given axes.
3 NON-NEGATIVE CONTRASTIVE LEARNING
In classical machine learning, an effective way to make matrix factorization (MF) features inter- pretable is through non-negative matrix factorization (NMF), where non-negative matrix V 2 is factorized into two non-negative features V ss W H where W, H 2 0 Inspired by the equiva- lence between MF and CL (Eq. (.) ) we propose a modern variant for NMF (also with mathematical equivalence) and show that it also demonstrates good feature interpretability as NMF.
In the equivalent MF objective CL. critical observation that the normalized co-occurrence matrix A I (Eq. (3) non-negative symmetric matrix (all values in A are probability values). Therefore, we can apply symmetric NMF to A into nen-negative features F4:
[NMF(F)= = | - F,FII? where F+ 2 0.
(4)
Unfornaturely, Eq. 4 is intractable with canonical NMF algorithms since the input space X' is ex- ponentially large (N + 20). To get rid of this obscure, we reformalize the NMF problem (Eq. cquivalently as a tractable CL objective (Eq. (5)) that only requires sampling positive/negative sam- ples from the joint/marginal distribution. The following theorem guarantees their equivalence.
Throughout this paper, 2 denotes dementtwwise comparison.
3

---

Published as a conference paper at ICLR 2024
050
(a) Semantic Consistency
(b) Feature Sparsity
Feature Correlation
Comaprisons between contrastive learning (CL) and non-negative contrastive learning (NCL): a) class consistency rate, measuring the proportion of activated samples that belong their most frequent class along each feature dimension; b) feature sparsity, the average proportion of zero elements (|x² Le Le=5) in the features of each test sample; c) dimensional correlation matrix C of 20
Figure 3:
random features: v(i,j), Cip = (x)) (x), where f.(x)) = f((x)/ V Ex (f(x)))²
Theorem 2. The non-negative matrix factorizatin problem in Eg. 4 is equivalent to the following non-negative spectral contrastive loss when the row vector (F+) )z: (P(x))+(x)),
CXCL -21E (+) f+ (x4) Ex,r
(f+(z)f++xx))
(5)
such that (+ 0,
Non- nog-neeative Reparameterization. Comparing Eq. (5) to the original spectral contrastive loss in Eq. 2) we find that the objectives remain the same, while the only difference is that NCL enforces non-negativity on output features f-). We can realize this constraint via a reparameterization of neural network outputs, e.g., applying non-negative transformation a (-) (satisfying o (z) 2 0, Vx) at the end of standard neural network encoder / used CL,
(6)
Potential choices of 0+ include ReLU, sigmoid, softplus, We find that the simple ReLU func- tion, (eLU(x) = max(x, (r. 0), works well in practice. Its zero-out effect can induce good feature sparsity compared to other operators with strietly positive outputs. Although adopt the spectral contrastive loss, this reparameterization trick also applies to other CL objectives (such the InfoNCE loss in Eq. (1) and transforms them the corresponding non-negative ver- sions. We regard these variants generally as Non- non-nvgative Contrastive Learning (NCL). We also briefly discuss how to extend
discussions mainly
other learning paradigms (supervised, multi-modal) in Section
Resurrecting the Dead Neurons. We notice that although ReLU output features enjoy better sparisty, they may suffer from the dead neuron problem during training eLu al. 2019), In practice, some dimensions of NCL features (e.g., 86/512) are inactive on all samples, sacrificing the given model capacity. To avoid this problem while ensuring non-negativity, inspired by Gumbel softmax (Jang et al. 2016), we adopt simple (biased) reparameterization trick (in PyTorch style).
z F relu (z) d)tach F.gelu (z) F. gege (de
where 2, detach () eliminates the gradients of z. In this way, the forward output still equals F rulu (z) while the gradient is calculated w.r.f. GELU (Hendrycks & Gimpel 2016). which is close to ReLU but has gradients for negative inputs. This reparameterization trick works on CIFAR-10 and CIFAR-100 and improves the overall feature diversity, but is inferior to ReLU on ImageNet-100. More systematic study of this problem is left for future work.
3.1 BENEFITS OF NON-NEGATIVITY. CONSISTENCY, SPARSITY, AND ORTHOGONALITY
In canonical NMF, researchers found that non-negative constraints help discover interpretable parts of facial images (Lee & Seung 1999). It also has applications in many scenarios for discovering meaningful features (Wang & Zhang 2012). such as text mining, factor analysis, and speech de- noising. In our scenario, the non-negativity inherent to NCL offers clear advantages in deriving interpretable representations. Unlike canonical CL, non negativity ensures that different features will not cancel one another out when calculating feature similarity. As NCL drives dissimilar neg- ative samples to have minimal similarity (f+(x) T f+ (x') - 0), each sample forced to have zero activation features it does not possess, rendering NCL. features inherently sparse.
4

---

Published as a conference paper at ICLR 2024
Furthermore, NCL offers robust solution to the rotation symmetry challenge associated with stan- dard CL, as outlined in Theorem [
Any rotation applied to features, especially sparse ones, often
0 This disrupts their non-negative nature. To exemplify, consider the feature matrix 1 0 1
matrix cannot undergo any non- pommttmuteeevve rotation while preserving its non-negativity As a con- sequence, NCL features are enforeed to be aligned with the given axes. In particular, with non- negative constraints, activated samples along each feature dimension have high similarity and thus have similar semantics.
To verify the analysis above, we conduct pilot study on SimCLR and its NCL variant on benchmark datasets. First, we find that NCL features indeed have better semantic consistency than CL (Figure Bap. aligning with the examples in Figure [ Second, around 90% dimensions of NCL features are zero per sample, while CL features do not have any sparsity, as we observed in Figure 3b Third, Figure 3c shows the feature correlation matrices, and can observe that NCL features are also more orthogonal than CL features. Overall, NCL enjoys much better semantic consistency, feature sparsity, and feature orthogonality than CL.
4 THEORETICAL PROPERTIES OF NON-NEGATIVE CONTRASTIVE LEARNING
In Section 3 we have proposed non-negative contrastive learning (NCL) and shown that simple reparameterization can bring significantly better semantic consistency, sparsity, and orthogonality. In this section, provide comprehensive theoretical analyses of non-negative contrastive learning in terms of its optimal representations, feature identifiability, and downstream generalization.
4.1 ASSUMPTIONS
As in Saunshi al. (2019). assume that in the pretraining data, samples belong to latent classes C (.,,...< Cm with non-neglectable support (cc) > 0, Ve E C. We assume that positive samples generated by contrastive learning are drawn independently from the same latent class.
Assumption (Positive Generation). Va, x!
E X, X(,, (()c(())))())))))
2022). criticized this assumption be non- Remark. Some previous works, egg., Wang et al. realistic. But this is mainly because Saunshi et al. (2019) only consider the extreme case when the latent classes are the same as the observed labels, i.e., C= =y;; while in fact, positive samples generated by data augmentations are still dependent within the same observed class. In fact, when we define latent classes using more fine-grained categorization (e.g., a specific type of car), this assumption is still plausible. In the extreme case, each original sample J. E X can be seen as a latent class, which reduces to the general definition in HaoChen et al. (2021) with P(x, x') = E2 P(x()) (x()) Z). Thus, our framework includes the two previous frameworks as special cases.
As common property of natural data, we assume different latent classes only have small overlap.
Assumption (Latent Class Overlap). The maximal class overlap probability no more than E, i.e., maxx4j P(e,, cς) E, where C,c,,cy) P(((()))) denotes the overlap probability between C; and cj. If each sample only belongs to one latent class, we have E O.
The latent class information of an example x can be denoted as an m-dimensional vector (5(x) [P(c)]z)..... P(ca (z)]. This vector captures the subclass-level semantics and can be ideal representation of x. Next, we show that non-negative contrastive learning can recover this ground-truth feature from observed samples, up to trivial dimensional scaling and permutation.
4.2 OPTIMAL REPRESENTATIONS
The following theorem states that when the number of feature dimensions k: is as large as the number of latent classes m, the following solution ((x) is an optimal solution to the NCL objective:
((x) =
(P(m)) P(T (1)
P(
E V
(8)
P pm
5

---

Published as a conference paper at ICLR 2024
where [T1, Tm, is a random permutation of latent classes [c1,..... Cm Nicely, ((x² admits natural interpretability since contains the posterior distribution along each dimension.
Theorem 3. Under the latent class assumption (Assumption
7 and choosing = m, 26.))
minimizer of the NCL objective Eq. 5 i.e., E arg min LNCI.
Compared to CL with non-explanatory eigenvectors of as the closed-form solution HHooChen
ct al. 2021). NCL's optimal features o(z) are advantages in the following aspects. 1) Semantic
Consistency. For each E C, the activated samples with Φc(z) > 0 indicates that P(c]z) > 0, meaning that x indeed belongs to c to some extent. Thus, activated samples along each dimension will have similar semantics. Instead, CL's eigenvector features containing both positive and negative values do not have clear semantic interpretation. 2) Sparsity. As each sample belongs to one or a few subclasses (Assumption 2). only one or a few elements in Φ(x) are positive with P(e[x) > 0. Thus, ((x) is sparse. In comparison, eigenvectors used by CL are usually dense vectors. 3) Orthogonality. Since feature dimensions in ((x) indicate class ussignments, that different feature dimensions have very low correlation when the latent class overlap E is small (Assumption 2
1 P(TT))((T,) E, B,
Vi E.
min, P(c)
In the ideal case when evey sample belongs to single latent class, we can show that (zz) perfectly recovers the one-hot ground-trun -tuth factors with high sparsity and perfect orthogonality.
Theorem (Optimal representations under one-hot latent labels). sfech sample x only belongs to only one latent class с = p(z),, we have 6(z) = yy(a), where μ(x) indicates x e's belonging class. Meanwhile, we have [])(z))lo (highly sparse) and (()))(( (perfectly orthogonal).
4.3 FEATURE IDENTIFIARILITY
Despite the advantages of 6 discussed above, it remains unclear whether it is the unique solution NCL. As discussed in Section 2.1 rotation symmetry will break the desirable properties of ő(x). In general context, the uniqueness of NMF is an important research topic in itself (Laurberg et al., 2008 Huang al. 2014).
also widely studied in signal analysis and representation learning, known as (feature) adentifiability (Fu et al. 2018 Zhang et al. 2023b), concerning whether NMF can recover ground-truth latent factors (up to trivial transformation), as defined below.
Definition 1 (Feature Identifiability). We say that an algorithm produces identifiable (or unique) features if any two solutions f. 9 are equivalent up to permutation and dimensional scaling: there exists a permutation matrix P and a diagonal matrix D such that f(x) = DP X.
For NCL., we find that mild condition unique samples (Assumption 3) can guarantee NCL's identifiability and disentanglement. This assumption is plausible in practice since only requires one unique sample in each latent class. Intuitively, the one-hot encodings of unique samples prevent any non-permutative rotation under the non-negative constraint. Therefore, NCL's non-negativity indeed helps break the rotation symmetry of CL features.
Assumption (Existence of unique samples). For every latent class E C, there exists one sample I E X such that it belongs uniquely to c, i.e., P(cl=)=11
Theorem 5. Under Assumptions 7 & the solution oxx) (Eq. (8)) is the unique solution to the NCL objective Eq. (5) As result, NCL features are identifiable and disentangled.
4.4 DOWNSTREAM GENERALIZATION
In real-world applications, pretrained features are often transferred to facilitate various downstream tasks. Following the previous convention, we consider the linear classification task, where linear head g(z) W is applied to pretrained features = o(x) (Eq. (8). The linear head is learned
FFor the ease of exposure, we consider k m. If m, we can fill the rest of dimensions with all zeros, while if A < m, we can always tune to attain a lower loss. In practice, some feature dimensions are never activated, so we believe that generally we have & > m and omit inactive dimensions for simplicity. Besides, although the discrete latent class is adopted here, the discovered optimal solutions also apply to continuous latent variables, long as the position generation still follows Assumption

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of CL and NCL (ours) on ImageNet-100 in terms of three feature selection tasks: linear probing (measured by classification accuracy), image retrieval (measured by mean Average Precision @ 10 (mAP@ 10), and transfer learning (measured by classification accuracy).
Probing
Image Retrieval
Transfer Leaming
Selection
CL.
NCL.
CL.
NCT.
CL
NCL.
(2048)
66.8 + 0.2
68.9 + 0.1
10.9++0.2
14.2 0.2
17,2 0.1
19.9 + 0.1
Random (512)
66.2 (-0.6)
64.3 (-5.6)
0.1 (-0.9)
0.1
1666 0.2 (-0.)
16.7 (-3.2)
EA (512, wio ReI.U)
66.3 = 0.2 (-0.5)
66.7 = (-2.2
0.9 (-0.9)
01.1+022.)
16.5 +..5++0.300 (0.7)
17.7+ (-2.2)
(512, ReLU) (ours)
66.5 0.1 (-0.3)
(400)
102202 (-0.7)
14.2 4M2::22 (+0.)
0. (-0.)
198 (-0.1)
on the labeled data (x. y) ~ P(z, y). where E y = fv...... .c.} denotes an observed label.
As discussed in Section 4.1 latent classes supposed to correspond more fine one-grained clusters within each observed class. For example, an observed class "dog" contains many species of dogs as its latent classes. Thus, we assume that each observed label y contains set of latent classes Cy C, and there is no overlap between different observed labels, i.e.. A i + j, C.. NCy, = 0.
The following theorem characterizes the optimal linear classifier under this setting, and shows that the linear classifier alone can attain the Bayes optimal classifier, arg max, P(y(z). In other words, ideally, linear classifier is enough for the classification task. Nevertheless, in practice, since the learning process is not as perfect as we have assumed, fully fine-tuning usually brings further gains.
Theorem 6. With the optimal NCL features 6(x), there exists a linear classifier g (z) = W such that its prediction p(x)) arg max, ((())())(()))))))))))))))))))))) is Bayes optimal, specifically,
W = [u...... wel, where w = Iv pPmm) (P((t, (P(M)) Ec,l. Vy E [C].
(10)
The optimal linear classifier g²(2) also enjoys good explanability, since only latent classes in Cy have non-zero weights in Wy. Therefore, we can directly inspect how each latent class belongs to a certain observed class by observing non-zero weights in W*.
5 APPLICATIONS
In this section, leveraging the ability of non-negative contrastive learning (NCL) to extract explana- tory features, we explore three application scenarios: feature selection, feature disentanglement, and downstream generalization. In the former two scenarios, NCL brings significant benefits over canonical CL. In the standard evaluation with downstream classification, NCL also obtains better performance than CL under both linear probing and fine-tuning settings. By default, we compare CL and NCL objectives using the SimCLR backbone on ImageNet- 100 with 200 training epochs. For all experiments, we run 3 random trials and report their mean and standard deviation.
5.1 FEATURE SELECTION
As shown in Figure 3b] NCL features are only sparsely activated on a few feature dimensions, and upon closer examination, many feature dimensions are only activated with few examples. Leveraging this feature sparsity property, we can sort out feature importance with non-negative features and use this guidance for selecting small subset of features while retaining most of the performance. This is particularly useful for large-scale image or text retrieval tasks by allowing flexible tradeoff between performance and computation cost, a natural way to realize shortening embeddings (OpenAl 2024)
2024) without additional training strategies as in Kusupati et al. (2022).
Here, we explore a simple strategy to rank non-negative features f f= [f1..... .. ful according to their Expected Activation (EA) on the training data, F.A (A((f) = (x)), where f(x) = (=)) is normalized. Intuitively, higher EA indicates common high-level features that represent class seman- tics, and lower EA corresponds to rare and noisy features that are less useful. In our experiments, select the top 512 features among all 2048 features of projector outputs, and evaluate them with linear probing, image retrieval, and transfer learning on ImageNet- (details in Appendix (G.4.
Results. Table I summarizes the results of feature selection experiments. First, when using all features, NCL achieves significant gains over CL on all three tasks. Meanwhile, when selecting 1/4 dimensions (512/2048), with EA criterion (w/ ReI.U), NCL. is almost "lossless", since it attains
7

---

Published as a conference paper at ICLR 2024
Feature disentanglement score (measured by SEPIN@ (Do & Tran 2020 of CL and NCL on ImageNet-100, where k denotes the top-k dimensions. Values are scaled by 10a.
Table 2:
SEPIN@1
SEPIN@
S 100
@El
CL
0.88 = 0.08
0.79 + 0.02
0.69 + 0.01
0.47 + 0.01
NCL
7.43 = 0.15
5.93 = 0.12
3.87 + 0.04
0.48 + 0.01
Table 3: Evaluation of learned representations on linear probing (LP) and finetuning (FT) tasks. (a) in-distribution evaluation on three benchmark datasets: CIFAR-100, CIFAR-1O, and ImageNet-100. (b) linear probing accuracy of ImageNet- 100 pretrained features three OOD datasets.
(a) in ii-distribution evaluation
(b) out-of-distribution transferability
Method
CIFAR-100
CIFAR-10
ImageNet. 100
Method
Stylized
Comupttion
Sketch
27.1 +
CL
58.6 0.2
12.6 2 0.1
87.6 =
0.2
92.3
= 0.1
68.7 1
0.3
77.3 = 0.5
NCL
21.2 1 0.2
36.1 0.3
28.0 1 0.2
NCL.
$9.7
730 0.2
87.8
92.6
0.1
69.4
79.2 0.4
comparable performance to using all features on all tasks. In comparison, CL features always suffer from larger drop when using fewer features. Thus, benefiting from the sparsity and disentangle- ment properties, NCL allows flexible feature selection that retains most of the original performance.
5.2 FEATURE DISENTANGLEMENT
According to Section 4.3 NCL features enjoy identifiability (under mild conditions), which is often regarded as sufficient condition to achieve feature disentanglement (Khemakhem et al.) [2020), while CL. suffers from severe feature ambiguities (Theorem T validate this property, we show that NCL. can significantly improve the degree of feature disentanglement real-world data.
Evaluation Metric. Many disentanglement metrics like MIG (Chen et al.. 2018) are supervised and require ground-truth factors that are not available in practice, so current disentanglement evaluation often limited to synthetic data (Zaidi et al. 2020). To validate the disentanglement on real-world data, adopt unsupervised disentanglement metric SEPIN @ (Do & Tran 2020) SEPIN @ measures how each feature (z) disentangled from others (z) by computing their conditional
mutual information with the top k features, i.e., SEPINGK = Σ [r, (z)),, (x)), which are
estimated with InfoNCE lower bound (Oord et al. 2018 practice (see Appendix G.3).
Results. As shown in Table 2 NCL. features show much better disentanglement than CL in all top-k dimensions. The advantage larger by considering the top features (e.g., 0.69 v.s. 3.87 SEPIN @ 100), since learned features also contain noisy dimensions. This verifies our identifia- bility result that with non-negativity constraints, NCL indeed brings better feature disentanglement on real-world data.
5.3 DOWNSTREAM GENERALIZATION
Finally, we evaluate CL and NCL on the linear probing and fine-tuning tasks using all features on three benchmark datasets: CIFAR-IO. CIFAR-100 (Krizhevsky al. 2009), and ImageNet-100 (Deng et al. 2009). Apart from in-distribution evaluation, we also compare them on three out-of- distribution datasets, stylized ImageNet (Geirhos al. 2018), ImageNet-Skent Sketch (Wang al. 2019). ImageNet-C (Hendrycks & Dietterich 2019) (restricted to ImageNet-100 100 classes). More details can be found in Appendix G.5, Following conventional setup, we adopt the encoder output for better downstream performance. See more results in Appendix F.4
Results. As shown Table 3 NCL shows superior generalization performance than original con- trastive learning across multiple real-world datasets. On average, it improves linear probing accu- racy by 0.6% and improves fine-tuning accuracy by 0.9%. Moreover, we observe from Table 3bb that NCL shows significant advantages in the out-of-distribution generalization downstream tasks (+1.4% on average). Considering that NCL only brings minimal change to CL by adding a simple reparameterization with neglectable cost, the improvements quite favorable. It also aligns well
8

---

Published as a conference paper at ICLR 2024
IiEE
Loss
From Scratch
Finetune
CE
76.1
78.6
NCE
78.6
80.2
CE + MLP projector
78.4
81.1
NCE MLP projector
79.2
82.0
Figure 4: Training from scratch with CE and NCE (w/o projector) on ImageNet-100.
Table 4: Test accuracy (%) of CE and NCE losses for supervised learning on ImageNer-100.
with the common belief that disentangled feature learning can yield representations that are more robust against domain shifts Guillaume et al. 2019).
EXTENSION TO BROADER SCENARIOS
In the above discussions, we have developed NCL in the context of self-supervised learning, where contrastive learning originates from (Oord et al., 2018). Meanwhile, contrastive learning also finds
wide applications in many other scenarios, such as, SupCon (Khosla et al., 2020)] Cui et al.] (2023) for supervised learning and CLIP (Radford et al. (2221a) for multi-modal learning. The generality of NCL enables easy extension. First, it is straightforward to generalize NCL to supervised contrastive learning since they adopt the same contrastive loss. Second. Zhang et al. (2023a) shows that multi- modal contrastive learning (MMCL) like CLIP is equivalent to asymmetric matrix factorization, ||A-FrFT 11 Leveraging this connection, we can similarly derive multi-modke NCL as equivalent to asymmetric NMF (Appendix C). We leave a systematic study on broad domains to future work.
Non-negative Cross Entropy (NCE). Besides existing variants of contrastive learning, we also note that NCL can be extended to standard supervised learning with cross- cross-entropy (CH) loss. Notice that exp(f(x) wwy)
the CE loss can be written as (fE(f) = Ez,y log Σ exp(f(w) Twa)' where fxx is the represen- tation of x and W l......, wc] contains the learned embeddings of each class in the last linear layer of C-way classification NNs. Thus, CE loss can be seen as a special case of InfoNCE when taking (z. y) as positive pairs and (f(z), wy) ) as their representations (i.e., a special MMCL loss (Appendix Ch.. Accordingly, we can derive the Non Non-negative Cross Entropy (NCE) loss by applying a non-negative transformation (e.g., ReL.U) to (f()), wyy in CE and eliminate rotation symmetry.
Setup. As a preliminary study, compare CE and NCE ImageNet- for 1) finetuning standard --peetrrrid encoder and 2) training from scratch, with ResNet-18 backbone (w/o projector). We also study adding MLP projector (as in SimCLR) before the last linear layer, allowing more flexible feature transformations before linear classification (details in Appendix G.6).
Results. Figure 4 shows that non-negative training with NCE is much faster than conventional CE loss at the early stage (around 2x before the 40th epoch). After training converges, NCL also enjoys slightly better performance. As shown in Table 4 in terms of the final performance, NCE is helpful for training from scratch (2.5%). With the help of the MLP projector network, NCL also attains significant gains by outperforming original CE by 3.1 % and CE-with-projeect by 0.8%. For finetuning, with MLP projector, NCE can attain a significant gain by improving 3.4% over CE and 0.9 over CE-with projector. These results show that although simple, non-negative training can be quite helpful for supervised tasks as well.
CONCLUSION
Despite the promising performance of contrastive learning, its learned features still have limited interpretability. Inspired by the classical non-negative matrix factorization algorithm, we proposed non-negative contrastive learning (NCL) to impose non-negativity on contrastive features. With minimal modifications to canonical CL, NCL can maintain or slightly improve its performance on classical tasks, while significantly enriching the interpretability of output features. We also provided comprehensive theoretical analyses for NCL, showing that NCL enjoys not only good downstream performance but also feature identifiability. At last, show that NCL can be extended to benefit other learning scenarios as well. Overall, we believe that NCL may serve as a better alternative to CL. in pursuit of better representation learning methodologies.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Yisen Wang was supported by National Key R&D Program of China (2022D0110030)), Na- tional Natural Science Foundation of China (62376010, 92370129), and Beijing Nova Program (20230484344).
REFERENCES
Amir H. Abdi, Purang Abolmaesumi, and Sidney Fels. Variational learning with disentanglement- pytorch. arXiv preprini urXiv:19/2.083 2019.
Yoshua Bengio, Aaron C Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. arXiv preprint arXiv:/206.5538, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 179911118888 2013.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In NeurlPS, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Zhikui Chen, Shan Jin, Runze Liu, and Jianing Zhang. A deep non-negative matrix factorization model for big data representation learning. Frontiers in Neurorobotics, 2021.
Jingyi Cui, Weiran Huang. Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix factoriza- tion and spectral clustering. In SDM, 2005.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. In ICLR, 2020.
Xiao Fu, Kejun Huang, and Nicholas D Sidiropoulos. On identifiability of nonnegative matrix factorization. IEEE Signal Processing Letters, 25(3):328-332, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 181.v:/811.1223// 2018.
Joseph HA Guillaume, John D Jakeman, Stefano Marsili-LLibellf Michael Asher, Philip Brunner, Barry Croke, Mary C Hill, Anthony J Jakeman, Karel J Keesman, Saman Razavi, et al. Introduc- tory overview of identifiability analysis: A guide evaluating whether you have the right type of data for your modeling purpose. Environmental Modelling & Software, 2019.
Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning per- spective on oversmoothing and beyond. In ICLR, 2023.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2024
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 1606.08415, 2016.
Kejun Huang, Nicholas D. Sidiropoulos, and Ananthram Swami. Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition. IEEE Transactions on Signal Processing, 62(1))211-222 2014.
Eric Jang, S. Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2016.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In AISTATS, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Bohdan Kivva, Goutham Rajendran, Pradeep Kumar Ravikumar, and Bryon Aragam, Identifiability of deep generative models without auxiliary information. In NeurIPS, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. NeurIPS, 2022.
Hans Laurberg, Mads Græsbøll Christensen, Mark D Plumbley, Lars Kai Hansen, Søren Holdt Jensen, et al. Theorems positive data: On the uniqueness of nmf. Computational Intelligence and Neuroscience, 2008.
Daniel D Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factor- ization. Nature, 401(6755):788-7911 1999.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In ICMI., 2019.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv: 1903.06733, 2019.
Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation in adver- sarial contrastive learning. In ICLR, 2023.
Henryk Mine. Nonnegative matrices, volume 170. Wiley New York, 1988.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini 1r07 2018.
OpenAl. New embedding models and api updates, 2024. URL httpsttss///ppeai.ccc hwtpeeanni...mm/ooo..g..o
Subhadeep Paul and Yuguo Chen. Orthogonal symmetric non-negative matrix factorization under the stochastic block model. arXiv preprint arXiv:1605.05345 2016.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In EMNLP, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021a.

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021b.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
Zhiquan Tan, Yifan Zhang, Jingqin Yang. and Yang Yuan. Contrastive learning is spectral clustering on similarity graph. arXiv preprint arXiv:2333./5103, 2023.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurIPS, 2019.
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and
Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurIPS, 2021.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
Yu-Xiong Wang and Yu-Jin Zhang. Nonnegative matrix factorization: A comprehensive review.
IEEE Transactions on Knowledge and Data Engineering, 25(6): 2012.
Julian Zaidi, Jonathan Boilard, G. Gagnon, and M. Carbonneau. Measuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. NeurIPS, 2022.
Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023a.
Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. In NeurlPS, 2023b.
12

---

Published as a conference paper at ICLR 2024
A RELATED WORK
The most closely related work is HaaChen et al. (202T). which firstly links contrastive learning back to matrix factorization. This link inspires us to leverage the classical non-negative matrix factor- ization technique, turn, to improve modern contrastive learning. Previously, matrix factorization has played important part in representation learning, for example, GloVe for word representation (Pennington et al. 2014), but was gradually replaced by deep models. Aside from our NMF-bbaed approach, many works study disentangled representation with generative models like VAEs (Kiggma & Welling 2014, Chen et af., 2018). Locatello et al. (2019) point that disentanglement is impossible without additional assumptions (Kivva al. [2222] supervision (Khemakhem et al., 2020). Nev- ertheless, these methods often have limited representation ability compared to contrastive learning in practice. In this work, our goal is to develop representation algorithm whose performance is as good as contrastive learning, while alleviating its interpretability limitations in real-world scenarios.
Relationship to CL. A deep connection between CL and NCL that they are both equivalent to matrix factorization problems w.r.t. a non-negative cooccurrence matrix (Eq. (3) & Eq. 4D; while the difference in non-negative constraints leads to different solutions, and it enables good explain- ability of NCL features. Theoretically. CL admits eigenvectors as closed-form solutions (though not computationally tractable). while NMF (NCL) problems generally do not permit closed-form solutions. Yet, we show that under appropriate data assumptions regarding latent classes, we can still identify an optimal solution, and establish its uniqueness and generalization guarantees under certain conditions. Thus, for anteed to deliver good performance, while offering significantly better explainability than canonical CL as discussed above.
data obeying our assumptions, NCL features are also guar-
Relationship to Clustering. In fact, Ding et al. (0055) also establish the equivalence between NMF, Kernel K-means and spectral clustering (SC) Leveraging this connection, NCL can also be regarded as an (implicit) deep clustering method. Previously, several works (HaoChen et al. [202]] Tan al. 2023) claim the equivalence between canonical CL and SC. However, this connection is not rigorous, because they only consider the equivalence between CL and eigendecomposition (the first step of SC), while SC also has a K-means step that utilizes eigenvectors produce cluster assignments as its final outputs (like NCL)
Relationship to MF and Deep NME. Matrix Factorization (MF) is a classical dimension reduction technique, with mathematical equivalence to many machine learning techniques, such as, PCA, LDA. However, it pointed out that features extracted by MF are often hardly visually inspected. A seminal work by Lee & Seung (1999) points that by enforcing non-negativity constraints on MF, i.e,, non-negative matrix factorization (NMF), we can obtain interpretable part-based features. In their well-known face recognition experiment, each decomposed feature is only activated certain facial region, while PCA components have no clear meanings. Ding et al.] (2005) further reveal the inherent clustering property of NMF, specifically, the equivalence between NMF and K-means clustering with additional orthogonal constraints. In this way, each NMF feature can be interpreted as the cluster assignment probability. Due these advantages, NMF widely applied to many scenarios, including but not limited to recommendation systems, computer vision, document clustering, etc. We refer Wang & Zhang (012) for a comprehensive review.
In the deep learning era, as a linear model, NMF is gradually replaced with deep neural networks (DNNs) with better feature extraction ability. Following this trend, various deep NMF methods have been proposed, where they replace the linear embedding layer with a multi-layer (optionally nonlinear) encoder, and enforce the non negaaiivyy constraint by either architectural modules or reg- ularization objectives. See Chen et al. (202T) for an overview of deep NMF methods. Nevertheless, these deep NMF methods can hardly achieve comparable performance to existing supervised and self-supervised learning methods for representation learning.
Our neg-negative contrastive learning can also be regarded as a deep NMF method, as we also lever- age neural encoder and show the equivalence between the NCL objective and the NMF objective.
IIn order make this equivalence hold, Ding 2005) add orthogonal constraints to NMF features, which are automatically satisfied under our one-hat latent label setting (Theorem
We note that theoretically, the K-means here is computed over the exponentillyy large population data X. So it not feasible practice utilize this connection to produce an NCL. solution using CL. features.
13

---

Published as a conference paper at ICLR 2024
However, the key difference is that do not apply NMF to the actual input (e.g.. the image), but to the co-oceurrence matrix implicitly defined by data augmentation. In this way, we can transform the original NMF problem to sampling based contrastive loss, which allows us to solve it with deep neural networks.
sampling-based
B PROOFS
B.1 PROOF OF THEOROEM
Proof. We expend LNMP and obtain:
CNMF (F) =IA 11
P(r,x+) VP(x)s+(a) (P(E+) (Pa
Σ
(P(o)P(14)
2,34
P(x,x+)2
(P( f+(++)
P(+)P((++
Plo,z+)2 P(x)P(14)
me f+) (x+ Bu, (f+(x)T(4x7)
=NCCL const,
which completes the proof.
B.2 PROOF OF THEOREM 3
Proof. We expand LNMP (0) and obtain:
(NMF(0) = =
p(x,
((T)((()) T [
((x))((s.)
2,
P(z,
(P(x)P(14)
(P(E)P(C)PP(T))
P(r,34)
P( 2
(c())((++)
((())((t))P(T))
P(c,zt)
P(m))( (T.)
(P(x)P(c4)
P(())((ot)
2(z,2+)
(((t()))(()tt)) 2
Σ JAE
(P(2))(14) PP(z)((zt)
With Assumption we know that P(x, (x = P,P(e) So LNMP(B) = 0, ie,, Φ e arg min LNME- Combing with the equivalence between NMF and NCL, we obtain that & is an optimal solution of LXCL-
B.3 PROOF OF THEOREM 4
Proof. When each sample only belongs to one latent class c = =((), we have
[P(m)]))) ... Pm [x)] 1 (())- Combined with Theorem we have Φ(x) / P(F p(m)-
14

---

Published as a conference paper at ICLR 2024
As there only exists non-zero elements for o(x). obtain ||o(x)llo 1. And for the thogonality, we have (z)r P(x) p(5,)P(F,) (())= ( M(a)-ay- So when i # j. (z))()) = 0. And when i = J. (z))(r) p(x)) PP(x) = 1. Then we obtain: (20(z)0(z) =I.
B.4 PROOF OF THEOREM 5
Proof. First, we introduce two useful results from previous works.
Lemma (Lemma of & Chen (2016)). For any N N symmetric matrix A, if (rrnk(() k S N, then the order l exact symmetric NMF of A is unique up to an orthogonal matrix.
Lemma 2 (Lemma 1.1 of Mine (1988). The inverse of a non-negative matrix matrix non- negative if and only if M is generalized permutation matrix.
Let & E Nok be the feature matrix composed of N features, i.e., the z-the row is v ((z))(()) Rt Note that & satisfies Á = ΦΦ exact non non-negative matrix factorization of Á (Theorem 4 If not unique, there exists another F+ * d and F+ R. Nx such that A = F.F..
deduce H 4T, where T E Rkxk is orthogonal matrix. As $T is also an exact symmetric NMF of A, it satisfies +7 2 0. For each x E X, holds that <xx) TT 2 0. Since for every latent class
According Lemma I
there exists a sample Te. E X' such that
P(c(..) = 1, the e-th entry of ((xxe) is V p(a.) while all other entries are O. Denote the row vectors
of matrix T Ti,,,...., Te. As d(xx) "T 2 we must have Te 2 Applying this deduction to every latent class, we have Ve C,T. 2 0, T
Recall that T' 2 0 also an orthogonal matrix with T 1. According to Lemma T' must be a permutation matrix. Therefore, o(-) is unique under permutation.
B.5 PROOF OF THEOREM 6
Proof. We consider the y-th dimension of the prediction:
(d)x))
=(W') " TT(x)
m
layEE, P(T
P(Tr)
Σ P( (2) TeCC
=v(vlæ)
So arg max g" ((((()) = arg max Ply|z). In other words, the classifier attains the Bayes optimal classifier.
C EXTENSION TO MULTI-MODAL CONTRASTIVE LEARNING
In the main paper, we have proposed neg-negat contrastive learning (NCL) for self-supervised learning, where the co-occurrence matrix A is symmetric by definition. Apart from the self- supervised setting, we know that contrastive learning is also successfully applied to visual-language representation learning that involves multi-modal data, e.g., CLIP (Radford et al. 2021b). Zhang et al. (2023a) show that multi-modal contrastive learning is mathematically equivalent to an asym- metric matrix factorization problem. Based on this link, non-negative contrastive learning in the multi modal setting, that can also enable the interpretability of visual-language representations.
develop the asymmetric version of
15

---

Published as a conference paper at ICLR 2024
Following the similar spirit of symmetric contrastive learning, asymmetric contrastive learning also aligns positive pairs together while pushing negative samples apart. However, there exist differences in the data sampling process. Taking the image-text pairs as example, the set of all visual data with distribution Pv, and Xr, to denote the set of all language data with distribution PL. Then the positive pairs (22) sampled from the joint multi-modal distribution PM and the negative pairs (z7, AM : (Aas)a.... = Pas (Te, 21) as the asymmetric co-occurrence matrix. For the ease of theoretical analysis, consider the multi-modal spectral contrative loss:
use Xy to denote
independently drawn from Pv nad PL. And denote
LMMCI( Sr.) = -22 my .n (v( Jr. (2r) + Be (n(
(11)
where fv and Sn encoders that respectively encode vision and language samples. Corre- spondingly, we propose the multi-modal variant of NCL loss:
(NM fL+) -2 (v, (x))))))))))))))) + F B7 (e+
(12)
such that (x+) 2 0, (x)) 2 0, Vx, E Xv, E Nt.
And we introduce asymmetric non-negative matrix factorization objective:
LNNMF 11AM F F2+ such that FV FL+ 2
(13)
where (Fv+)a. PP((x,JJ++ (Fr.).,, = (P.(z)) (ai) and Av is the normal- ized co-occurrence matrix. Then we establish the equivalence between MMNCL and asymmetric non-negative matrix factorization objective following the proofs in Theorem 2
Proof.
LANME 11AM - Fv+ FT+1
Pas (As, 11)
(Pv(c.)lv=(E (PL(()))--( 2
= =
(P (P.)))
PM x1)²
MPM (1+(x1)
(v(x.))P((m))
+ PP(x.))L(z)) (fv+( ( (L+()))))
(2 x1)
2E, mV f1+ + En;,, (fv+ f(+
(r.
const
LMMNCL + const.
Following the proofs of Theorem 3] then we can obtain the optimal solutions of MMNCL under the positive generation assumption (VZ, Xy, 24 E X1, PM (2., 21) = (=x((x))))(x(
(x)x)=
((TII
P (mm](x))
E ER Vx x. E Xv:
(((m))
(((mm)
(1(x) = VP((T)
P(
(P(Tm) P( (21)
E ERT in E Xi.
We observe that the optimal representations of MMNCL have similar form to NCL, As result, we can directly follow the theoretical analysis in NCL and prove that the learned representations of MMNCI also show semantic consistency such as dimensional clustering, sparsity and orthogonality.
16

---

Published as a conference paper at ICLR 2024
(a) Linear Accuracy
(b) Class Consistency
(c) Feature Sparsity
(d) Feature Correlation
Figure 5: Comparison between spectral contrastive learning (SCL) and non-negative contrastive learning (NCL) on CIFAR-I0: a) linear probing accuracy: b) class consistency rate, measur- ing the proportion of activated samples that belong to their most frequent class along each fea- ture dimension; c) feature sparsity, the average proportion of zero elements ([æl 1e-5) in the features of each test sample; d) dimensional correlation matrix of 20 random features:
vii,), C.j = [x fz(x), where fi(x)) [((z)// /
D ADDITIONAL EMPIRICAL ANALYSES
D.1 PERFORMANCE ON THE SPECTRAL CONTRASTIVE LEARNING LOss
In the main text, we mainly consider the widely adopted InfoNCE loss. Here, we provide the results with the spectral contrastive loss (SCL) that our analysis is originally based on.
Setup. During the pretraining process, we utilize ResNet-18 the backbone and train the models on CIFAR-10. We respectively pretrain the model with Spectral Contrastive Learning (SCL) and NCL. for 200 epochs. We pretrain the models with batch size 256 and weight decay 0.0001. When implementing NCL, we follow the default settings of SCL. During the linear evaluation, we train classifier following the frozen backbone for 100 epochs. For the experiments that compare semantic consistency, feature sparsity and feature correlation, we follow the settings Section 3.1
Results. As shown in Figure 5 NCL can also attain slight improvement over SCL in linear probing (88.6% v.s. 88.0%), while bring clear interpretability improvements in terms of class consistency (Figure 5b), sparsity (Figure 5c) and correlation (Figure 5d). It shows that NCL indeed works well across different CL objectives.
D.2 COMPARISON TO SPARSITY REGULARIZATION
Another classic method to encourage feature sparsity is directly adding & regularization on the output features, leading to the following regularized CL objective,
LCL (E
(14)
where À coefficient for regularization strength. We find that À 0.01 is good tradeoff between accuracy and sparsity on CIFAR-10.
Results. As shown in Table 5 NCL outperforms & regularized SCL both classification accuracy (88.5% v.s. 87.2%), which indicates that & regularization hurts downstream performance while our nen-negative constraint can help improve it. As for feature sparsity, we find that 69.654% of NCL. features are sparse while & regularization only achieves 0.03% sparsity. Therefore, NCL. is more effective than sparsity regularization terms of both downstream performance and promoting feature sparsity.
Table 5: Comparing NCL to f based sparsity regularization CIFAR-10.
Linear Accuracy (Encoder) (%)
Linear Accuracy (Projector) (%)
Feature Sparsity (%)
SCL
88.0
85.4
0.00
+ 4 reg
87.2
84.1
0.03
NCL
88.6
86.2
69.65
17

---

Published as a conference paper at ICLR 2024
E EMPIRICAL VERIFICATIONS OF THEORETICAL ANALYSIS ON NCL
E. EMPIRICAL VERIFICATIONS OF OPTIMAL REPRESENTATIONS
To further verify the theoretical analysis of the optimal solutions in Section 4 we further investigate properties of the learned representations on real-world datasets, e.g., CIF-AR-1O. As we have no knowledge of the latent classes of CI0 we consider simplified case, where we assume that the 10 classes in CIFAR-10 are latent classes, and generate positive samples by randomly sampling two augmented samples from the same class, in other words, supervised contrastive learning (Sup- Con) (Khosla et al. 2020). Then we train a ResNet-18 18 with CL and NCL objectives for 200 epochs, respectively.
By observing the largest 50 expected activation (EA) values of learned representations (Figure 6a find that the features of NCL are mostly activated in the first 10 dimensions, which is highly consistent with the ground-truth number of (latent) class in this setting with C = 10. In comparison, CL features do not differ much from one feature another, showing that they do have good dimen- sional feature interpretability. Also, the eigenvalue distribution in Figure revealss that CL features collapse even quicker than NCL features, and degrade small values even before 10 dimensions, while the eigenvalues of NCL features only degrade quickly after the first 10 dimensions. Therefore, we believe that the learned NCL features do lie closer to the ideal ground truth.
1402
0.5
1253
Eigono un
6.6
9303
822
6.2
20
Dimona on
Dimursion
(a) Distribution of the 50 largest EA values (b) Distribution of the 50 largest eigenvalues
Figure 6: Analysis of the learned features of CL and NCL on CIFAR-10.
E.2 EMPIRICAL VERIFICATIONS OF ASSUMPTIONS IN THEOREM 5
To verify the assumptions we used in Theorem 5 (for each latent class, there exists at least one sample in the datasets only has that feature activated), we conduct experiments to observe the activated dimensions of different samples. As shown in Figure samples generally have much fewer activated dimensions under NCL: around 70% examples have less than 20 activated dimensions, and in terms only activated on 2 dimensions. Considering that the noisy training
of the smallest, 16 examples in practice can lead to some redundant features and some gaps to the optimal solution, we believe that our assumption is indeed approximately true.
F ADDTIONAL RESULTS OF FEATURE PROPERTIES IN NCL
F.1 DISENTANGLEMENT EVALUATION ON SYNTHETIC DATA
In the main paper, we evaluate feature disentanglement of CL and NCL on real-world datasets, which are difficult without knowledge of ground truth. Here, following the common practice of disentangiement literature, we also evaluate disentanglement on Dsprites, a toy dataset where we are aware of the ground-truth latents (Abdi et al.] 2019). To be specific, we respectively train the models with original contrastive loss and NCL loss Dsprites following the settings of Abdi et al. (2019). Then we evaluate the disentanglement by the widely adopted disentanglement metric Mutual Information Gap (MIG) (Chen et al. 2018).
18

---

Published as a conference paper at ICLR 2024
182
NCL
1.25
200
Samples
Figure 7: The number of activated dimensions of the representations learned by CL and NCL on CIFAR-10.
shown in Table 6 NCL features also show much better disentanglement than CL on Dsprites as measured by MIG, which further verifies the advantages of NCL. Since Dspirtes is toy model that very different from real-world images, the default SimCLR augmentation could be suboptimal to obtain good positive samples. We leave more finetuning of NCL on these toy datasets for future work.
Table 6: Comparing disentanglement between CL and NCL on Dsprites.
Method
MIG
CL
0.037
NCL
0.065
F.2 MORE INTERPRETABILIN VISUALIZATION
Besides the CIFAR-10 examples in Figure I we perform the interoperability visualization exper- iments on CIFAR- 100 (Figure 8) and ImageNet-100 (Figure D) We can still observe clear dis- tinction in semantic consistency: activated samples along each feature dimension often come from different classes in CL while those in NCL mostly belong to the same class. verifies that NCL has clear advantages over CL in feature interpretability multiple datasets.
activation magnitude
activation magnitude
0 1 2
10 11
32
13
10 11 12 13 14
feature dimension
feature dimension
(a) Contrative Learning
(b) Non-negative Contrative Learning (ours)
Figure 8: Visualization of CIFAR-100 test samples with the largest values along each feature di- mension (sorted according activation values, see Appendix Gb.
F.3 PERFORMANCE UNDER DYNAMIC FEATURE LENGTH
To further investigate the benefit of sparsity in the representations learned by NCL, conduct ad-
ditional experiments to observe the performance expected activation (EA) with different numbers of selected dimensions. As shown in Figure T0 EA-based NCL. features keep high level of mAP
19

---

Published as a conference paper at ICLR 2024
Testt sampless top valuees
testt
14
feature dimension
Teature dimension
(a) Contrative Learning
(b) Non-negative Contrative Learning (ours)
Figure 9: Visualization of ImageNet-100 test samples with the largest values along each feature dimension (sorted according to activation values, see Appendix G
and have almost no degradation of performance. Instead, CL features keep degrading under fewer features and underperform NCL a lot at last. This experiment shows the clear advantage of the sparsity and disentanglement properties of NCL.
0.14
0.13
0.12
mAP
0.11
0.10
NCL (random)
(random)
0.09
CL (EA)
2048 1920
1408 12801152 1154 896 768 640 512
Dimensions
Figure 10: Image retrieval with CL and NCL features and two feature selection criteria (random, EA) on ImageNet-100.
F.4 DOWNSTREAM PERFORMANCE OF PROJECTOR OUTPUT
Following the common practice (Chen et al. 2020 He et al. 2020), in the main paper, feature trans- ferability by default evaluated based the encoder features (before projector) attain better downstream performance. For completeness, we also evaluate the features of projector outputs on CIFAR-10 and CIFAR-100. As shown in Table we can see that NCL (w/ ReLU) output can achieve comparable performance to CL outputs (w/o ReLU) while providing much better feature interpretability shown Figure Besides, Tables 3 & 3b shows that when pursuing inter- pretability, NCL can also induce better encoder features than CL. Overall, we believe that NCL is indeed favorable since it attains better feature interpretability while maintaining feature usefulness.
G DETAILS OF EXPERIMENT SETUP
G.1 IMPLEMENTATION AND EVALUATION OF NON-NEGATIVE FEATURES
In standard contrastive learning method like SimCLR, the encoder is composed of two subsequen- tial components: the backbone network (e.g., ResNet) g, and the projector network (usually
20

---

Published as a conference paper at ICLR 2024
Table 7: Downstream classification performance (accuracy in percentage) with projector outputs.
CIFAR-10
CIFAR-100
Method
LP
FT
LP
FT
CL (w/o ReLU)
85.6++.11
92.1+0.1
53.6+0.3
71,4+0.2
NCL. (w/ ReLU)
85.4++0.3
92.4+0.1
53.7+0.2
71.5+00.
MLP network) p. The projector output f() p(g(2)) is then used compute the InfoNCE, using the cosine similarity between positive and negative features. We apply the reparameterization at the final output of the projector output, Le., (x) = a+ (f()) Then, the non-negative feature is fed into computing the InfoNCE loss. All hyperparameters stay the same as the default ones.
Evaluation. Since non-negative constraints are only imposed on the projector output, we regard the projector outputs the learned features in the visualization, feature selection, and feature disentan- glement experiments, which aligns well with our theoretical analysis. When evaluating the feature transferability downstream classification, following the convention, also discard the projector and use the backbone network output for linear probing and fine-tuning.
G.2 EXPERIMENT DETAILS OF VISUALIZATION RESULTS AND PILOT STUDY
In all experiments in this part, adopt SimCLR the default backbone method (with ResNet-18), train the models for 200 epochs, and collect results on CIFAR-10 test data. In Figure [ we sort feature dimensions according expected activation (taking absolute values for contrastive learning features), and select the top samples with the largest activation in each dimension. In other results, the feature dimensions and samples are all randomly selected (without cherry-picking). Dimensions that have zero activation on all test samples (i.e., dead features) are excluded.
G.3 EXPERIMENT DETAILS OF FEATURE DISENTANGLEMENT
In practice, for the ease of computation, we first decompose the SEPIN metric based on the definition of the mutual information:
[(x, (r) (x)) [xx (x))
(15)
= [xx - (x, (+),
In the next step, we calculate two terms respectively. As it is quite hard to calculate the mutual information directly, we use tractable lower bound: the InfoNCE (Oord al. 2018 estimate
that. To be specific, for the mutual information (Ix, f()),, we estimate it with (xCE()),
exp(f(x)T f(x*))
I(x.f(x) LNCE(f) -EE,,r log
(16)
cxp(f(=) + E cxp(f(=)T((x7))
where f is the learned encoder. Similarly, for the mutual information [(x, fz)())), we obtain:
(x) fx (z+))
T(x, f+i(x)) z LNCE(f#) = -E log
( + Σ exp( (4((
(17)
Then we respectively calculate two terms on the test data of ImageNet- 100 with the encoders learned by NCL and CL. Following the definition of SEPIN @ ks we sort the dimensions based on the values of SEPIN, and SEPIN @ calculates the mean values of k largest SEPIN values of learned represen- tations.
G.4 EXPERIMENT DETAILS OF FEATURE SELECTION
During the pretraining process, utilize ResNet-18 18 (He al. 2016) the backbone and train the models on ImageNet- 100 (Deng et al. 2009). We follow the default settings of SimCLR (Chen 2020 and pretrain the model for 200 epochs. et al.
Linear Probing on Selected Dimensions. During the evaluation process, we train a linear classifier following the frozen representations with the default settings of linear probing in SimCLE L.R. As shown
21

---

Published as a conference paper at ICLR 2024
in Table D the features of NCL selected with EA show significantly better linear accuracy, which verifies that the features of NCL are interpretable and EA values can discover the most important semantic dimensions related to the ground-truth labels.
Image Retrieval on Selected Dimensions. For each sample, we first encode it with the pretrained networks and then select dimensions from the features. Then we find 10 images that have the largest cosine similarity with the query image and calculate the mean average precision @ 10 (mAP@@ 10) that returned images belong to the same class as the query ones. As shown in Table [ we observe the dimensions selected by EA values of NCL can find the intra-class neighbors more accurately, which further shows that NCL can find the most important features that are useful downstream tasks.
Transfer Learning on Selected Dimensions. In the downstream transfer learning tasks, we train a linear classifier on stylized ImageNet- 100 (Geirhos et al.) 2018) following the frozen representations with the default settings of linear probing. As shown in Table the 512 dimensions with the largest EA values show superior performance, which implies that the estimated importance of NCL is robust and effective in downstream transfer learning tasks.
G.S EXPERIMENT DETAILS OF DOWNSTREAM CLASSIFICATION
During the pretraining process, we utilize ResNet- 18 He et al. 2016) the backbone and train the models on CIFAR- 10, CIFAR-100 and 100 (Deng et al. 2009). We pretrain the model for 200 epochs on CIFAR-IO, CIFAR-100, and 100 epochs for ImageNet- 100. We compare NCL and the original contrastive learming with the SimCLR (Chen et al. 2020) backbone. For CIFAR- 10 and CIFAR-108 100, the projector is a two-layer MLP with hidden dimension 2048 and output dimension 256. And for ImageNet- 100, the projector is a two-layer MIP with hidden dimension 16384 and output dimension 2048. We pretrain the models with batch size 256 and weight decay 0.0001. When implementing NCL, we follow the default settings of SimCLR.
During the evaluation process, we consider three generalization tasks: in-distribution linear evalua- tion, in-distribution finetuning and out-of-distribution generalization. During the linear evaluation, we train a classifier following the frozen backbone pretrained by different methods for 50 epochs. For the in-distribution finetuning, train both the backbone and the classifier for 30 epochs. And for the ation process original ImageNet- 100. Then we evaluate the linear accuracy stylized ImageNet (Geirhoo et al. 2018). ImageNet-Sketeh (Wang et al., 2019), mmageNet-Corrputii Dietterich] 2019). As we pretrain the network on ImageNet-100,, we select the samples of the corre- sponding 100 classes from these out-of-distribution datasets and evaluate the accuracy.
generalization, we use the linear classifier obtained with the linear evalu-
(Hendrycks &
G.6 EXPERIMENT DETAILS OF SUPERVISED LEARNING
For the fine-tuning tasks, we first train a model by SimCLR. During the pretraining process, we utilize R8sNet-18 (He et al.) 2016) as the backbone and train the models on ImageNet-100 (Deng) (et al., 2009). We pretrain the model for 200 epochs. We use a projector which is a two-layer MLP with hidden dimension 16384 and output dimension 2048. We pretrain the models with batch size 256 and weight decay 0.0001. During the fine-tuning process, we train a classifier following the backbone for 50 epochs respectively with supervised and non-negative supervised learning and we follow the default settings of fine-tuning. When implementing the non-negative supervised learning, we select the ReLU function as the non-negative operator.
For the training from scratch, we also follow the default settings of fine-tuning. And we training the randomly initialized ResNet-18 with supervised and non-negative supervised learning for 100 epochs on ImageNet-100. When implementing the nen-negative supervised learning, we select the ReLU function as the non-negative operator. worth noting that we also apply ReLU functions on the linear classifiers during the training process.
In both tasks, we respectively apply the CE and NCE loss on the encoders with and without the projectors. We follow the default settings of the projectors used in SimCLR, i.e., the projector is a two-layer MLP with hidden dimension 16384 and output dimension 2048.
22


Reviews:
[{"review_id": "0oE3Od5Kg6", "reviewer": "Reviewer_Szt3", "summary": "The paper propose Non-negative Contrastive Learning (NCL), which is inspired by Non-negative Matrix Factorization (NMF), to derive interpretable features. Specifically, NCL add a non-negative transformation at the end of a standard encoder to generate non-negative features. The paper also demonstrates the advantages of NCL by experiments.", "questions": "1. In the first paragraph of sec 1, the authors show top activated examples along each feature dimension to demonstrate the interpretability of the features. Is this a common practice? Why is this approach considered reasonable?\n2. In the second paragraph of sec 2.2, why does the rotational symmetry in the optimal solution hurt the performance?\n3. The proposed method, simply adding a relu layer after the encoder, seems to be too easy to improve the performance.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The writing of this paper is clear, and the descriptions and justifications of the methods are comprehensible.", "weaknesses": "1. When presenting key arguments in the paper, some claims lack proper foundation, while others are based solely on partial visualization results from a specific dataset. I remain somewhat skeptical of these points.\n2. The results in Table 2 of the article indicate that the improvement from the proposed method is rather limited. I remain uncertain about the method's true efficacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQt7o2yuMi", "reviewer": "Reviewer_R2Pb", "summary": "The manuscript introduces a new self-spervised paradigm known as non-negative contrastive learning (NCL). Drawing inspiration from spectral contrastive learning (SCL) by HaoChen et al., 2021 and non-negative matrix factorization (NMF), the authors proposed a contrastive loss that enforces non-negativity constraints on the extracted features.\n\nThe authors demonstrate that, assuming minimal class overlap, NCL has the potential to learn a set of sparse features with low correlation. Under a one-hot representation, NCL achieves orthogonality among its features (Theorem 4). Furthermore, when at least one sample is uniquely assigned to each class, NCL learns a set of distinguishable and disentangled features (Theorem 5).\n\nTo validate the effectiveness of NCL, the authors conducted experiments on three benchmark datasets: CIFAR-10, CIFAR-100, and ImageNet-100, comparing it with a conventional contrastive learning methods, i.e. SimCLR. Across all datasets, NCL's features exhibit greater sparsity, consistency, and less entanglement, leading to improved representation of class identity in classification tasks.", "questions": "- It seems there might be a typo error in the proof of Theorem 2. It appears that the negative sample $x^-$ should be the positive one, $x^+.\" right?\n\n- If the primary objective of introducing non-negativity constraints is to encourage sparser representations, have you considered whether regularizing SCL loss with a sparsity regularizer might produce similar results?\n\n- Could you clarify whether the results presented in Tables 1 and 2, as well as Figure 5, are based on a single round of experimentation or multiple iterations? When the text mentions *\"on average\"* it would be helpful to specify the number of runs that the average is calculated over.\n\n- In section 4.1, Results, it is mentioned that *\"The advantage is larger by considering the top features... since learned features also contain noisy dimensions.\"* This statement appears to be somewhat contradictory to the claim of a less-entangled representation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The manuscript is generally well written. \n- The proposed contrastive learning paradigm is theoretically grounded, with a connection to NMF problem.", "weaknesses": "- **Novelty:** While the author has made a great effort to establish a connection betweenNMF and contrastive learning in the NCL approach, it appears that NCL can be seen as an extension of SCL with non-negativity constraints and similar justifications and proofs.\n\n- **Contribution:** The reported results in Table 2, and Figure 5 do not convincingly support a significant improvement brought about by NCL compared to the classical contrastive learning method. Furthermore, the absence of a direct comparison between NCL and SCL is notable.\n\n- **Generality and applicability of NCL's representation:** While the authors have theoretically demonstrated that NCL can yield sparse and less entangled representations suitable for downstream tasks like clustering and classification, the critique is that the suggested definition for an optimal and interpretable representations does not necessarily covers the broader goal of capturing all latent factors within the data structure. In many practical applications, e.g. in VAE studies or mixture modeling, the aim is to obtain comprehensive representations that also cover continuous and non-sparse variabilities, which remains unaddressed in this work.\n\nI appreciate the problem addressed by the author and introducing the constrained contrastive learning, which I find intriguing. The manuscript seems to have established a reasonable foundation. However, I think there are certain concerns that need to be addressed. I am open to revising my evaluation pending the author's response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZZCCbxKmw9", "reviewer": "Reviewer_eLwm", "summary": "The paper proposes the use of non-negative transformations on learned features in contrastive learning, which it calls Non-negative Contrastive Learning (NCL). It shows that non-negativity improves feature interpretability, sparsity and orthogonality compared to traditional CL. For each of these properties, the paper derives relevant guarantees for the ideal features, under certain assumptions of the data generating process. The paper also includes the results of experiments to show that NCL is superior to CL in feature disentanglement, feature selection and downstream classification.", "questions": "In section C.1, it is stated that the feature transferability to downstream classification is evaluated without using the projector. Does this mean that the features used are not non-negative?\n\nFor each feature (representing a latent class), is there at least one sample in the actual data sets used that (approximately) only has that feature activated (as in assumption of Theorem 5).\n\nIs learning something close to the ideal features reliant on a feasible amount of training data and a feasible amount of augmentations?\n\nWhy is the sparsity evidenced by performance with a subset of the original features rather than a subset of the learned features?\n\nIn Appendix C.3, the NCE loss does not include positive samples as it does in the main text. Is this intentional?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper derives several useful guarantees for NCL under ideal circumstances. Although I did not delve deep into the derivations, they seem sound based on the non-exhaustive checking I did.\n\nThe proposed benefits of NCL over CL are further backed up by the experimental results. Although the derivation are in-depth, the proposed modification to CL is fairly simple to understand and implement. Whilst some parts would benefit from a more thorough explanation, the paper is generally well-written.", "weaknesses": "A few sections required re-reading and/or reading other related papers to fully comprehend. For example, in Section 2.1 Preliminary on Contrastive Learning, the notation, the ideas about the natural data vs. augmentation data, population data vs. empirical data, positive samples and negative samples could be introduced more thoroughly, as is done in HaoChen et al. (2021). I appreciate that this may be due to lack of available space.\n\nIn section C.1, it is stated that the feature transferability to downstream classification is evaluated without using the projector. Does this mean that the features used are not non-negative? Section 4.3 claims that the downstream classification “aligns well with the common belief that disentangled features are more robust against domain shifts”, and yet the empirical and theoretical justification for the features being disentangled is based on the non-negative features (after the projector). This appears inconsistent.\n\nThe theoretical guarantees derived in the paper are based on assumptions. There is insufficient discussion of the applicability of these assumptions, and the guarantees, when real world data is used and the learned features are not ideal. For example, for each feature (representing a latent class), is there at least one sample in the data sets used that (approximately) only has that feature activated? Is learning something close to the ideal features reliant on a feasible amount of training data and a feasible amount of augmentations?\n\nIt is unclear to me why the feature selection task is chosen to show the sparsity of the NCL solution. Why is the sparsity evidenced by performance with a subset of the original features rather than a subset of the learned features?\n\nIn Appendix C.3, the derivation of lower bound of mutual information as the NCE loss should be more thorough, or cite a paper with a more closely aligned derivation, as it has a different form to the original paper (Oord et al., 2018). Also, the NCE loss does not include positive samples as it does in the main text. Is this intentional? \n\nThe expectation in equation 1 appears to be over just positive samples.\n\nThe paper would be more reproducible if the code were made public.\n\nComment: My initial rating is \"5: marginally below the acceptance threshold\", but if the points above are well-addressed, I would be willing to increase my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owDsZrxU1Z", "reviewer": "Reviewer_kpGZ", "summary": "The authors propose a novel contrastive learning method called \"nonnegative contrastive learning\", which takes its name from nonnegative matrix factorization. Using an output activation function for the learned features, e.g. $ReLU(f(\\mathbf{x}))$, the authors make the connection of the nonnegative matrix factorization minimization loss to the spectral contrastive learning loss.\nThe novel objective should lead to more interpretable features.", "questions": "- what is the connection/link between semantic consistency and clustering accuracy? I could not find any details on that in the manuscript.\n- Is the CL objective also trained with the spectral loss function? The results in the original paper seem to indicate that it does not lead to the same performance as in the original SimCLR paper. Maybe it would make sense to include the results generated by the standard SimCLR objective.\n- are there more interpretability results than the qualitative results provided in Fig. 1?\n- In sec. 4.2 you describe the feature selection procedure based on the TA. Do the results hold if you take the sum over absolute feature values, e.g. $ TA_i (f) = \\sum_x \\tilde{f}_i (x)$? Given the nonnegative constraints of the NCL features, the metric seems more tailored towards NCL than CL.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Interpretability of ML methods and learned representations is an important problem. For supervised and self-supervised learning. Therefore, I like the idea of the paper and its research direction.\n- Combining contrastive learning and nonnegative matrix factorization is a cool idea and innovative.\n- The paper is well written and a good read. Interesting!", "weaknesses": "- In my opinion, running experiments over multiple seeds and reporting the mean and standard deviation is essential and. necessary. For me, this is an important point. And to me, it has to be fulfilled to be accepted.\n- The y-axes in Figure 5 give a biased/wrong impression of the performances of the different models, in my opinion. In combination with the lack of standard deviations, it is difficult to evaluate the performance differences of the two methods.\n- I understand that evaluating disentanglement on real-world datasets is challenging given the lack of ground truth. Another possible way of really evaluating the disentanglement of the learned representations would have been the use of a toy dataset for this experiment. The known problems with estimating MI and not only a lower bound of it make it difficult to evaluate the results for this experiment.\n\n\n### Comment\nI really like the idea of the paper and the direction of research. Therefore, I am more than happy to change my score if the experimental evaluation and the reporting of results include some measure of their stability  (e.g. mean and standard deviations or something similar).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)