Learning Contrastive Embedding in Low-Dimensional Space
Authors: Shuo Chen
Chen Gong
Jun Li
Jian Yang
Gang Niu
Masashi Sugiyama
Venue: NeurIPS 2022 Accept (2022)
Keywords: contrastive learning
dimensionality reduction
autoencoder
representation learning

Abstract: Contrastive learning (CL) pretrains feature embeddings to scatter instances in the feature space so that the training data can be well discriminated. Most existing CL techniques usually encourage learning such feature embeddings in the highdimensional space to maximize the instance discrimination. However, this practice may lead to undesired results where the scattering instances are sparsely distributed in the high-dimensional feature space, making it difficult to capture the underlying similarity between pairwise instances. To this end, we propose a novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which adopts a regularized projection layer to reduce the dimensionality of the feature embedding. In CLLR, we build the sparse / low-rank regularizer to adaptively reconstruct a low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastive embeddings that alleviate the above issue. Theoretically, we prove a tighter error bound for CLLR; empirically, the superiority of CLLR is demonstrated across multiple domains. Both theoretical and experimental results emphasize the significance of learning low-dimensional contrastive embeddings.

Learning Contrastive Embedding in Low-Dimensional Space
Shuo Chen', Chen Gong", Jun Li", Jian Yang", Gang Niu", Masashi Sugiyama*
Abstract
Contrastive learning (CL) pretrains feature embeddings scatter instances in the feature space so that the training data can be well discriminated. Most existing CL techniques usually encourage learning such feature embeddings in the high- dimensional space to maximize the instance discrimination. However, this practice may lead undesired results where the scattering instances are sparsely distributed in the tigh-dimensional feature space, making it difficult to capture the underlying similarity between pairwise instances. To this end, we propose novel framework called contrastive learning with low-dimensionul reconstruction (CLLR), which adopts a regularized projection layer to reduce the dimensionality of the feature embedding. In CLLR, we build the sparse llowerawk regularizer to adaptively reconstruct low-dimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning contrastive embeddings that alleviate the above issue. Theoretically, we prove a tighter error bound for CLLR; empirically, the superiority of CLLR is demonstrated across multiple domains. Both theoretical and experimental results emphasize the significance learning oow-dimesiiona contrastive embeddings.
Introduction
Recently, unsupervised learning approaches have been greatly promoted by the contrastive learning (CL), which shows encouraging performance compared to fully supervised approaches [8] 21 CL
pretrains deep neural networks with unlabeled instances, and the learned feature embeddings can be directly used to extract features from the raw data [35]. Thereby, CL has been successfully applied in many downstream recognition tasks such as classification 442 retrieval 41, and clustering 3
As an unsupervised learning problem setting where the human annotation is not available, CL approaches usually consider building the pseudo supervision in their lcarning objectives [36] 19], and thus CL is also regarded as a self-supervise of CL is to push away each pair of instances scatter data points in the feature space, by which all instances in the training data can be well discriminated (i.e., the instance discrimination) [4] 40).
learning approach. Originally, the pseudo supervision
This original design has been empirically validated be particularly effective in the representation learning (28 () and has also been theoretically proved to approximate an unbiased supervised learning objective B2 Many recent efforts have increasingly focused on two different directions to further improve the performance of CL. The first one is to introduce plentiful lata-augmentation
SS. Chen and G. Niu are with RIKEN Center for Advanced Intelligence Project (AIP), Japan (E- mail: achuo.chen.yee ya @a@riken.jjpp.. gang. niu.ml mi @ com)).
&C. Gong, J. Li. and J. Yang are with the PCA Lab, Key Lab of Intelligent Perception and Systems for High- Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology, China (E-mail: fjunli, chen.gong, esjyang] @ @njust.edu.cn)
"M. Sugiyama is with RIKEN Center for Advanced Intelligence Project (AIP), Japan; and also with the Graduate School of Frontier Sciences, The University of Tokyo, Japan (E-mail: sugi@ u.o-koyo...c.jp).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

to generate the positive pair which consists of each instance and its perturbation [35 28 Then, any two instances the training data are regarded as the negative pair, and the objective of metric can be used to learn feature embeddings that distinguish positive pairs and negative pairs. Nevertheless, the negative pairs built CI. are inherently noisy because they contain false negatives consisted of semantically similar instances [30]. Therefore, the second way to improve the performance of CL is to reduce the impact of false negative pairs. To this end, some recent works convert it to positive-unlabeled learning I1 27 and clustering problems [3 46] to reweight the importance of negative pairs, and thus constraining the undesired repelling of negative pairs 3
learning 33 43
Although the existing methods have achieved promising results to some extent, their reliabilities highly depend on the effectiveness of instance discrimination [20 [32. However, recent works usually encourage learning contrastive embedding in high-dimension space to maximally dis-
criminate instances, so that the dimensionality of self-supervised contrastive embedding 7 is set to be much larger than the dimensionality of traditional fully supervised embedding 110 44]
This practice makes data points sparsely distribute in the feature space (which is similar to the curse of dimensionality (88), and thereby the corresponding CL methods may fail to capture the intrinsic similarity between pairwise instances. Such a problem can hardly be solved by simply setting a low dimensionality for the output layer, it will cause the dimensional collapse with insufficient instance discrimination [20). Some popular compression approaches such as distillation techniques [S] 47] enable us to train small networks under the supervision of original contrastive embeddings, yet the improper similarity predictions can still be inherited from the original networks. Therefore, new CL method is desired to effectively learn the oow-imensional feature embedding.
In this paper, we propose a novel frame- work dubbed contrastive learning with low- dimensional reconstruction (CLLR) to explicitly address the above issue caused by the high di- mensionality CL. Specifically, we introduce a new sparse projection layer to reconstruct the features of instances low-dimensional space and meanwhile scatter all instances in the orig- inal high-dimensional space (see Fig. [T. Then, obtain the now-dimensional contrastive em- bedding which can also effectively distinguish instances in the training data. Theoretically, we prove a lower bound for the min-max distance ra- tio of the learned contrastive embedding, which ensures that CLLR can better capture the in- stance similarity than the existing CL models. Experimentally, our approach consistently improves the state-of-the-ard methods on vision, language, and reinforcement learning benchmarks. To the best of knowledge, are the first to propose learning the original contrastive embedding in low-dimensional space. The proposed method is very generic, so it can be applied in many existing CL models. Our main contributions are summarized as: D). we propose a novel framework to enhance the generalization ability of contrastive learning via introducing sparse/ / low-rank regularized projection layer to adaptively reduce the high dimen- sionality of contrastive embedding; II). we establish complete theoretical guarantees for our method by analyzing the error bound of distance predictions and the convergence of the learning algorithm, respectively; III). we conduct extensive experiments on real-world datasets to validate the superiority of our method over the sfute-of-the-ar CL approaches, and the results consistently emphasize the necessity/ / significance of learning low-dimensional contrastive embeddings.
Conventional Contraetive Lourning
Low--Bank
X
Regeal /Altrrcct Palrs Instances
Feature Recovesy
Daln
Encuder
Data
High Dimensional Space
Low www.Dimensiinallaccc Space 2'
Conceptual illustration of our proposed CLLRR. In our method, we discriminate all instances in high-dimensional space and introduce a sparse projec- tion layer (the red part) to reconstruct the features of instances in the low-dimensional latent space.
Figure 1:
Notations. We write matrices and vectors as bold uppercase characters and bold lowercase characters, respectively. We denote the training dataset X' = (æz; E Rem 1,2,.... .N where m is the data dimensionality and N is the sample size. Operators 11- I1-1|2,1. and ||-||- denote the vector/matrix fn-norm, f2.1 -norm (i.e., the sum of f2-norm for columns), and nuclear-norm, respectively.
1.1 Background & Related Work
In this subsection, we briefly review the background of contrastive learning and the related work.

---

Instance Discrimination & Contrastive Learning. As popular unsupervised/ self supfrvuuee learning approach, the basic goal of contrastive learning (CL) algorithm to learn a generic feature embedding Φ: Rm - TR.", which transforms the data point from m-dimensional sample space to H-ddiennional embedding space. The primitive CL. method called instance discrimination learns such an embedding by directly enlarging the following distance between each pair of two instances 2; and 2y in the training data [14 40
m.y) = (<x)) $2z,)ll2HH
where H is the dimensionality of the learned feature embedding. The design philosophy for instance discrimination is that when we scatter all instances the feature space, the characteristic of each instance are captured and thus the training data can be well memorized by the neural network [20] When we further generate the positive pairs (z, xt) by combining each single instance æ and its perturbation x we are able to use the noise contrastive estimation (NCE) loss [16] to learn a feature embedding & from positive and negative pairs. this paper, we focus on such NCE loss which
has the form of (+) -E m. exoel (ax) (( [(e((a) T D ((x) (4(a))))) ))) Here
instances æ and {zx 1 uniformly sampled from the training data X, and n is the batch size.
Admittedly, as the original prototype of CL, instance discrimination is very critical to ensuring the effectiveness of most CI. methods. However, the feature dimensionality settings in existing CL. methods are usually very high (e.g.. dog8-dimmnnioon and 4096-dimension in 9 ). which are much larger than the feature dimensionality in most fully supervised learning methods (e.g.. 512-dimension and (024-dimension in [TO 15)). We demonstrate that learning contrastive embeddings in such high-dimensional space can be weak in capturing the similarity between pairwise instances. To address this issue, in this paper, we propose novel framework to learn contrastive embedding in low-dimensional space, which uses sparse / low-rank regularized projection layer for reconstruction.
PCA & Autoencoder. As a classical unsupervised / sulf-super learning method, principal component analysis (PCA) has shown promising results many machine learning tasks [39 45
Actually, PCA shares very similar motivation with the instance discrimination of CL. It is well-known that PCA seeks for a vector p E R" to scatter instances in the projection space by maximizing the variance ExEx [(x )) pp
}. where E E R is the mean of all instances in the training data X.. Enlarging such a variance is quite similar to the instance discrimination of CL which also pushes away data pairs to scatter instances. PCA has another reconstruction based form Ege [|] x x1]]] which is equivalent to the variance maximization E mx x the projection matrix and E is the dimensionality of orthogonal space). To further improve the fitting ability of PCA for complex data, the non-linear extension Autoencoder introduces the non-linear activation function a and two different projection matrices P and P' to reconstruct the training data in the by minimizing the objective Exex (P"T o(p æ)) æ -|]]]_ Some further extensions such as masked autoencoder (MAE) 17 achieved very promising results in several downstream tasks.
In this paper, we are inspired by PCA/ Autoencoder to reduce the dimensionality of contrastive embedding based on a sparse / low-rank regularized reconstruction loss. Interestingly, from this perspective, our method can also be regarded as natural combination of two main existing self- supervised learning approaches.
2 Methodology
In this section, we first investigate the distribution of instances scattered by CL in the high-dimensional feature space. After that, we propose a novel framework dubbed contrastive learning with low- dimensional reconstruction by introducing a new sparse projection layer. The learning objective and the corresponding optimization algorithm are finally designed with convergence guarantee.
2.1 Motivation
As we mentioned before, the contrastive embedding & maps an m-dimensional instance into the H-dimensional feature space. Now we want to investigate the distribution of data points in such an H H-ddmensiioal space. We consider the H-ddimensional hypercube and its inscribed- susbbbehdssrrrper..e.. We suppose that the edge length of the H --dimensiinal hypercube is 2r, and the radius of its inscribed-
3

---

suprasphere will be r. Then their corresponding volumes in the high-dimensiona space are
Veube H) (2r) and Vsphere H = ((
(2)
respectively, where r(() is the gamma function [37 having a form of T(2)= = Ju et dt. We
further study the ratio of the suprasphere volume to the hypercube volume. We let /l and the formulation = Vephore (H) (en)/Ven((HH) equals to
lim (TT (HH ((H/2)))/2H-1 - lim x (H-11 22 (HH-1//22H--1== #-+00
lim ( (T/4) 1)/2 0, (3)
H-000
and thus we have sr Venhere (a)hV = 0 by using the fact that Venhere (H) Veube H 2 0. This result of volume ratio clearly reveals that the proportion of the inscribed-suprant suprasphere in the hypercube will gradually converge to with the increase of the dimensionality H. It means that, in the high-ding eigh-dimensional hypereube, random given data point is less likely to appear inside of the inscribed sirriedd-suppsphere (i.e., in the central area of the hypercube) but it will usually exist outside of the inscribed- inseribed-supraaee suprasphere (i.e., in the corner area of the hypercube).
However, the learning objective of CL expects to scatter all instances in the H-dimensional hypercube, and thus making the N instances sparsely distribute in the Ñ 2 corners. Specifically, for the common dimensionality setting H = 2048 in popular CL methods, we have that
Ñ = 2d = 22048 16512 > 10 >> 10 NN
which implies that the comner number Ñ is significantly larger than the sample number N. In this case, the distribution of instances in the feature space will be very sparse, and all instances are far away from each other. Thereby the learning algorithm can hardly capture the intrinsic similarity between intra-class instances, and the downstream recognition tasks will be affected.
To be more religious, we consider the min-max distance ratio to investigate the distance contrast in the high-dimensional space. For the indepen- dent and identically distributed (i.id.) instances x, Iz E ITT (1=1,2.... .......,) 12), their embeddings ((x:) and ((x:)) are also i.i.d. matter how the embedding is learned [3]. The following Theorem 0 reveals that the minimal distance (H) and the maximal distance Diaax (H) tend be equivalent in the high-dimensional space [I] (provided that (xx) and ((x,) are i..d. to each other), so the similarity between pairwise instances can hardly provide contrast to discriminate the intra-class and inter-class (as shown by the distance distributions in Fig. 2)
100
Cinonsienaaly 256
130 Dinessioriay -512
Ra Dimorsionist 2048
30
Frequnccy
0.5 Aoveinc
2.5 Distatce
2.5
Dileacc
Figure Distance distributions contrastive embed- dings Iearned on STL-10 with diffcrent feature dimen- sionalities 256, 512, and 2048.
Theorem 1. For any given i.i.d. random data points x, 31, I2, In E R", we denote D)*((H)= = max (I. x. have that =im { var De (x, (B)) 2;))33 0 and
(H)= n) and Duin (H) = mmin (D (l. Then we
P lim (Dmax (D#ax (H) - (H)) Diin (H) H
-0]=1
(5)
where the distance function D is defined in Eq. 1 and the feature embedding &b is learned from the training data and independent to the data points æ, LT, F2, Fn-
In summary, from the above analytical results, we can clearly find that it is very necessary to constrain the dimensionality of existing CL approaches in reasonable range. Motivated by this, in the next subsection, we provide the formulation of our proposed framework CLLR which reduces the dimensionality of contrastive embedding by sparse projection layer.
2.2 Formulation
As we discussed in the previous subsection, the feature embedding & transforms the raw data from m-dimeeniionl space into H-dimmnsiinnal space, where & is learned by the NCE loss. To avoid high-dimensional features, we may directly reduce the dimensionlaity of the output layer, but this will cause the dimensional collapse with insufficient instance discrimination (as discussed in

---

Thereby, we consider to use an additional matrix L E R to transform the feature embedding result (x)) into the latent vector (x)), and then we minimize L (x)) (ax))) 112 encouraging the latent vector to preserve the useful information in ((x). When we further introduce the low-rank constraint for L, we can obtain a low-dimensional latent space for contrastive learning.
Section 1.
lg 1-oomm based Regularization. The row (column) sparsity is long-standing concept which aims to maintain very few non-zero columns for a matrix. When we employ the well-known é -norm to restrict the projection matrix L, We can certainly have a column-sparse L which selects the important features in ((x) E I corresponding to the non-zero columns. Then we use the selected features to reconstruct the original feature embedding, i.e..
R2,1 Φ L) EmEx I| ((x) - ((x)) + al1.,
(6)
where L LER and a > 0 is tuned by users. Note that the column sparsity is just special case of the low-rank, but considering its good usability, here we can easily obtain a -iwdimmnnional feature embedding L (ær) if the above L is column sparse. We also provide the following nuclear-norm based formulation to consider the more general case of low-dimensional space.
Nuclear-Norna based Regularization. To ensure the projection result L ((x)) in a low-dimensional space, a more general way is directly restricting the projection matrix L be low-rank. Then, the column vectors of L will be linearly dependent so that we can remove the redundant column to achieve -iwddmmnsional projection space. The realized formulation can be written
Rmuetcar @2 L) = Exex I L. -(()) - ((x))111 + a: DI|
(7)
where L E H and > 0 is tuned by users. When we obtain the learned projection matrix L', we need to further compute its maximal linearly independent set A, and then we calculate the final projection matrix L R H by setting] the redundant columns of L' to O.
For the above two different formulations in Eq. O6 and Eq. 7 it is hard to say which one is theoretically better. Actually, their final performance may also be influenced by the non-convexity of the learning objectives. Therefore, in our experiments, we evaluate both the two regularizations on multiple domains. Now we want to summarize our final learning objective as follows.
Learning Objective of CLLR. Based on the realized formulation in Eq. 66 and Eq. 7. we can casily deploy the proposed two regularizers in the learning objective of conventional CL methods. Without loss of generality, for most existing CL methods equipped with NCE loss, we build the following framework of contrastive learning with low-dimensional reconstruction (CLLR)
min LEH,
(F( L) LxCE ( + XR( L)).
where the regularization parameter x > 0 is tuned by users and the regularizer R (($, L) can be realized by R2,1(1 (#5 L) and Rescteu ② L) in Eq. (6) and Eq. 7 respectively. As a regularized learning objective, CLLR is very generic because here the loss term LxCE(#) can be implemented by many existing CL methods. In the next subsection, we provide iteration algorithm to solve Eq.
2.3 Optimization
Minimizing the objective function Eq. (8) is a typical batch optimization problem 48 where both the loss function LNCE($) and the regularizer R (1Φ, L) involve all training data. Therefore, we adopt the stochastic gradient descent (SGD) method (22) solve it, and here we demonstrate the stochastic gradient for the objective function F( L). Specifically, for n + 1 (i.e., the batch size) randomly selected data points (It, |2t, EX' by E B the NCE loss already has a stochastic form we need to demonstrate the stochastic loss for the regularizer in the mini-batch, i.e.,
n+1
so here
Rn L) = 11/(n + 1)) Σ
I L. ..((() - (1 + RR(L),
(10)
Here the i-column of L. is F, = L if L; E A and L. = 0 Di A. in which 1,2..... H.
Here we denote NCE loss (xce(pp) = Blllyp (20, |ntt where the function E(pp; (æo, )1th) = logg(expyy((ss,.. "p(xt, dex vector set B = (b- (l==(1,..... (bi,..., bn+1) Tb,, =
(0 (1 )) + exp(sspa 60x55, )))). The in-
N. bb. i 1 1......,++
5

---

Algorithm Solving Eq. (8) via SGD.
Input: Training Data X a ) m Step Size rl > 0; Regularization Parameter X, a > 0; Batch Size
N4.
Initialize: Iteration Number = 0. For /. from to T:
1). Uniformly pick (n + 1) data points {æa, | from X::
2). Compute the gradient of f(æ; (a,, } = ( ( XR ( L; (oo, }ب=ر) via
Eq. (
3). Update the learning parameters:
$ ((+1)) + (1) (0) ( L; (ab, ) and La++ +- V L; (mbb)) j=1 (9)
End.
Output: The converged & and L.
where R(L) indicates the penalty || ||2, or ||]|, for different regularizations. Here we use the subgradients (2) of f2,1-norm and nuclear-none -orm for optimziation. Then the learning objective F( L) in Eq. (8) has the stochastic form f(Ø; ) B L; ) } Based on such
stochastic loss, we further provide the SGD iteration steps in A solve Eq. (8)
In summary, introducing the projection layer (i.e., the projection matrix L) merely incurs additional stochastic gradient in Eq. (10). It means that our method can be easily implemented in most existing CL methods and only introduces very little computational overheads. In the next section, prove that the iteration sequence & (I)..... & (T) in Algorithm converges to a stationary point of the learning objective FF with a convergence rate (L/VTT), where T is the number iterations.
3 Theoretical Analyses
In this section, we further provide in-depth theoretical analyses for our proposed method. We investigate the convergence of learning algorithm and the lower bound of min-max distance ratio to demonstrate the effectiveness of our method. All proof's are given in supplementary materials.
3.1 Convergence Analysis
As we described before, the learning objective of CLLR is regularized empirical loss which is different from the traditional empirical loss solved by SGD, so here we provide careful convergence analysis for the SGD based iterations, i.e., the Algorithm I Specifically, objective has d- b-bounded gradient, and then we have the following Theorem 2
suppose the learning
Theorem 2. If the function F(Ø, L) has 8-bounded gradient (i...,
(VF < 8), then let
17 2( (0). (() F(Φ", L'" (S82)), and for the iterations in Algorithm 1 we have that
min B B[IVF(( [ S / 2S( (0), L(o) - F(Ø", L'))/TJO,
(11)
0S1ST-1
where S> 0 is lipschitz constant such that |VF Φ L) VF
- L'll22
The above Eq. (IT) clearly reveals that the iteration results in Algorithm can gradually converge stationary point with convergence rate (11/VT) when setting the proper learning rate 7) and increasing the iteration number T. Therefore, the convergence of our learning algorithm is guaranteed though the additional projection layer and regularization term introduced.
3.2 Lower Bound of Min-Max Distance Ratio
Now, we further analyze the distance between pairwise instances low-dimensional space. As we mentioned before, in the high-dimensional space, the min-max distance ratio trends to be 0 and thus the distance function will lose discriminatory. Therefore, we want to investigate the value of min-max distance ratio in low-dimensions feature space Iearned by method.

---

Our method explicitly constrain the dimensionality of the feature space, so it is intuitive that the min- max distance ratio ( - (H)) (H) (H) in Eq. ( should certainly be bower-bounded. To be religious, we have the following Theorem 3 to reveal the lower bound of distance ratio.
Theorem 3. For any given n + 1 i.i.d. random data points x; æ1; 22 In E RRT we denote that $ max{P 2 b.i (x, xi)li= = 1.2,.... ... and Dmin min(DD i (z, 2i)li ..2....,77) nz.. $
and then we have that
P (Diaxy
Dimnn
til) &L /D 2 2XC(2))=1 =1,
(12)
L,L $
where Dæ.ū. (x. xi) = [ L ar 1l/// D). and T and L are learned from Eq. (8)
From the above Eq. bound which is mainly determined by the two regularization parameters a and À (given the training data X²) It means that the low-rank reconstruction terms (i.e., Eq. (6) and Eq. make the min-mat distance ratio be controllable, and the larger regularization parameters can produce the better lower bound. When the min-max distance ratio is lower- bounded, our CLLR predicts low similarities for inter-cluster and high similarities for intra-cluster, so that the learned embedding effectively captures the intrinsic similarities/ features and thus improving the performance of downstream tasks.
we can easily observe that the min-max distance ratio has explicit lower
4 Experimental Results
In this section, we show experimental results on real-world datasets to validate the effectiveness of our proposed method, In detail, we first conduct ablation study to reveal the usefulness of our introduced new block and new regularizers. Then, compare our proposed learning algorithm with existing state-of-the-art models on vision and language tasks. Finally, we test our method on the CL based reinforcement learning task. Further experiments such as parametric sensitivity and running time comparison are given in supplementary materials. The training process is implemented on Pytorch [29] with NVIDIA Tesla 100 GPUs. We adopt the projection result (Φ(x)) for feature extraction, where regularization parameters À and a are fixed to 0.1 and 10, resepectively. The hyper- parameters of compared methods are set to the recommended values according to their original papers.
4.1 Ablation Study
Classification accuracy rates (mean + std) of high- dimensional embedding and low-dimensional embedding on STL- 10 and CIFAR-10 datasets (negative sample size 256).
Table 1:
In this subsection, we conduct abla- tion study on the superiority of the low-dimensional contrastive embed- ding (i.e., our method) over the tradi- tional contrastive embedding (i.e., the baseline method). We use the STL-10 and CIFAR- 10 datasets to train the base- line SimCLR [7] and two implemen- tations of CLLR, i.e., the f2 based regularization and nuclear-norm based regularization. We train all mod- els with 100 and 400 epochs with the same batch size and learning rate, respectively, and we record the test accuracy of all methods by fine- tuning a linear softmux. The baseline method learns contrastive embeddings in the high-dimensional space (dimension = 2048, 3072, and 4096) and the simply fixed low-dimensional space (dimension 256 and 512). We also include the baseline results that do not use the -norm and nuclear norm constraints (i.e., 0). Our method learns embeddings in low-dimensional space, where we use the regularizer to maintain the corresponding non-zero columns in the projection matrix L.
METHOD
STL-10 10
CIEAR 10
spochs=
epochs=100
epochs=
epochs=
2096-dim. (wlo R (b)
L))
55.1
75.2
3.1
65.1 +: 1.9
85.4
4.2
30772-dim. (wlo R. db.
L5)
1.4--:1 3.1
75.2 2: 2.1
67.2
2: 3.5
X6.9 6.1
2048-dim. (wlo R. (P.
L.) 56.1 1-211:21
76.
2 1.1
66.3 2 3.1
89.3
2.1
512-iim. (wla R2 T.)) 564
75. 0.
66.4 2
90.3
= 06
256-dim. (woa R( I.))
5-.1.141
742
64.32511
88.3
1.1
512-dim. (wil spacity
(6.50)66511
1.
75.5 0.55
66. 4.49
90.1
1.2
256-dim. (wil spaciyy a
55.9-t211
74.1 2:3
64.7 2-1
884
26
512-dim. (w/ (w//2))-))m)))
nomm)
56.3 56.3
78.3 055
67.5
0.2
92.5
0.2
512dii.. (w/ memeleeeee-nnooo)
56.2
791 022
67.5
92.5
2.3
256diim. (w/ (-00m)))
663.112-
2.55
65.5
0.5
92.5
2 0.S.
256-dim. (w/ nuclesr-numm) nurm)
533
202
55
93.1
1.3
We record the test accuracy (mean + std, 5 random trials) of compared methods at the 100-rh epoch and 400-th epoch in Tab. [ We can observe that the baseline method is better than our method in the first 100 epochs, but the two implementations of our method can outperform the baseline method with the increase of iterations. This is because that the baseline method only emphasizes on the instance discrimination, so it can quickly discriminate the training data in the early epochs. However, in the latter epochs, the low-rank reconstruction in our method becomes useful in capturing the similarity between pairwise instances. Mcanwhile, we can find that the average accuracy of nuclear-norm based
7

---

Accuracy Rate (8)
80
80
70
22
64
128
236
512
(a). Classification accuracy of all compared methods on STL-10 dataset
Accuracy Rate (8)
22
64
128
236
512
(b). Classification accuracy all compared methods dalaset.
Figure 4: Classification accuracy of all methods on STL-I0 and CIFAR-IO datasets. The negative
sample size is from 32 to 512.
regularization is slightly higher than the f2, -norm based one on both two datasets. Furthermore, we also perform the t-test at significance level 0.05 in the last column, and "v" indicates that our method is significantly better than the best baseline result. In our following experiments, we employ the 256-dimensional latent features for multiple domain tasks.
4.2 Experiments on Sentence Representation
In this subsection, employ the BookCorpus dataset compared methods on six text classification tasks, including movie review sentiment (MR), product reviews (CR), subjectivity classification (SUBJ), opinion polarity (MPQA), question type classification (TREC), and paraphrase identification (MSRP). We follow the experimental settings in the baseline method quick-thought (QT) (26), which chooses the neighboring sentences positive pairs. Here the 10-fold cross validation is adopted, and the average classification accuracy is listed in Tab. 2
23 evaluate the performance of all
For the six classification tasks, our method improves the classification accu- racy of baseline method QT for at least one percentage on most classification benchmarks. The distance histograms of QT. debiased contrastive learning (DCL) 11. hard negative based contrastive learning (HCL) (30]. and our CLRR are
Table Classification accuracy (%) all methods on Book-
Corpus dataset including six text classification tasks.
METIOD DEED HCT T0
MR
SUBJ
MPQA
TREC
MSRP
76.8
81.3
86.6
93.4
89.8
73.6
76.2
82.9
86.9
93.7
89.1
74.1
77.4
83.6
86.8
91.4
88.1
73.5
CLI.DDTT+a. 1TDCT442,,11 :-n0m))
77.9
83.3
87.9
93.7
91.3
75.2
CLLRIDCLHmcllr numn
78.2
87.2
95.8
75.7
shown in Fig. A We clearly observe that our method obtains the more accurate distance determination
than baseline methods, and this reveals that our method effective for the text classification task.
4.3 Experiments on Image Classification
In this subsection, we validate the effective- ness of our method on the image classification task. Here we select contrastive multiview cod- ing (CMC) (35) as baseline methods, and imple- ment our method CL.LR under such classical framework. We also compare our method with three additional state-of-the-a art methods includ- ing DCL, HCL, SwAV 3 and CO2 38 on STL- 10 [1). CIFAR- 10 24 and ImageNet- 100 [1] datasets. All methods are fairly implemented by the ResNer50 with the same training epoch 100.
DCL
CLLA
EJ
Delarce
Dsaee
Detsnce
Figure 3: Distance histograms obtained by different methods (QT. DCL, and our proposed CLRR) on Book- Corpus dataset. The proportion of incorrect prediction of CLLR is clearly lower than the compared methods.
For STL-10 and CIFAR-JO 10 datasets, we record the classification accuracy of all compared meth- ods with varying numbers of negative sample. From Fig. we can clearly observe that method CLLR successfully improves the baseline for at least 1% and 2% on CIFAR-10

---

dataset and STL-I0 dataset, respectively. Similar experiments are conducted on ImageNet-100 100 dataset, and Tab. shows that our method consistently improves all baseline methods, where our method improves the baseline CMC from 73.58% to 76.91%. For different negative sam- ple sizes, the accuracy rates of our method are also higher than all compared methods, and clearly demonstrates the effectiveness of our method. Since CLLR is implemented on different baselines, our method has good compatibility with existing CL algorithms on the image clas- sification task. In supplementary materials, we further compare our method with the distilla- tion based CL models [5] 47 (i.e., the low-dimensional small networks supervised by the orig- inal contrastive embeddinns), and the results clearly demonstrate the superiority of our method.
4.4 Experiments on Reinforcement Learning
This subsection further extends our experiments on reinforcement learning task, which is another application scenario of contrastive learning. Here the contrastive unsupervised representations for reinforcement learning (CURL) (25] method is employed to perform image-based policy control on representation learned by the CL algorithm. All methods are tested on the DeepMind control suite [44. which consists of six control tasks
Table 3: Classification accuracy (%) of all meth- ods on ImageNet-100 100 dataset with negative sam- ple size 1024 and 4096.
METHOD
1024
4096
Topl I
Top5
Tapl
Top5
CMC
60
79.23
73.58
SWAV
60:93
79.43
75.78
DCL
61.01
78.99
74.60
92.08
listed in Tab. 4 By following the experimen- tal settings in CURL, the positive pair is built
HCL X0
60.89
79.33
74.66
92.32
CO2 35
61.21
79.32
73.96
93.02
CLL norm)
62.03
80.64
75.97
94.22
by simply cropping a single image, and the neg- ative pair is composed of each two images the control sequence. All methods are retrained for 3 times, and the corresponding means and standards of 100K scores are shown in Tab. 4
CL. MCC
01.23
80.50
76.91
94.03
CLI B.LR(HCC++ -00m))
61.29
81.10
76.88
94.19
CI.I R(CC) anaclar-nom))
62.43
80.98
76.89
94.25
For the six control tasks, method consistently outperforms the baseline method CURL with higher means. When compared to DCL and HCL methods, our method almost achieves the best re- sults in all six scenarios. Although our method CLRR (CURL+nuclear norm) has slightly lower scores than CURL or DCL on the Run / Walk tasks, our method shows smaller variance. Moreover, when we incorporate our method to DCL, and HCL,, our method could further improve the overall scores of compared methods on the six tasks. This also reveals that our method is compatible with existing CL algorithms on the reinforcement learning task.
Table 4: 100K Scores (mean + std, 3 random trials) achieved by all methods on the six control tasks.
METIOD
Spin
Swingup
Easy
Run
Walk
Catch
CURL (25
413++-33
680 32
908 86
288
621-1121
8264-42
DCL [1]
422--23
672.452
878496
96
148
626199 %
836 12
HCL 50
420:61
678-122 82
869
116
2681-2
625
26
8194-22
CLLRCCURL
424--53
683-23
2964
625-13 23
843+1
CL (LLRDCL++)
423--13
83
919-57
267+67
625-13
844-27
CLLRICCL)
422--41
681
911
2924188
626:5.9
839
5 Conclusion and Future Work
In this paper, we considered the issue of nigh-diimessioal features existing in the current contrastive learning method. To overcome such an issue, we proposed novel framework called contrastive learning with low-dimensional reconstruction (CLLR), which uses a sparse projection layer to reduce the dimensionality of the feature embedding. We reconstructed the original high-dimensional features in the oowdiimensional projection space while preserving the basic objective for instance discrimination, and thus successfully learning low-dimensional contrastive embeddings. To the best of our knowledge, this is the first work in CL that considers reducing the feature dimensionality. We conducted intensive theoretical analyses guarantee the effectiveness of our method. Comparison experiments on real-world datasets across multiple domains indicated that our learning algorithm acquires more reliable feature embedding than state-of -f-ee-o--eeeeeeee t-thhartt methods. Both the theoretical and experimental results clearly demonstrated the necessity/ / significance of learning low now-dimensional contrastive embeddings. Our approach mainly focuses on the mainstream CL models which use both positive and negative pairs. The effectiveness of negative-free free CL has also been shown by recent works such as BYOL and SimSiam. When the negative pairs are unavailable, exploring the corresponding optimal dow-dimensional) projection space would be interesting future work.

---

Acknowledgment
S.C., G.N., and M.S. were supported by JST AIP Acceleration Research Grant Number JPMJCR20U3,
Japan. M.S. was also supported by the Institute for Al and Beyond, UTokyo.
C.G.,.L.., and J.Y. were supported by NSF of China (Nos: U1713208, 61973162, 62072242), NSF of Jiangsu Province (No: B72021013), NSF for Distinguished Young Scholar of Jiangsu Province (No: BK20220080), and the Fundamental Research Funds for the Central Universities (Nos: 30920032202. 30921013114).
References
[1] Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? In International conference on database theory, pages 217-235. Springer, 1999.
2.1
[2] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. 2.3
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in neural information processing systems (NeurIPS), pages 1401-1413, 2020.
[4] Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning (ICML), pages 1419-14299 2021.
[5] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages (6296-16300) 2021.
[6] Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, and Masashi Sugiyama. Large-margin contrastive learning with distance polarization regularizer. In International Conference on Machine Learning (ICML), pages 1673-1683, 2021. D
[7] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597-1607, 2020.
4.1
[8] Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated dositive-unlabelee training. In International Conference on Machine Learning (ICML), pages 1510-1519, 2020.
[9] Anoop Cherian and Shuchin Aeron. Representation learning via centsariilly-cootasttiin optimal transport. In International Conference on Machine Learning (ICML), pages 1820-1830,
2020.
1.1
[10] Xu Chu, Yang Lin, Yasha Wang, Xiting Wang, Hailong Yu, Xin Gao, and Qi Tong. Distance metric learning with joint representation diversification. In International Conference on Machine Learning (ICML), pages 1962-1973, 2020. I.I
[11] Ching- Ching-Yao Chuang. Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
[12] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks unsu- pervised feature learning. In International conference artificial intelligence and statistics (AISTATS), pages 215-223, 2011. 4.3
[13] Shib Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Li, and Andrew McCallum. Improving local identifiability in probabilistic box embeddings. Advances in Neural Information Processing Systems (NeurIPS), 33:182-199 192, 2020. 2.1
10

---

[14] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems (NeurIPS), volume 27, pages 766-774, 2014.
[15] Jason Xiaotian Dou, Lei Luo, and Raymond Mingrui Yang. An optimal transport approach to deep metric learning (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pages 19935-12936, 2022. I.1
[16] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation princi- ple for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 297-304, 2010. 1.1
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
[.]
[18] Gordon Hughes. On the mean accuracy of statistical pattern recognizers. IEEE transactions information theory, 14(1):55-63, 1968.
[19] Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging contrastive learning. In International Conference Machine Learning (ICML), 2021. [
[20] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.0934 2021.
[21] Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions Patlern Analysis and Machine Intelligence, 2020.
[22] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems (NeurIPS), 26:315-323, 2013.
2.3
[23] Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip thought vectors. Advances in neural information processing systems (NeurIPS), 28:3294-3302, 2015. 4.2
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009. 4.
[25] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning (ICML), pages 5639-5650, 2020. 4.4
[26] Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence rep- resentations. In International Conference on Learning Representations (ICLR), 2018. 4.2
2
[27] Gang Niu, Marthinus Christoffel du Plessis. Tomoya Sakai, Yao Ma, and Masashi Sugiyama. Theoretical comparisons of positive-umlabeled learning against positive-negative learning. In
Advances in neural information processing systems (NeurIPS), pages 1199-1207, 2016. [
[28] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
1807.037443 2018. [
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems (NeurIPS), 32, 2019. 4
[30] Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representation (ICLR), 2021.
2 4.2
[31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11(23) 2015. 43
11

---

[32] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. theoretical analysis of contrastive unsupervised representation learning. In Interna- lional Conference on Machine Learning (ICML), pages 5628 5637, 2019.
[33] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems (NeurIPS), 29:1857-1865, 2016.
[34] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXi:: 1801.00690 2018. 4.4
[35] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV), pages 1-18, 2020. 4.3
[36] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. !
[37] Martin J Wainwright. High-dimensional statistics: non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019. 2.1
[38] Chen Wei, Huiyu Wang, Wei Shen, and Alan Yuille. Co2: Consistent contrast for unsupervised visual representation learning. In International Conference Learning Representations (ICLR), 2021.
[39] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
[40] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3733-3742, 2018.
[41] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint vrXiv:2007.00808, 2020. [
[42] Xinyi Xu, Cheng Deng, Yaochen Xie, and Shuiwang Ji. Group contrastive self-supervised learning graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[43] Jiexi Yan, Cheng Deng, and Xianglong Liu. Dictionary learning in optimal metric space. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), volume 32, 2018. D
[44] Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12465- 2021.
[45] Jian Yang, David Zhang, Alejandro F Frangi, and Jing-yu Yang. Two-dimensional pea: new approach to appearance-bast face representation and recognition. IEEE transactions on pattemn analysis and machine intelligence, 26(1):131-13) 2004. I.1
[46] Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. arXiv preprint arXiv:2008. 003000000.000030, 2020.
[47] Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong, Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9260-9269, 2021.
[48] Martin Zinkevich, Markus Weimer, Alexander J Smola, and Lihong Li. Parallelized stochastic gradient descent, In Advances in neural information processing systems (NeurIPS), volume 4, page 4, 2010.
23
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theo- rem1/2/3.
(b) Did you include complete proofs of all theoretical results? [Yes] See the Appendix in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?[Yes] See Section and supplemental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Section 4
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4 and the supplyment material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "u0Bc-6Ssiv", "reviewer": "Reviewer_VTWD", "summary": "This work proposes a framework (CLLR) to learn embeddings through contrastive learning (CL). The proposed approach learns a low dimensional projection of the feature space used for CL. Furthermore, theoretical bound is provided for CLLR along with empirical explorations using the learned low-dimensional embedding.", "questions": "As mentioned earlier, a central question remains regarding the whole approach. The low dimensional embeddings can directly be learned through CL and the proposed regularizations. Therefore, learning embeddings through CL followed by a projection layer seems like added complexity. Perhaps an explanation on why the low dimensional embeddings cannot be learned directly might be useful.\n\nIn the empirical explorations, were the models trained using both CL and the downstream task based loss (like multi-task learning) ?", "limitations": "The authors do touch upon limitations of their approach where others approaches can potentially work better. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality :\n\nThe work is somewhat original. The authors combine well known techniques in manner that produces a useful framework for producing low dimensional features through CL\n\nQuality :\n\nThis work proposes a framework that is decently analyzed through theoretical bounds and empirical explorations using multiple diverse tasks. The authors also provide ablation studies to strengthen the proposed framework. However, a central question still remains. Why is it that the low dimensional embedding cannot be learned directly through CL. The proposed regularizations can also be applied to the final matrix that produces the features. The need for learning the CL based embeddings followed by a projection layer seems to add complexity that perhaps could have been avoided.\n\n\nClarity:\n\nThis work follows a structured approach and explains the procedure in a clear manner.\n\nSignificance :\n\nThe concept of low dimensional embeddings learned through CL is very useful. This work provides a useful step in that direction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q0yU6OJh-QX", "reviewer": "Reviewer_cX3t", "summary": "This paper discusses the contrastive embedding in low-dimensional space. The authors mentioned that contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Then the authors propose a new formulation of contrastive embedding under low low-dimensional space, and provide the corresponding convergence analysis and lower bound of min-max distance ratio. The experimental results were provided to show the advantages of the proposed algorithm.", "questions": "It would be great if the author could address the questions in weakness and comments.", "limitations": "none", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper has a good motivation, that is contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Based on the motivation, the authors proposed a new algorithm and give the corresponding theoretical analysis.\n2. The presentation of the paper is good. Have a clear problem setting, and provide a clear description of the target algorithm. For the theoretical analysis, the authors provide a clean derivation. \n\n\n\nWeakness:\n\n1. The authors claims that contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Thus, in the experiments of ablation study, the extreme cases should be provided to show that the contrasitive embedding in high dimensional space could conduct very poor performance. That is not included in Table 1.\n\n2. It is good to provide section 2.1 to show the motivation of this paper. If the example could be more direct to show the disaster of contrasitive embedding in high dimensional space, that would improve the quality of the paper.\n\n3. It is fine for this paper to plug the low-dimensional mapping into contrastive embedding. However, the novelty and contribution is not obvious to reach the bar of Neurips.\n\n\n\n===============================\n\nThanks the authors for providing the more experimental results. However, I do not think the new results convince me. I would like to decrease my score to 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KSzdA9-zohT", "reviewer": "Reviewer_eAPH", "summary": "This paper focuses on the issue of high dimensionality in the existing contrastive learning (CL) approaches. As CL usually encourages learning high-dimensional features to discriminate data points as much as possible, it may make the data points sparsely distributed in the high-dimensional feature space. In this case, the authors argue that the learning algorithm can hardly capture the underlying similarity between pairwise instances. Then the authors propose a new sparse regularized projection layer to address this problem, reconstructing the learned features in the low-dimensional (sparse) projection space and meanwhile maintaining the conventional instance discrimination of CL in the original high-dimensional space. The generic sparse/low-rank regularized loss functions are also designed with providing the corresponding optimization algorithm. Some theoretical analyses (Theorem 1, 2, and 3) are explored to demonstrate the motivation and effectiveness of the proposed method. Finally, experiments on multiple domains including image, text, and reinforcement datasets validate the superiority of the method.", "questions": "I have some more questions regarding the details of this work.\n\n1).  What is the dimensionality of the finally used embedding for fine-tuning (with softmax)? It seems that the reconstructed features are still high-dimensional. If they use the low-dimensional latent features, this should be clarified if this paper is accepted.\n\n2). I find that the authors omit some details in Eq. (3). It is unclear to us that why $\\lim_{H\\rightarrow\\infty}(\\pi^{H/2}/(H\\cdot\\Gamma(H/2)))/2^{H-1}\\leq\\lim_{H\\rightarrow\\infty}\\pi^{(H-1)/2}/2^{H-1}$ can hold. I think this is critically import to the motivation, so please try to explain this more carefully.\n\n3). In the optimization section, the authors use the classical SGD for model training. Is that possible if we try to use some more popular optimizers such as Adam to learning the objectives? Will this have any impact on the loss function $\\mathcal{L}_{NCE}(\\boldsymbol{\\varPhi})$ and the reguarlizer $\\mathcal{R}(\\boldsymbol{\\varPhi},\\boldsymbol{L})$?\n\n4). It seems that the concrete form of the lower-bound $\\alpha\\lambda C(X)$ is not described in the manuscript. I checked the mathematical proof in the appendix, and found that this lower-bound contains the learning parameter $\\boldsymbol{L}$ and $\\boldsymbol{\\varPhi}$. In this case, can we still say that the lower-bound $C(X)$ only depends on the training data $X$?\n\n", "limitations": "I think there is no potentially negative social impact of this paper. For improvement suggestions, please see the above strengths-weaknesses and questions. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper is interesting and easy to follow. I think it is worth to share this paper to the machine learning community, and the results seem to have the potentiality to emphasize the importance of learning low-dimensional contrastive embedding . Here are some pros and cons of this paper:\n\n**Strengths:**\n\n1). This paper is well-written. The storyline is very smooth to me. Fig. 1, Fig. 2, and Fig. 3 clearly depict the idea, motivation, and effectiveness of the proposed method. It has a strong motivation, which considers a generic case of contrastive learning. Especially, section 3 starts with an interesting and convincing example which illustrates that the hypercube corner number is significantly larger than the data point number. \n\n2). The issue of high-dimensional contrastive embedding pointed out in this paper is important. And I think this contribution is even more valuable than the proposed method itself. Nowadays, it is actually a common issue that the contrastive embedding has very high dimensionality which may lead to the curse of dimensionality. However, as the authors reviewed in the introduction, when we simply reduce the dimensionality of the output layer, it will yet cause the insufficient instance discrimination and harm the generalizability. So this phenomenon indeed leads to an awkward dilemma for us.\n\n3). The proposed idea is non-trivial and concise, and it is well supported by the theoretical analyses. This paper proposes using the additional sparse-regularized layer to reconstruct the high-dimensional features, which I think can well solve the dilemma I mentioned above. More importantly, this idea is supported by the theoretical analyses in Theorem 1, 2, and 3, where the lower-bound of the distance contrast is explicitly given so that the similarity and dissimilarity between pairwise instances can be distinguished from each other.\n\n4). The proposed method is built and implemented in a rather generic way. The authors conduct extensive experiments on multiple domains of data, and the improvements are also non-trivial. I find that the authors further supply the comparison experiments with the distillation-based methods. In the results, the proposed method can consistently outperform the distillation-based methods, and it seems that this can also demonstrate that why we need the new reconstruction layer for solving the issue of high-dimensional contrastive embedding.\n\n**Weaknesses:**\n\n1). As Theorem 1 is an existing work, the author should introduce the related preliminary of Eq. (5) more carefully. For example, does the convergence condition $\\lim_{H\\rightarrow\\infty} var[D_{\\boldsymbol{\\varPhi}}(x, x_i)/E(D_{\\boldsymbol{\\varPhi}}(x, x_i))]$ have any relationships with the final conclusion of the distance contrast? This is not very clear to me.\n\n2). It seems that the authors only conduct experiments on the regular positive-negative CL baselines. Does the proposed method work well with the negative-free CL approaches? As we know, currently the negative-free CL has also show encouraging performance which is even better than the positive-negative CL in some cases.\n\n**In summary:**\n\nBy considering the above pros and cons, I would believe this is a strong paper in both theories and experiments. I think the quality of paper undoubtedly reaches the acceptance bar of NeurIPS, so I would like to vote for a “clearly accept” and give a score 8 for it. In the rebuttal, I want the authors to further clarify and solve my concerns that I mentioned in the weaknesses and questions.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7I3hG8ICLnb", "reviewer": "Reviewer_4xAW", "summary": "This paper proposes a novel regularization method for contrastive representation learning. The main idea is to introduce an additional layer that projects the representation to a lower-dimensional space. The additional projection layer is trained by minimizing the reconstruction loss where sparse regularization is applied. The projected (relatively) lower-dimensional representation is shown to be more effective in multiple downstream tasks.", "questions": "* Is the representation space constrained to a hypersphere or not? \n* Does it assume a particular type of objective functions to be regarded as \"contrastive learning\"? \n* Is there any reason that the proposed method is not applicable to non-contrastive classifiers?\n* It would be informative if we can separate the contributions from reconstruction loss and sparsity regularization. What if we use reconstruction loss but without sparse regularization? There is no row in Table 1 corresponding to this setting.", "limitations": "I don't see any critical limitation of this work regarding negative social impact.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I vote for rejection mainly due to the lack of clarity and the lack of good explanation. Meanwhile, I do find the experimental results compelling.\n\nStrengths:\n\n* Contrastive representation learning is an important field of machine learning and is a relevant topic.\n* The proposed algorithm seems novel.\n* The experiments are fairly extensive and the results are impressive. The proposed method is tested across multiple tasks, including image classification, text classification, and reinforcement learning, showing consistent improvements. Also, some of the results are provided with confidence intervals from repeated runs.\n* I appreciate the significance of the empirical finding in this work.\n\n\nWeaknesses:\n* There is a large room for improvement in clarity. The paper needs to clearly define what contrastive learning is and what is not. Even though there is no single definition that is accepted across the whole community, at least the authors need to clarify what class of algorithm they are focusing on. This unclarity raises several questions during reading. Is the representation space constrained to a hypersphere or not? Does it assume a particular type of objective functions to be regarded as \"contrastive learning\"? Is there any reason that the proposed method is not applicable to non-contrastive classifiers? I will post this questions again on \"Questions\" section of the review.\n* I am not persuaded by the argument made in Section 2.1 Motivation in general. As far as I know, many contrastive learning algorithms assume a hyperspherical latent space, so there is no point of comparing the volumes of a high-dimensional cube and a sphere. \n* The argument from Theorem 1 is only indirectly relevant at best. In classification, what matters is the ordering of distances, not the exact values of distances. There is not direct evidence that high dimensionality hurts the quality of representation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept