Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper discusses the contrastive embedding in low-dimensional space. The authors mentioned that contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Then the authors propose a new formulation of contrastive embedding under low low-dimensional space, and provide the corresponding convergence analysis and lower bound of min-max distance ratio. The experimental results were provided to show the advantages of the proposed algorithm.
strengths_and_weaknesses: Strength:

1. The paper has a good motivation, that is contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Based on the motivation, the authors proposed a new algorithm and give the corresponding theoretical analysis.
2. The presentation of the paper is good. Have a clear problem setting, and provide a clear description of the target algorithm. For the theoretical analysis, the authors provide a clean derivation. 



Weakness:

1. The authors claims that contrasitive embedding in high dimensional space would result in the curse of dimensionality and making it difficult to capture  the underlying similarity between pairwise instances. Thus, in the experiments of ablation study, the extreme cases should be provided to show that the contrasitive embedding in high dimensional space could conduct very poor performance. That is not included in Table 1.

2. It is good to provide section 2.1 to show the motivation of this paper. If the example could be more direct to show the disaster of contrasitive embedding in high dimensional space, that would improve the quality of the paper.

3. It is fine for this paper to plug the low-dimensional mapping into contrastive embedding. However, the novelty and contribution is not obvious to reach the bar of Neurips.



===============================

Thanks the authors for providing the more experimental results. However, I do not think the new results convince me. I would like to decrease my score to 3.
questions: It would be great if the author could address the questions in weakness and comments.
limitations: none
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the issue of high dimensionality in the existing contrastive learning (CL) approaches. As CL usually encourages learning high-dimensional features to discriminate data points as much as possible, it may make the data points sparsely distributed in the high-dimensional feature space. In this case, the authors argue that the learning algorithm can hardly capture the underlying similarity between pairwise instances. Then the authors propose a new sparse regularized projection layer to address this problem, reconstructing the learned features in the low-dimensional (sparse) projection space and meanwhile maintaining the conventional instance discrimination of CL in the original high-dimensional space. The generic sparse/low-rank regularized loss functions are also designed with providing the corresponding optimization algorithm. Some theoretical analyses (Theorem 1, 2, and 3) are explored to demonstrate the motivation and effectiveness of the proposed method. Finally, experiments on multiple domains including image, text, and reinforcement datasets validate the superiority of the method.
strengths_and_weaknesses: Overall, this paper is interesting and easy to follow. I think it is worth to share this paper to the machine learning community, and the results seem to have the potentiality to emphasize the importance of learning low-dimensional contrastive embedding . Here are some pros and cons of this paper:

**Strengths:**

1). This paper is well-written. The storyline is very smooth to me. Fig. 1, Fig. 2, and Fig. 3 clearly depict the idea, motivation, and effectiveness of the proposed method. It has a strong motivation, which considers a generic case of contrastive learning. Especially, section 3 starts with an interesting and convincing example which illustrates that the hypercube corner number is significantly larger than the data point number. 

2). The issue of high-dimensional contrastive embedding pointed out in this paper is important. And I think this contribution is even more valuable than the proposed method itself. Nowadays, it is actually a common issue that the contrastive embedding has very high dimensionality which may lead to the curse of dimensionality. However, as the authors reviewed in the introduction, when we simply reduce the dimensionality of the output layer, it will yet cause the insufficient instance discrimination and harm the generalizability. So this phenomenon indeed leads to an awkward dilemma for us.

3). The proposed idea is non-trivial and concise, and it is well supported by the theoretical analyses. This paper proposes using the additional sparse-regularized layer to reconstruct the high-dimensional features, which I think can well solve the dilemma I mentioned above. More importantly, this idea is supported by the theoretical analyses in Theorem 1, 2, and 3, where the lower-bound of the distance contrast is explicitly given so that the similarity and dissimilarity between pairwise instances can be distinguished from each other.

4). The proposed method is built and implemented in a rather generic way. The authors conduct extensive experiments on multiple domains of data, and the improvements are also non-trivial. I find that the authors further supply the comparison experiments with the distillation-based methods. In the results, the proposed method can consistently outperform the distillation-based methods, and it seems that this can also demonstrate that why we need the new reconstruction layer for solving the issue of high-dimensional contrastive embedding.

**Weaknesses:**

1). As Theorem 1 is an existing work, the author should introduce the related preliminary of Eq. (5) more carefully. For example, does the convergence condition $\lim_{H\rightarrow\infty} var[D_{\boldsymbol{\varPhi}}(x, x_i)/E(D_{\boldsymbol{\varPhi}}(x, x_i))]$ have any relationships with the final conclusion of the distance contrast? This is not very clear to me.

2). It seems that the authors only conduct experiments on the regular positive-negative CL baselines. Does the proposed method work well with the negative-free CL approaches? As we know, currently the negative-free CL has also show encouraging performance which is even better than the positive-negative CL in some cases.

**In summary:**

By considering the above pros and cons, I would believe this is a strong paper in both theories and experiments. I think the quality of paper undoubtedly reaches the acceptance bar of NeurIPS, so I would like to vote for a “clearly accept” and give a score 8 for it. In the rebuttal, I want the authors to further clarify and solve my concerns that I mentioned in the weaknesses and questions.


questions: I have some more questions regarding the details of this work.

1).  What is the dimensionality of the finally used embedding for fine-tuning (with softmax)? It seems that the reconstructed features are still high-dimensional. If they use the low-dimensional latent features, this should be clarified if this paper is accepted.

2). I find that the authors omit some details in Eq. (3). It is unclear to us that why $\lim_{H\rightarrow\infty}(\pi^{H/2}/(H\cdot\Gamma(H/2)))/2^{H-1}\leq\lim_{H\rightarrow\infty}\pi^{(H-1)/2}/2^{H-1}$ can hold. I think this is critically import to the motivation, so please try to explain this more carefully.

3). In the optimization section, the authors use the classical SGD for model training. Is that possible if we try to use some more popular optimizers such as Adam to learning the objectives? Will this have any impact on the loss function $\mathcal{L}_{NCE}(\boldsymbol{\varPhi})$ and the reguarlizer $\mathcal{R}(\boldsymbol{\varPhi},\boldsymbol{L})$?

4). It seems that the concrete form of the lower-bound $\alpha\lambda C(X)$ is not described in the manuscript. I checked the mathematical proof in the appendix, and found that this lower-bound contains the learning parameter $\boldsymbol{L}$ and $\boldsymbol{\varPhi}$. In this case, can we still say that the lower-bound $C(X)$ only depends on the training data $X$?


limitations: I think there is no potentially negative social impact of this paper. For improvement suggestions, please see the above strengths-weaknesses and questions.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel regularization method for contrastive representation learning. The main idea is to introduce an additional layer that projects the representation to a lower-dimensional space. The additional projection layer is trained by minimizing the reconstruction loss where sparse regularization is applied. The projected (relatively) lower-dimensional representation is shown to be more effective in multiple downstream tasks.
strengths_and_weaknesses: I vote for rejection mainly due to the lack of clarity and the lack of good explanation. Meanwhile, I do find the experimental results compelling.

Strengths:

* Contrastive representation learning is an important field of machine learning and is a relevant topic.
* The proposed algorithm seems novel.
* The experiments are fairly extensive and the results are impressive. The proposed method is tested across multiple tasks, including image classification, text classification, and reinforcement learning, showing consistent improvements. Also, some of the results are provided with confidence intervals from repeated runs.
* I appreciate the significance of the empirical finding in this work.


Weaknesses:
* There is a large room for improvement in clarity. The paper needs to clearly define what contrastive learning is and what is not. Even though there is no single definition that is accepted across the whole community, at least the authors need to clarify what class of algorithm they are focusing on. This unclarity raises several questions during reading. Is the representation space constrained to a hypersphere or not? Does it assume a particular type of objective functions to be regarded as "contrastive learning"? Is there any reason that the proposed method is not applicable to non-contrastive classifiers? I will post this questions again on "Questions" section of the review.
* I am not persuaded by the argument made in Section 2.1 Motivation in general. As far as I know, many contrastive learning algorithms assume a hyperspherical latent space, so there is no point of comparing the volumes of a high-dimensional cube and a sphere. 
* The argument from Theorem 1 is only indirectly relevant at best. In classification, what matters is the ordering of distances, not the exact values of distances. There is not direct evidence that high dimensionality hurts the quality of representation.
questions: * Is the representation space constrained to a hypersphere or not? 
* Does it assume a particular type of objective functions to be regarded as "contrastive learning"? 
* Is there any reason that the proposed method is not applicable to non-contrastive classifiers?
* It would be informative if we can separate the contributions from reconstruction loss and sparsity regularization. What if we use reconstruction loss but without sparse regularization? There is no row in Table 1 corresponding to this setting.
limitations: I don't see any critical limitation of this work regarding negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a framework (CLLR) to learn embeddings through contrastive learning (CL). The proposed approach learns a low dimensional projection of the feature space used for CL. Furthermore, theoretical bound is provided for CLLR along with empirical explorations using the learned low-dimensional embedding.
strengths_and_weaknesses: Originality :

The work is somewhat original. The authors combine well known techniques in manner that produces a useful framework for producing low dimensional features through CL

Quality :

This work proposes a framework that is decently analyzed through theoretical bounds and empirical explorations using multiple diverse tasks. The authors also provide ablation studies to strengthen the proposed framework. However, a central question still remains. Why is it that the low dimensional embedding cannot be learned directly through CL. The proposed regularizations can also be applied to the final matrix that produces the features. The need for learning the CL based embeddings followed by a projection layer seems to add complexity that perhaps could have been avoided.


Clarity:

This work follows a structured approach and explains the procedure in a clear manner.

Significance :

The concept of low dimensional embeddings learned through CL is very useful. This work provides a useful step in that direction.
questions: As mentioned earlier, a central question remains regarding the whole approach. The low dimensional embeddings can directly be learned through CL and the proposed regularizations. Therefore, learning embeddings through CL followed by a projection layer seems like added complexity. Perhaps an explanation on why the low dimensional embeddings cannot be learned directly might be useful.

In the empirical explorations, were the models trained using both CL and the downstream task based loss (like multi-task learning) ?
limitations: The authors do touch upon limitations of their approach where others approaches can potentially work better.
review_title: null

Meta-review: metareview: This paper proposes a new  plug-in style method for contrastive self-supervised learning to yield low-dimensional embeddling. The authors argue that high-dimensional embeeding induced by negative pair-based constrastive learning leads to "curse of dimensionality", which might harm the generalization performance. So, the authors employs a sparse project layer on the top of typical CSSL encoder for low-dimensional representation. 

The scores of the reviewers are split: two strong acceptance (8 and 7) and two rejection (3 and 4). 
Unfortunately, the overall situation does not changes (8, 6->7, 4->3, 3->4) after rebuttal and two discussion periods.

AC carefully read the paper, the author rebuttal, and discussion comments. 

All reviews agree that this paper has novelty, interesting idea, theoretical justification, and extensive-promising results. 
The main controversal point is the motivation that "curse of dimensionality" by high-dimensional representation harms the generalization performance. On this issue, three reviewers debated very enthusiastically. It is difficult to decide whether the performance degradation of high-dimensional embedding is meaningful or not. Also, it is not very clear that this degradation is directly caused by the curse of dimensionality. However, empirical results show the proposed low-dimensional method consistently improves the high-dimensional method with sparse regularization including negarive-free CL methods. Practically, low-dimensional embedding has advantages in memory footprint and inference time in the real-world applications, which these merits are not discussed. 

Overall, this paper can contrubte to ML community despite its controversial motivation. So AC recommedns accepting this paer.
Please clarify the motivation considering two reviewers' comments.


confidence: Less certain
Decision: Accept