Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proves inaccurate label assignment heavily  impairs the generalization of contrastive learning, and then proposes self-labeling refinery (SLR) to  generate accurate labels and momentum mixup (MM) to enhance similarity between query and its positive. 
main_review: This paper proves inaccurate label assignment heavily impairs the generalization of contrastive learning, and then proposes self-labeling refinery (SLR) to  generate accurate labels and momentum mixup (MM) to enhance similarity between query and its positive. I didn't check the proof, but the idea and the methodology looks good for me.
The major limitations in my opinion:
1) The issue of inaccurate label assignment exists but the severity of this issue is arguable. The proportion of negative samples with the same semantic label as query is very small, given the large batch size.
2) Experimentally the proposed method outperforms CLSA (Moco with strong augmentation) marginally (0.2 - 0.4 %) in Table 2.
3) I don't understand why p and q are sum-to-one vectors. It is possible that there're multiple samples with the same label as query.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper argues that the one-hot label for instance discriminative contrastive loss will impair the performance in downstream tasks. By assuming ground-truth soft labels, the authors prove that the generalization of the network depends on the discrepancy between ground-truth soft labels and refined labels. Therefore, two approaches, SLR and MM are proposed to address the issue. Empirically, the proposed approach outperforms the baselines by non-trivial margins.
main_review: Over all, the paper is well written and easy to follow. The motivation is clear and supported theoretical analysis. The proposed approach seems to be simple yet effective. It is good to see paper include theoretical analysis beyond intuitive motivation, Although I found theorem 2 a bit hard to follow. The experiment results are impressive, achieving state-of-the-art in various benchmarks including transfer learning.

Here are a few questions and suggestions:

1. In Theorem 1, shall we add an assumption that the loss function $\ell$ is smooth? As for non-smooth function, the norm of Jacobian might be ill-defined or not be bounded by the Lipschitz constant (the proof in Appendix).

2. In equation (5), what is the main reason to keep both $p$ and $q$? As line 190 states, $p$ could be very concentrated, which makes itself similar to the role of one-hot label $y$. It is actually interesting to see including both $p$ and $q$ improves the performance in Table 4. I try to go over the proofs of Theorem 2 in appendix to investigate the role of $p$, would you mind briefly explain the role of $p$ in theoretical analysis?

3. Theorem 1 bounds the difference between risks with different label distributions. Nevertheless, it is a bit hard to tell which label distribution benefits the downstream tasks. For instance, [1] shows that supervised loss is upper bounded by the contrastive loss under specific assumptions. If one can show the risk with ground-truth smooth labels is a tighter bound or a better estimator of the unknown supervised risk, it would make the big picture more clear.

4. Other contrastive frameworks such as SimCLR also leverage negative samples to learn representations. Can SANE be extended to SimCLR? It seems that the only difference is that there could be more dynamics for the ket set $B$ which makes it hard to be theoretically analyzed. But I hypothesize that SANE can also improve the performance of other contrastive learning frameworks that base on info-NCE loss.

Overall, I think this is a good paper with both theoretical and empirical analysis. I did not observe obvious flaw in the paper.



[1] Arora et al., A Theoretical Analysis of Contrastive Unsupervised Representation Learning, 2019.

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses why inaccurate one-hot label in contrastive learning cannot reveal well semantic similarities between samples, and how these inaccurate label assignment would impair its generalization for semantic instance discrimination. 

To address this issue, a novel self-labeling refinement method is proposed to obtain more accurate labels for contrastive learning. This is done via two complementary modules that respectively generates accurate soft-labels and enhances semantic similarity between queries and their positives during training.

Both theoretical analysis and empirical results are provided to support the claims made in this paper.
main_review: ### Originality
The theoretical analysis shows that one should provide accurate labels close to the soft true labels to better capture the underlying semantic similarity between queries and different samples. This is an interesting discovery and it motivates the authors to develop a new self-labeling refinement method that includes soft labels and mixup. Although both of these two techniques are not new, the authors modify each of them in the framework of contrastive learning and make them complementary to each other.

### Quality
* **Theorem-1** shows that the generalization error of MoCo depends on the discrepancy between the assigned labels and the true labels that really reflect the semantical similarity. While this theorem is of vital importance for this paper, the demonstration of it is completely moved to the supplementary materials, which makes the paper not self-content.
* **Theorem-2** leads to several important findings of the method, however, it is difficult to understand the intuitions and mathematical proofs behind it due to the complicated  equations and abrupt transitions between them.
* **Robustness Analysis** is sort of missing for the five hyper-parameters of the proposed method. Regarding the choice for each dataset described in L334-337, more ablation studies should be included for a better understanding of how the proposed method is sensitive to these hyper-parameters.
* **Baseline Performance** is a bit confusing in Table-4 and Table-5. As shown in Eq-9, the SANE model combines the vanilla contrastive loss with weight (1-$\lambda$) and the the momentum mixup loss with weight $\lambda$, the first entry of Table-5 should be the same as the first entry of Table-4, why they are different?
* **Semi-Supervised** setting comparison could be added. Although the authors claim that they focus on self-supervised learning in this paper, a comparison in the semi-supervised setting, e.g., use 1% or 10% labels as done in SwAV, could be interesting and might help to understand better the proposed methods that leverage soft labels.

### Clarity
The motivation of this paper is clear, but the theoretical analysis provided in the main paper is not self-content. Key assumptions and theorems come out suddenly without a smooth roadmap, and the mathematical notations are over-complex (see L232-256). I'd recommend to reshape the Section-3 to make it easier to follow and more understandable.

Discussion of related works seem also a bit rushed, and extend it with more self-supervised learning methods considering label noises could be welcome.

### Significance

While the contrastive loss such as infoNCE become a popular choice in many self-supervised learning methods, the authors discuss why the one-hot label assignment commonly used in these methods could not reflect the true semantic similarity between different samples. This is an important problem and resolving it could benefit many contrastive-based methods as well as the semantic-related downstream tasks.

limitations_and_societal_impact: The limitations and societal impact has been well discussed in the paper.
review_title: null

Meta-review: The reviewers are unanimously positive about the paper. Relevant question, good analysis and well-justified remedy.
Decision: Accept (Spotlight)