Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on how to fine-tune Contrastive Self-supervised Learning (CSL) models. In particular, it proposes a new fine-tuning method based on a hard-pair mining strategy. The authors provide a theoretical analysis to justify the benefits of supervised contrastive loss and then demonstrate empirically. 

main_review: Strength:
1. This paper focuses on a real and practical problem in Self-supervised Learning.
2. This paper shows that contrastive loss has regularization effect on representational learning and can improve the optimization theoretically. 
3. The paper is well-written and easy to follow, and the authors have done a good job of empirically analyzing the effect of proposed fine-tuning strategy in a number of different scenarios. 

Weakness:
1. In section 4.1, the paper states that the generated negative pairs are semi-hard because the hard negatives may cause performance degradation, but there is little analysis about the choice given here, either empirical or theoretical. 
2. Recent SSL work, e.g. BYOL[1], achieves SOTA performance without using any negative samples. I wonder whether the proposed method can be used for fine-tuning such SSL models. 

[1]G, J.-B. et al. Bootstrap your own latent - a new approach to self-supervised learning, NeurIPS, 2020.


limitations_and_societal_impact: The authors states that the potential limitation of the proposed methods is on the limited application range.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a fine-tuning method called core-tuning for Contrastive Self-supervised Learning (CSL). Core-tuning employs contrastive loss as a regularizer for preserving intra-class features and further improves the finetuned performance. Besides, hard sample mining, focal loss, and mixup are integrated to further boost the performance. The contributions are 1. explore the fine-tuning problem for CSL. 2. provide the theoretical analysis 3. good experiment result.
main_review: Originality: The novelty of this work is limited. It is more like a simple extension of several existing techniques: there are existing works that have explored adding contrastive loss as a regularization in NLP [1]. Hard sample mining is also explored in many contrastive learning works [2][3], [2] also employs mix-up for generating hard samples. 

Quality: The overall quality is good. There are many experiments supporting the claim.

Clarity: This work is well written. The descriptions are overall clear and easy to understand. 

Significance: The significance of this work is limited: it involves heavy hyperparameters tuning, which limits its practicability. (Would explain in detail in the Limitations section).

Citation:
[1] Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning; ICLR 2021
[2] Hard Negative Mixing for Contrastive Learning; Neurips 2020
[3] Contrastive learning with hard negative samples; ICLR 2021
limitations_and_societal_impact: 1. Limited novelty. As explained in the Main Review.

2. Heavy hyperparameters tuning: The proposed approach involves many hyperparameters ($\eta$, $\alpha$, $\lambda_n$). Among them, $\eta$ and $\alpha$ are not consistent among datasets and requires grid search for EACH dataset. Besides, a different set of ($\eta$ and $\alpha$) may lead to different optimized learning rate, which makes the hyper-parameters even harder to tune. This greatly limits the practicability of this work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposed a new problem: how to fine-tune a contrastive pretrained model, and proposed hard example minining and smooth boundary to help contrastive fine-tuning. The proposed Core-tuning method has been theoretically and empirically validated on a variety of tasks.
main_review: Overall, I like this paper, which is well-written and the reading is quite smooth. 

On the positive side, the theoretical bounds are both reasonable and give insights to the problem. The proposed methods have also brought about significant accuracy improvements. Ablation studies are extensive, and its application to semantic segmentation, robustness, and domain adaptation are also interesting.

Table 1 shows that supervised pretrained model indeed fine-tunes better than self-supervised models. Some earlier self-supervised learning methods have used misleading setups to lead to a false claim. It is useful that the author(s) pointed this out in Table 1 and Line 243.

But I also have some (relatively) minor issues to be clarified by the author(s).

First, Core-tuning is designed for contrastive methods. However, if we ignore the theoretical side, details of the algorithm did not depend on the InfoNCE loss. Hence, I wonder if Core-tuning is applied to BYOL, will it still help? If Core-tuning works for BYOL and other non-contrastive self-supervised methods, it will be more useufl.

Second, in Line 115 and the pseudo code (in appendix), it is not clear whether the features are extracted by Ge and fixed? In other words, will the feature learning part (backbone network) be updated in the fine-tuning?

Third, in Line 181, what is an anchor? It is not defined.

Fourth, the difference between the last 2 rows in Table 2 is small (but they are indeed consistent). So, some ablation studies are missing. What if we use supervised pretrained model + mixup + CE as a baseline (standard mixup, not manifold)? Our experience tells us that mixup + CE is signfiicantly better than CE alone in fine-tuning (both supervised and self-supervised) pretrained models.

Fifth, Table 4 is not that diverse -- they are mostly 1 architecture. In fact, it is more interesting to see results on different models like MobileNet (compact), EfficientNet (accuate), etc.
limitations_and_societal_impact: There are discussions of main limitations. The BYOL type is a good plus.
review_title: null

Meta-review: This paper investigates ways to improve the training of self-supervised models on downstream tasks by adding a contrastive loss to the fine-tuning process as well as augmenting this loss with methods for hard negative and hard positive pair mining. The authors show theoretically that a supervised contrastive loss benefits performance and show empirically that this method results in improved performance on a number of downstream datasets. Reviewers agreed that the clarity and quality of the work was high, though there were some concerns regarding the novelty of the proposed approach. While I agree with reviewers that this paper combines several previously known innovations rather than introducing a completely novel approach, I do not think this will limit the impact of the paper, as there is substantial value in combining previously independent observations. As such, I recommend the paper be accepted as a poster. 
Decision: Accept (Poster)