Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: == Summary ==

The paper proposes to use a contrastive cross-entropy loss during fine-tuning, for improving transfer accuracy of both supervised and unsupervised pre-training methods in image classification. The paper builds on the intuition that both class discriminative information and intrinsic structure of the downstream task are useful for fine-tuning, and existing fine-tuning approaches only use the former. The authors conduct experiments on four image classification datasets, using a modern ResNet-50 architecture.

== Pros ==

- The authors use two contrastive losses during fine-tuning, which has not been deeply explored, since most works typically use only class cross-entropy loss. One contrastive loss acts on the classification head, while the second, an extension of InfoNCE, acts on a projection head (on top of the representation layer).

- The proposed method achieves very good results across the four image classification datasets (CUB, Cars, Aircraft and a custom version of MS-COCO), using both supervised and unsupervised pre-training. The authors perform 5 runs for each experiment and report the average as well as statistical significance metrics (although it's not clear if the provided interval is standard deviation or a confidence interval). 

- When using unsupervised pre-training, the authors explored 6 different pre-training algorithms, and Bi-tuning improves the standard supervised fine-tuning in all cases.

- The authors also show that their method outperforms 4 baselines: standard (supervised) fine-tuning, BSS, DELTA, and L2SP by a significant margin across different data sizes available for fine-tuning (25%, 50%, 75% and 100% of the original dataset size).

- Despite the fact that they have three terms in the final loss, no additional hyper-parameter needs to be introduced (in addition to choosing the number of keys, which depends on the amount of training data).

== Cons ==

- Despite the fact that contrastive losses have not been widely used for fine-tuning (as far as this reviewer is aware), the authors should probably tone-down statements such as "Bi-tuning, a general learning framework to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks". There's a plethora of works using multiple loss/regularization terms during fine-tuning, such as DELTA or BSS, which the authors compare to. The proposed work only proposes a different type of loss/regularization. One could even argue that "Bi-tuning" is only a particular instance of multi-task learning. 

- The authors could have run additional experiments with other modern deep neural network architectures alternative to the ResNet50 (Inception, EfficientNet, DenseNet, AmoebaNet, etc.) to show whether the benefits transfer to other architectures.

- The fact that the authors coined the method "Bi-tuning" but the loss has three terms (CE, CCE and CCL) is confusing. Probably, the "Bi" is due to the two heads, but still.

== Typos ==

The paper contains several typos and some parts are not clear or could be improved, please read it carefully, correct them. Some typos that I've found.

- Not a typo, but you should indicate whether intervals in Tables are +/- std. deviation or confidence intervals.
- Not a typo, but there are much earlier works showing that fine-tuning (usually) performs better than training from-scratch. In the introduction you cite He et al. (2019). A quick search on the Internet yields a survey from 2009 on the topic, for example.
- Equation 1, bold q, k_+ and k_i, since these denote vectors.
- Figure 1, missing h_0^k (according to equation 3, j starts at 0).
- "Previously, we propose" -> "Previously, we proposed".
- "As shown in 1" -> "As shown in Table 1".
- "Results in Table 2 reveal that Bi-tuning brings general gains for all tasks, even provided with sufficient training data". I don't understand the phrase after the comma.
- "SimClR" -> "SimCLR" in Table 4.
- The result of Bi-training with MoCov2 (Table 4) is not statistically significant than the supervised baseline, thus it should not be bolded. 

== Reasons for score ==

The authors show that the proposed method achieves significantly better results than strong baselines across multiple datasets, data set sizes, pre-training paradigms (supervised vs. unsupervised) and algorithms. Despite introducing an additional head and two additional losses during fine-tuning, the proposed algorithm does not introduce more hyper-parameters than other alternatives, which makes it easier to be applied by other researchers. This reviewer believes that some claims are a bit overstated (e.g. claiming that the proposed approach is a "framework"), but there's no doubt that the proposed algorithm achieves excellent results and the experimental work is solid. 
review_title: Great work using contrastive losses during fine-tuning

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary

The paper suggests to extend fine-tuning of pre-trained representations by adding additional contrastive losses to leverage the intrinsic structure of the downstream training data. The authors call the presented method Bi-tuning and evaluate it by comparing it to other variants of fine-tuning.

### Quality, clarity, originality and significance

The paper describes its motivation well, and Sections 1 and 2 are well-written and clear. Sections 3, 4, and especially 5, I found less clear and in some parts a bit confusing.

One point that I found not entirely clear in the motivation is the focus on the conjunction of *fine-tuning* and *supervised contrastive learning*. Others have looked at the latter for general training, e.g. [arXiv:2004.11362](https://arxiv.org/pdf/2004.11362.pdf) or [arXiv:1808.04699](https://arxiv.org/pdf/1808.04699.pdf), so it is not clear to me why the Bi-tuning loss is restricted (in this study) to fine-tuning only. If the loss is universally applicable, shouldn't it also work for training in general? Or if there is a reason to not expect this (i.e. to expect that it should only work well with pre-training) this should be explained (or the explanation emphasized if I missed it). If the focus is on bringing contrastive learning to the fine-tuning stage, then maybe an inclusion of some more "plain" contrastive  losses on the way to the "novel" loss introduced here may lead to a better understanding, focusing specifically on the fine-tuning stage.

The experimental results look good at first glance, but overall I found it hard to evaluate how convincing they are because of several potential problems:
* The experimental results are not compared to any results from the existing literature, all baselines and comparisons are entirely from this paper. This makes it hard to know how much careful tuning went into the proposed algorithm versus the baselines. I think at least *some* comparison to results from the literature should be possible or it should be carefully explained why such an overfitting to the proposed algorithm clearly did not happen. For example, the [SpotTune paper](https://www.cs.utexas.edu/~grauman/papers/CVPR19_spottune.pdf) contains numbers for CUBS and Cars that look comparable at first glance because they also use pre-trained ResNet-50 models, and they include numbers for L2-SP that are better than the ones in this paper. It may be very insightful to compare results to these or other similar results. There seems to be some overlap with methods/datasets to (Chen et al., 2019), but then again there are differences like the percentages of data samples.
* There are a lot of seemingly random choices in the selection of the datasets, e.g. why CUB/Cars/Aircraft and not e.g. CIFAR, Dogs, Pets, Flowers, or Birds? I'm not saying any of these choices are inherently better, but the authors should explain *why* these datasets are chosen, to avoid the impression that the datasets may have been chosen because the proposed method works particularly well on these dataset combinations. One way to make the results stronger would therefore be to use combinations of datasets that other, previous work has already used, which would also enable a direct comparison (see above).
* Again in a similar fashion, Table 4 evaluates only one target data set, why only this one?

### Pros and cons
* Pros: interesting exploration of fine-tuning with added contrastive losses
* Cons: experimental results not very convincing, main message is not very clear

### Minor details and comments
* Novelty of the presented components seemed a bit overstated to me in some parts. E.g. words like "improved", "novel", "newly-designed" are used throughout the work when it is not entirely clear (at least to me) where exactly the novelty lies. 
* " the best learning rate is selected by cross-validation under a 100% sampling rate and applied to all four sampling rates." - how exactly is the cross-validation performed here?
* Sec. 5.1. does not seem to specify the pretraining dataset. I assume it's ImageNet, but giving some specifics (e.g. was a particular publicly available checkpoint used) might be helpful.
* "Previous fine-tuning methods mainly focus on improving performance under low-data regime paradigms." It seems recently there are several works using fine-tuning to ImageNet, which may count as large-scale, too? E.g. [PapersWithCode-ImageNet](https://paperswithcode.com/sota/image-classification-on-imagenet) has several approaches near the top-performing methods that would fall into that category I think.
* Figure 2(b): performance seems not yet saturated going to higher dimensionality - would it make sense to go even higher?
* Figure 3 is very hard to interpret I think and it is only explained extremely briefly by citing a reference. Why is the visualization uniform in case (d)? What does the visualization show? Which model and data were used? Also the conclusion "Bi-tuning in 3(e) captures both local details and global category-structures" is not evident from the figure alone, I think it may need more explanation and interpretation.

review_title: Using contrastive losses may improve fine-tuning

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The authors propose to augment the regular fine-tuning stage by two additional loss: a contrastive cross-entropy loss (L_CCE) and a categorical contrastive learning loss (L_CCL).  For each input example, L_CCE minimizes the relative distance between its representation and the softmax linear weight (compared to the distance between another image from the same class and the linear weight). L_CCl encourages representations from the same class to stay close with each other. Empirical results on CUB, Cars, Aircraft and COCO-70 show the advantage of the method. 

Strength:
1. Exploring the intrinsic structure of the downstream task is an interesting direction to improve transfer learning performance.
2. The proposed method shows consistent improvements across various datasets.

Weakness:
The additional loss terms are not well-motivated and difficulty to justify. How do they help the model exploit the intrinsic data structure?
  a. In Eq3, L_CCE computes the log ratio between 
        {the distance between the representation h_{i}^{q} and the weight of the true class w_{y_i}}
      and 
        the sum of {the distance between the positive key set h_{j}^{k} and the weight of the true class w_{y_i}}
  What does this log ratio term represent? The numerator can be bigger than the denominator in this case (and then the log term will be negative). Also if you just want to encourage instances from the same class to stay closer in the representation space, why don't you just explicitly minimize the L2 distance between the representation and the prototype?
  b. The goal of L_CCL is to encourage examples from the same class to stay closer with each other. Since the positive and negative sets are constructed using the label information, I don't see how it is complimentary to the original cross entropy loss. Also, in the case where the size of the positive set |S_i| is bigger than one, equation 4 may not make much sense as there are positive examples in the denominator.

Maybe I missed or misunderstood something, but based on these questions and concerns, I found it hard to justify the improvement.
review_title: good empirical results, unclear intuition and justification

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a general Bi-tuning approach to fine-tuning both supervised and unsupervised representations to downstream tasks. The main contribution of this paper is integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss and a projector head with a newly-designed categorical contrastive learning loss. Results are shown that Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins.

Pros: 

+ Overall, the paper is well written. In particular, the Introduction section has a nice flow and puts the proposed method into context. Despite the method having limited novelty (only improve the loss function on the basis of contrastive learning), the method has been well-motivated by pointing out the limitations in SOTA methods. 

+ The idea of designing a novel categorical contrastive loss $L_{CCL}$ on the projector head, by expanding the scope of positive keys to a set of instances instead of a single one, it is very interesting.

+ The results section is well structured. It's nice to see analysis on components of contrastive learning and the collaborative effect of the loss functions. 

 
Cons: 

- The key concern about the paper is the lack of enough novelty. It seems that the core idea of Bi-tuning is the last overall loss function: $L_{CE} + L_{CCE} + L_{CCL}$, but the author stated that "the naive combination of the supervised cross-entropy loss and the unsupervised contrastive loss is not an optimal solution for fine-tuning" in section 4.4. In my opinion, $L_{CE}$ and $L_{CCE}$ are from the classifier head, they are supervised cross-entropy loss, while $L_{CCL}$ is from the projector head, it is an unsupervised contrastive loss. Isn't adding these three values a "naive combination"? If the core idea is only improved the loss function, I think such novelty is limited.

- Considering the comprehensive experiments, a deeper analysis of the proposed method would have been nice. The idea of novel categorical contrastive loss $L_{CCL}$ on the projector head, expanding the scope of positive keys to a set of instances, what kind of scope? What effect will the scope change have on the result? The author stated that "resulting in more harmonious cooperation between the supervised and unsupervised learning mechanisms", some form of theoretical/analytical reasoning behind the effectiveness would provide greater insights into the community and facilitate further research in this direction. 

- The claim of the 10.7\% absolute rise on CUB with a sampling rate of 25\% mentioned in the abstract has not been clearly addressed and pointed out in the results section of the paper. From Table 1, I don't know how to calculate the 10.7\% **absolute rise**. Do you actually mean the **relative rise** including *the margin of error*?
 
- Fig 3: Why interpretable? The author stated that "The visualization shows that MoCo focuses only on local details.", I don't understand how Fig 3(d) expresses local features. What's more, Fig 3 was quoted many times in the paper, such as sections 2.3 and 5.2, but I really don't understand the connection with Fig 3.
 
- In section 5.3, the author stated "It is worth mentioning that CCE and CCL can work independently of CE", why not set up experiments for CCE and CCL is independent?

 
Minor comments: 

* In Section 5.2, "As reported in Figure 3", the link of Fig 3 is an error.

* The labels of the figures and tables in the paper should preferably appear and quote in order.
review_title: Official Blind Review #2

Meta-review: The authors propose an alternative fine-tuning procedure by introducing a projection head and two new losses to be combined with the vanilla cross-entropy loss. The authors introduce and jointly optimize the standard cross-entropy loss, the contrastive cross-entropy loss for classifier head and the categorical contrastive learning loss for projector head in an end-to-end fashion. The authors empirically confirmed that this setup compares favorably to existing baselines.

The reviewers found the setting challenging and worth investigating. The idea of exploring the intrinsic structure of the downstream task to help with fine-tuning was deemed useful. The reviewers appreciated the thorough empirical validation. While the proposed approach was not yet explored in this specific context, most reviewers were concerned with the lack of novelty. In addition, there seems to be a large gap between the quality of exposition in the introduction and results section with respect to the rest of the paper which introduces confusion. 

As it currently stands, the paper is not yet ready for publication and I will recommend rejection. To improve the manuscript the authors should incorporate the received feedback and significantly improve the exposition and justification of the proposed loss. In terms of empirical results, the authors should also explore alternative neural architectures to validate whether the proposed approach is general and whether the need for hyperparameter tuning arises.

Decision: Reject