Bi-tuning of Pre-trained Representations
Authors: Jincheng Zhong
Ximei Wang
Zhi Kou
Jianmin Wang
Mingsheng Long
Venue:  (2021)
Keywords: Deep learning
fine-tuning
pre-training

Abstract: It is common within the deep learning community to first pre-train a deep neural network from a large-scale dataset and then fine-tune the pre-trained model to a specific downstream task. Recently, both supervised and unsupervised pre-training approaches to learning representations have achieved remarkable advances, which exploit the discriminative knowledge of labels and the intrinsic structure of data, respectively. It follows natural intuition that both discriminative knowledge and intrinsic structure of the downstream task can be useful for fine-tuning, however, existing fine-tuning methods mainly leverage the former and discard the latter. A question arises: How to fully explore the intrinsic structure of data for boosting fine-tuning? In this paper, we propose Bi-tuning, a general learning framework to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss to better leverage the label information in an instance-contrast way, and a projector head with a newly-designed categorical contrastive learning loss to fully exploit the intrinsic structure of data in a category-consistent way. Comprehensive experiments confirm that Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins (e.g.~10.7\% absolute rise in accuracy on CUB in low-data regime).

Under review as a conference paper at ICLR 2021
BI-TUNING OF PRE-TRAINED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
It is common within the deep learning community to first pre-Irain deep neural network from a large-scale dataset and then fine-tune the pre-trained model to a specific downstream task. Recently, both supervised and unsupervised pre-training approaches to learning representations have achieved remarkable advances, which exploit the discriminative knowledge of labels and the intrinsic structure of data, respectively. It follows natural intuition that both discriminative knowledge and intrinsic structure of the downstream task can be useful for fine-tuning, however, existing fine-tuning methods mainly leverage the former and discard the latter. A question arises: How to fully explore the intrinsic structure of data for boosting fine-tuning? In this paper, we propose Bi-tun Bi-tuning, a general learning approach to fine- tuning both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss to better leverage the label information in instance- contrast way, and a projector head with newly-designed categorical contrastive learning loss fully exploit the intrinsic structure of data in category-consisten way. Comprehensive experiments confirm that Bi- Bi-tuning achieves state-of-the -te- results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins (e.g. 10.7% absolute rise in accuracy CUB in low-data regime).
INTRODUCTION
In the last decade, remarkable advances in deep learning have been witnessed in diverse applications across many fields, such as computer vision, robotic control, and natural language processing in the presence of large-scale labeled datasets. However, in many practical scenarios, we may have only access to a small labeled dataset, making it impossible to train deep neural networks from scratch. Therefore, has become increasingly common within the deep learning community to first pre-train deep neural network from a large-scale dataset and then fine-tune the pre-trained model to a specific downstream task. Fine-tuning requires fewer labeled data, enables faster training, and usually achieves better performance than training from scratch (He et al., 2019). This two-stage style of pre-training and fine-tuning lays as the foundation of various transfer learning applications.
In the pre-training stage, there are mainly approaches to pre-train deep model: supervised pre-training and unsupervised pre-training. Recent years have witnessed the success of numerous supervised pre-trained models, e.g. ResNet (He et al. (0016). by exploiting the discriminative knowl- edge of labels on a large-scale dataset like ImageNet (Deng et al.) 2009). Meanwhile, unsupervised representation learning is recently changing the field of NLP by models pre-trained with large-scale corpus, e.g. BERT (Devlin et al., 2018) and GPT[Radion & Sutskever (2018). In computer vision, remarkable advances in unsupervised representation learning (Wu al. 2018 He et al.) 2020 Chen et al. 2020). which exploit the intrinsic structure of data by contrastive learning Hadsell et al. (2006). are also to change the field dominated chronically by supervised pre-trained representations.
In the fine-tuning stage, transferring a model from supervised pre-trained models has been empirically studied in Kornblith et al. (2019) During the past years, several sophisticated fine-tuning methods were proposed, including L2-SP Li et al. 2018), DELTA (Li et al. 2019) and BSS (Chen et al. 2019). These methods focus on leveraging the discriminative knowledge of labels by a cross-entropy loss and the implicit bias of pre- trained models by regularization term. However, the intrinsic structure of data in downstream task is generally discarded during fine-tuning Further, we empirically observed that unsupervised pre-trained representations focus more the intrinsic structure, while supervised

---

Under review as a conference paper at ICLR 2021
pre-trained representations explain better the label information (Figure B). This possibly implies that fine-tuning unsupervised pre- vreetrainng representations is may be more difficult He et al. (2020).
Regarding to the success of supervised and unsupervised pre-training approaches, it follows natural intuition that both discriminative knowledge and intrinsic structure of the downstream task can be useful for fine-tuning. A question arises: How to fully explore the intrinsic structure of data for boosting fine-tuning? To tackle this major challenge of deep learning, we propose Bi-tuning, a general learning approach to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks. Bi-tuning generalizes the vanilla fine-tuning by integrating two heads upon the backbone of pre-trained representations:
A classifier head with an improved contrastive cross-entropy loss to better leverage the label information in an instance-contrast way, which is the dual view of the vanilla cross-entropy loss and is expected to achieve a more compact intra-class structure.
A projector head with newly-designed categorical contrastive learning loss to fully exploit the intrinsic structure of data in a category-consisten way, resulting in a more harmonious cooperation between the supervised and unsupervised fine-tuning mechanisms.
As general fine-tuning approach, Bi-tuning can be applied with variety of backhones without any additional assumptions. Comprehensive experiments confirm that Bi-tuning achieves state-of-the-an results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins (e.g. 10.7% absolute rise in accuracy on CUB in low-data regime). We justify through ablation studies the effectiveness of the proposed two-heads fine-tuning architecture with their novel loss functions.
RELATED WORK
2.1 PRE-TRAINING
During the past years, supervised pre-trained models achieve impressive advances by exploiting the inductive bias of label information large-scale dataset like ImageNet (Deng et al. 2009).
such as GoogleNet (Szegedy et al. 2013), ResNet (He et al. 20166), DenseNet (Huang et al.
201 77) )
to name few. Mcanwhile, unsupervised representation learning is recently shining in the field of
NLP by models pre-trained with a large-scale corpus, including GPT (Radford & Sutskever 2018)
BERT (Devlin et al. [ 2018) and XLNet (Yang et al. (209)) Even in computer vision, recent ressiveeee advances in unsupervised representation learning (Wu et al. 2018 He et al. 2020 Chen et al. 2020).
which exploit the inductive bias of data structure, are shaking the long-term dominated status of
representations learned in a supervised way. Further, a wide range of handcrafted pretext tasks have been proposed for unsupervised representation learning, such as relative patch prediction (Doersch) et al. 2015), solving jigsaw puzzles (Noroozi & Favaro 2016), colorization (Zhang et al. 2016), etc.
2.2 CONTRASTIVE LEARNING
Specifically, various unsupervised pretext tasks are based on some forms of contrastive learning.
in which the instance discrimination approach (Wu et al. 2018 He et al.] 2020 Chen et al.) 2020)
is one of the most general forms. It is noteworthy that the spirits of contrastive learning actually
can date back very far (Becker & Hinton| 1992 Hadsell al. 2006 Gutmann & Hyvärinen| 2010).
The key idea is to maximize the likelihood of the distribution p(x D) contrasting to the artificial noise distribution Pra (x), also known as poise-contrastive estimation (NCE). Later, Goodffllow et al. 2014) pointed out the relations between generative adversarial networks and noise-contrastive estimation. Meanwhile, (van den Oord et al. 2018) revealed that contrastive learning is related to mutual information between query and the corresponding positive key, which is known InfoNCE.
Other variants of contrastive learning methods include contrastive predictive learning (CPC) (van den Oord et al. 2018) and colorization contrasting (Tian al. 2019). Recent advances of deep contrastive learning benefit from contrasting positive keys against very large number of negative keys. Therefore, how to efficiently generate keys becomes fundamental problem in contrastive learning. To achieve this goal, Doersch & Zisserman (2017) explored the effectiveness of in-batch samples, Wuu et al. (2018) proposed to memory bank to store all representations of the dataset, He et al. (020)
2

---

Under review as a conference paper at ICLR 2021
further replaced memory bank with the momentum contrast (MoCo) to be memory-efficient, and Chen et al. (2020) showed that a brute-force huge batch of keys works well.
2.3 FINE-TUNING
Fine-tuning model from supervised pre-trane rre-trained models has been empirically explored in Kornblith et al.] (2019) by launching a systematic investigation with grid search of the hyper syper-parameters. During the past years, a few fine-tuning methods have been proposed to exploit the inductive bias of pre-
trained models: L2-SP (Li et al. 2018) drives the weight parameters of target task to the pre-trained values by imposing 1.2 constraint based the inductive bias of parameter; DELTA (Li al. 2019)
computes channel-wise discriminative knowledge to reweight the feature map regu larization with an attention mechanism based on the inductive bias behavior; BSS (Chen et al.) 2019) penalizes smaller singular values to suppress untransferable spectral components based on singular values.
Other fine-tuning methods including learning with similarity preserving (Kang et al., 2019) or learning without forgetting (Li & Hoiem, 2017) also work well on some downstream classification tasks. However, the existing fine-tuning methods mainly focus on leveraging the knowledge of the target label with cross-entropy loss. Intuitively, encouraging model to capture the label information and intrinsic structure simultaneously may help the model transition between the upstream unsupervised models with the downstream classification tasks. In natural language processing, GPT (Radford Sutskever 2018) Radford et al.) 2019) has employed a strategy that jointly optimizes unsupervised training criteria while fine-tuning with supervision. However, we empirically found that trivially following this kind of force-combination between supervised learning loss and unsupervised contrastive learning loss beneficial but limited. The plausible reason is that these losses will contradict with each other and result in an extremely different but not discriminative feature structure compared to that of the supervised cross-entropy loss (See Figure B).
3 CONTRASTIVE LEARNING WITH MULTIPLE POSITIVE KEYS
Instance discrimination approach (van den Oord et al.
2018 Wu et al. 2018 a.k.a. InfoNCE, is one
of the most general forms of standard contrastive learning. Given a query q with a large key pool (ko, K1, k2. kx) where K is the number of keys, this non-parametric contrastive loss is
exp(q k+/T)
LInfaNCE = - log
Σx exp(a (k,xp)
where is the hyper-parameter for temperature scaling. Intuitively, contrastive learning can be defined as a query-key pair matching problem, in which a contrastive loss is K-way cross-entropy loss to distinguish k+ from large key pool. From this perspective, a contrastive loss is to maximize the similarity between the query and the corresponding positive key k4.
When fine-tuning the pre-trained model on labeled downstream dataset, intuitive way is to combine InfoNCE and cross cross-entropy as LInfANCE + LCE. However, InfoNCE tends to generate an extremely different but not discriminative feature structure compared with that of the supervised cross-entropy loss, making the classifier struggle. To this end, we proposed a novel idea of contrastive learning with multiple positive keys tailor contrast into cross-entropy loss in both the projector head and classifier head. Before describing the losses defined on these two heads in detail, let us make a brief introduction of contrastive learning with multiple positive keys.
Definition (Contrastive Learning with Multiple Positive Keys) In the context of tailoring con- trastive into fine-tuning the pre-trained model a labeled downstream task, it is the mechanism that expands the scope of positive keys to a set of instances instead of a single one.
Formally, given a query q with a large key pool {ko, k1, k2, kgk } where K is the number of keys and ko is the positive key k,. Suppose the intance number in each class is equal the key pool, i.e., the equality K - k- C' is held where C' is the size of label space and k: is the instance number in each class. The straightforward approach to expand the standard contrastive loss (LinfoNce) into a form of multi musti-positives (denoted by positive key set K2) is
exp(q k4/T)
T/ InfoNCE
log exp(q kt/r) Σx_ek, exp(q k//))
(2)
K
K+EK,
3

---

Under review as a conference paper at ICLR 2021
where K,, denotes the negative key set. We denote this loss function by L' IaIoNCE, which essentially performs multiple individual contrasts with different positive keys for each query q. However, this form of multi positive key contrastive can be seen simple generaliztion of the standard contrastive loss by repeating the positve keys several times, without fully exploiting the intrinsic structure of intra-class dataset. To this end, the losses (LoCn and TecI detailed in Section 4.3 and Section 4.4 respectively) we proposed are based on the following formula:
exp(q k4/r)
Lproposed
Σ log ExtEK, exp(q k+/T) + Ek EK,, exp(q - k//r) K,EKK
(3)
KKpl
Different from L' InfoNCE, in the denominator of Lproposed, both the positive keys in the same class with the query and the negative keys from other classes are presented. For each contrast with multiple positive keys, query here needs to balance all positive keys simultaneously. In another view, /proposed can be regarded as performing cross-entropy on soft labels of uniform probability |K₂ for each positive key. prpopseed introduces uninformative prior that the positive keys uniformly distributed around the query. Table compared the accuracies of Bi-tuning implemented with both forms of multiple-positivee and revealed that the proposed form is the better choice.
METHOD
4.1
PRE-TRAINED REPRESENTATION
Bi-tuning is general learning approach to fine-tuning both supervised and unsupervised represen- tations. Without any additional assumptions, the pre-trained feature encoder f(-) can be various network backbones according to the downstream tasks, including ResNet (He et al.) 2016) and DenseNet (Huang et al. 2017) for supervised pre-trained models, and MoCo (He et al., 2020) and SimCLR (Chen al. 2020) for unsupervised pre-trained models. Given query sample x,' we can first utilize a pre-trained feature encoder f()) to extract its pre-trained representation as h = f(x)).
4.2
VANILLA FINE-TUNING
Given pre-trained representation hς fundamental step of vanilla fine-tuning is to feedforward the representation h into C-way classifier g(-), in which C' is the number of categories for the down- stream classification task. Denote the parameters of the classifier g(-) as W [W1. W2.- we]. where W, corresponds to the parameter for the j-th class. Denote the training dataset of the down- stream task as {x{, )
W can be updated by optimizing a standard cross-entropy (CE) loss as
N
exp( Wg. h9)
LCE
log
(4)
E exp(w, h?)
4.3 CONTRASTIVE CROSS-ENTROPY LOSS ON CLASSIFIER HEAD
From another perspective, the cross-entropy loss of vanilla fine-tuning on a given dataset with N instance-class pair (x") y; can be regarded class-wise championship. i.e., the prediction that is the same as the ground-truth class of each instance is expected to win the game. To further exploit the label information of the downstream task, propose novel contrastive cross-entropy loss LECE on the classifier head via the dual view of cross- cross-entropy loss. Similarly, LOCE can be seen as an instance-wer wise championship. i.e., the prediction belongs to the nearest instance towards the prototype of each class is expected to win the game. Similar to CE loss, LCCE can be formulated as
LCCE =
1
Σ K+EK,
log Σn-EK, exp(w, h- /T) Σn-ek, exp Ww h hr/r)²
exp(w, h7)
(5)
KPp
where K, is the positive key set (including the current query example h and keys with the same label )),, and K,, is the negative set (including examples with other classes). Note that, h's are samples from the hidden key pool produced by the key generating mechanism (except h9). Though Bi-tuning is general for it, we adopt the key generating approach in Momentum Contrast (MoCo) (He et al.
4

---

Under review as a conference paper at ICLR 2021
w/h
w h
w
w
CE
Classifier g(.)
w. h
w h
w
we
x9
h9
wi
wih
wThk
wThh
k
h
Encoder f())
LCCE
C
h
t E.
E
Inputs
Representations
Projector (
Z
22
Z
LCCL
Figure The architecture of the proposed Bi-tuning approach, which includes an encoder for pre-trained representations, classifer head and projector head. Bi-tuning enables a dual fine-tuning mechanism: contrastive cross-entropy loss (CCE) on the classifier head to exploit label information and categorical contrastive learning loss (CCL) on the projector head to model the intrinsic structure.
2020) default one due to its simplicity, high-efficacy, and memory-efficine implementation. As intuitively illustrated Figure column-wised LECE operates loss computation along the bank dimension whose size is K while row-wised LeE performs along the class dimension with a size of C. By encouraging instances in the training dataset to approach towards their corresponding class prototypes, LCCE tends to achieve more compact intra-class structure than the vanilla fine-tume Ane-tuning.
4.4 CATEGORICAL CONTRASTIVE LEARNING Loss ON PROJECTOR HRAD
Previously, we proposed an improved-verse verroveddveeioonnoooo of vanilla exploit label information. However, this kind of loss function design may still fall short in capturing the intrinsic structure. Inspired by the remarkable success of unsupervised pre-training which also aims at modeling the intrinsic structure in data, we first introduce a projector @(-) which is usually off the shelf to embed a pre-trained representation h into latent metric space as z/. However, the standard contrastive learning loss (InfoNCE) defined in Eq. (1) assumes that there is single key k in the dictionary match the given query q. which implicitly requires every instance belong to individual class. If we simply apply InfoNCE loss on the labeled downstream dataset, it will result in an extremely different but not discriminative feature structure compared with that of the supervised cross-entropy loss, making the classifier struggle. Obviously, this dilemma reveals that the naive combination of the supervised cross-entropy loss and the unsupervised contrastive loss is not optimal solution for fine-tuning which is also backed by our experiments in Table 3
the classifier head to fully
To capture the label information and intrinsic structure simultancously, we propose novel categorical contrastive loss LCCL the projector head based on the following hypothesis: when fine-tune pre-trained model to a downstream task, is reasonable to regard other keys in the same class as the positive keys that the query matches. In this way, LOcI expands the scope of positive keys a of instances instead of a single one, resulting in a more harmonious cooperation between the supervised and unsupervised learning mechanisms, Similar the format of InfoNCE loss, LCCL defined as
exp(2 1xp(2..z//1 2
LOCL
Σ log ztEK,
K,
Ex-EK, exp(zx . zz -tt--==//==)) - exp 9. -zz 2/7
where the notations are identical Eq. (5), well as the positive key set. Note that, the outer sum is over all positive keys, indicating that there may be more than one positive key for a single query, i.e. the inequality that K,, | > holds.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Top-1 accuracy on various datasets using ResNet-50 by supervised pre-training.
Sampling Rates
Dataset
Method
25%
30%
75%
100%
Fine-tuning (haseline)
61.36++0.11
73.61+0.23
18.49-40018
80.74++0.15
1.2SP (Li ct al. 2018)
61.21+0.19
72.99-00.13
18.11+0.17
80.92++022
CUB
DELTA (Ta et al. 2019
62.89++0.11
74.35++0.28
79.18 19.18-+0.24
81.33++022
BSS (Chen Bi-tuning
2019
64.69+0..31
74.96+0.21
78.91=00.15
81.52++0.11
67.47.10.08
77.17110.13
81.07
82.93
Fine-tuning (bascline)
56.45+0.21
75.24=0017
83.22+0.17
86.22+:0.2
L2SP (Li et al. 2018
56.29:0021
75.62+0.32
83.60+0..3
85.85+0.12
Cars
DELTA (Li et a 2019
08.74+3.23
76.53=0008
84.53+00229
86.01+0.37
BSS (Chen el al Bi-tuning
2019
59.74110.14
76.78110.16
85.06110.13
87 87.61110.21
66.15
81.10:1..07
86.07-10.23
88.47=0.11
Fine- sonettning (baseline)
51.25=00.18
67.12=0..41
75.22=00.09
79.18100.0
L2SP (Li et 2018)
51.07+0.45
67.46-0.22
75.06=+0.45
79.07
Aircraft
DELTA (Li et al 2019
53.71+0.30
68.51=0.24
76.51+0.55
80.34+0.14
BSS (Chen et al.
2019
53.38++0.22
69.19-t0.18
76.39++0222
80.83++0.2
Bi-tuning
58.27-40.26
72.40++0.2
80.77+0.110
84.01=0.33
4.5 BI-TUNING
Finally, we reach a novel approach to fine-tuning both supervised and unsupervised representations, Lee. the Ri-tuning, which jointly optimizes the standard cross-entropy loss, the contrastive cross- entropy loss for classifier head and the categorical contrastive learning loss for projector head in an end-to-end deep architecture. Note that, Bi-tuning refers to the proposed two heads (a.k.a, bi-head) with two novel losses. The overall loss function of Bi-tuning can be formulated follows:
min LEF. + LCCE + LeCL,, è
where è denotes the set of all parameters of the backbone, the classifier head and the projector head. Specifically, since the magnitude of the above loss terms comparable, we empirically find that there is no need to introduce any extra hyper-parameters to trade-off them. This simplicity makes Bi-tuning easy to be applied different datasets or tasks. The full portrait of Bi-tuning is shown Figure
EXPERIMENTS
We follow the common fine-tuning principle described in Yosinski al. (2014), replacing the last task-specific layer in the classifier head with randomly initialized fully connected layer whose learning rate is 10 times of that for pre-trained parameters. Meanwhile, the projector head is set to be another randomly initialized fully connected layer. For the key generating mechanisms, we follow the style in He et al. (2020), employing a momentum contrast branch with a default momentum coefficient 0.999 and two cached queues both normalized by their L2-norm (Wu et al. 2018) with dimensions of 2048 and 128 respectively. For each task, the best learning rate is selected by vols-vvaliation under 100% sampling rate and applied to all four sampling rates. Queue size K is set as 8, 16, 24, 32 for each category according to the dataset scales respectively. Other hyper- parameters in Bi-tuning as 0.07 (Wu et al. 2018). The trade-off coefficients between these three losses are kept as since the magnitude of the loss terms is comparable. All tasks are optimized using SGD with momentum 0.9. All results in this sections are averaged over 5 trails and standard deviations are provided.
fixed for all experiments. The temperature in Eq. (5) and Eq. (6) is
5.1 BI-TUNING SUPERVISED PRE-TRAINED REPRESENTATION
Standard benchmarks. We first verify our approach on three fine-grained classification bench- marks: CUB-200-2011 (Welinder et al. 2010 (with 11788 images for 200 bird species), Stanford
Cars (Krause et al.) (2013) (containing 16185 images 196 classes of cars) and FGVC Aireraft
(Maji al. 2013) (containing 10000 samples 100 different aircraft variants). For each benchmark,

---

Under review as a conference paper at ICLR 2021
Table 2: Top-1 accuracy on COCO-70 dataset using DenseNet-122 by supervised pre-training.
Sampling Rates
Dataset
Method
25%
50%
75%
100%
Fine-tuning (baseline)
80.01+0.25
82.50-+0.25
83.43-+0.18
84.41-0022
1.2SP (Li et al. 2018
80.37+0.47
80.67+0.29
83. 83.71+0.24
84.78+0.166
COCO-70
DELTA (La al.
2019)
76.39+0.37
79.72++0.24
83.01+0.11
84.66=00.08
BSS (Chen 2019 Bi-tuning
77.29=0.115
80.74+0.22
83.89+0.009
84.71=0.13
80.68.10.23
83,48:10.13
84.16110.05
85.41
we create four configurations which randomly sample 25%, 50%, 75%. and 100% of training data
for each class respectively, to reveal the detailed effect while fine-tuning to different data scales. We choose recent fine-tuning technologies: 1.2-SP (Li et al., 2018), DELTA (Li et al. 2019), and the ttateoof-the-an method BSS (Chen et al., 2019). as competitors Bi-tuning while regarding vanilla fine-tuning baseline. Note that vanilla fine-tuning is strong baseline when sufficient data is provided. Results are averaged over 5 trials. As shown in Table I Bi-tuning significantly outperforms all competitors across all three benchmarks by large margins (e.g. 10.7% absolute rise CUB with sampling rate of 25%). Note that even under 100% sampling rate, Bi-tuning still outperforms others.
Large-scale benchmarks. Previous fine-tuning methods mainly focus improving performance under low-data regime paradigms. We further extend Bi-tuning to large-scale paradigms. We use annotations of COCO dataset (Lin et al. 2014) construct large-scale classification dataset, cropping object with padding for each image and removing minimal items (with height and width less than 50 pixels), resulting a large-scale dataset containing 70 classes with more than 1000 images per category. The scale is comparable to ImageNet in terms of the number of samples per class. On this constructed large-scale dataset named COCO-70, Bi-tuning also evaluated under four sampling rate configurations. Since even 25% sampling rates of COCO-70 is much larger than each benchmark Section 5.T previous fine-tuning competitors show micro contributions to these paradigms. Results in Table reveal that Bi-tuning brings general gains for all dasksbeeynne the low-data regime. We hypothesize that the intrinsic structure introduced by Bi-tuning contributes substantially.
Table 3: Top-1 accuracy on various datasets using ResNet-50 unsupervisedly pre-trained by MoCo.
Sampling Rates
Dataset
Method
25%
50%
75%
100%
Fine-tuning (baseline)
38.5710.13
58.9710.16
69.5510.11
04.35=10.18
GPT* (Radford et al. 2019)
36.43-10.17
57.62110114
67.82:10.05
72.95=0.29
CUB
Center Wen 2016
42.53-+0.41
62.15++0.51
70.86++0.39
75.61=0.33
BSS (Chen et al.
2019
41.73+t0.14
59.15++0.21
69.93:10.19
74.16=0.09
Bi-tuning
50.54+0.23
66.88=0.13
74.27+00.05
77.14+00.2
Fine-tuning (baseline)
62.40.10.26
81.55:10.36
88.0710.119
89.8110.48
GPT (Radford et al 2019
65.83+0.27
82.39+0.117
88.62#0.11
90.56=0.11 18
Cars
Center Won et 2016)
67.5740.12
82.78+00.0
88.55=0024
89.95=#011
BSS CChen et al.
2019
62.13+:.22
81.72+00.22
88.32+0.17
90.41
Bi-tuning
0..4
84.41
89.32-10.23
90.88
Fine-tuning (baseline)
58.98-10.54
77.39-L0.31
84.82-10.24
87.35
GPT (Radford et al. 2019
60.70+10.08
78.93-00.17
85.0910.100
87.56+0.15
Aireraft
Center Wen et al 2016)
62.2310.009
79.30
85.2010.41
87.52=0.20
BSS (Chen et al.
2019
60.13:0.32
77.98-0.229
84.85=0.211
87.25=0.07
Bi-tuning
63.16+0.26
79.98-0.22
86.23--0.29
88.55
5.2 BI-TUNING UNSUPERVISED PRE-TRAINED REPRESENTATIONS
Bi-tuning representations of MoCo He et al. 2020 In this round, use ResNet-50 pre-trained unsupervisedly via MoCo on ImageNet as the backbone. Since suffering from the large discrepancy between unsupervised pre-trained representations and downstream classification tasks demonstrated
7

---

Under review as a conference paper at ICLR 2021
Table 4: Top-1 Accuracy on Car dataset with different unsupervisedly pre-trained representations.
Pre-training Method
Fine-tuning (100% dataa)
Bi-tuning (100% data)
Deep Cluster (Caron et al.
2018
83.90:10.48
87.71=0.344
InsDisc (Wu cl ar.] 2018)
86.59-+0.22
89.54--0225
CMC (Tian ct al. 2019
86.71+0..2
88.35+00.44
MoCov2 (He al.
2020
90.15+t00.4
90.79++0.34
SimCLR(TX) (Chen ct a
2020
89.30+-0.1
90.84=0.22
SimCLR(2x)) Chhn el al.
2020
91.22:10.19
91.93:10.19
13:2
cib wthh 25% data
cub wtth 25%
300
2.75
Eaytounceed parches (opaae
Projectss dhrendens
(a) Number of Sampling Keys (K)
(b) Dimension of Projector Head (T.)
Figure Sensitivity analysis of haper-parameters and for Bi-tuning.
in Figure 3 previous fine-tuning competitors usually perform very poorly, Hence we only compare
Bi-tuming to the state-of-the-ar method BSS (Chen et al. 2019) and vanilla fine-tuning as baselines. Besides, we add two intuitively related baselines: (I) G which follows GPT (Radford & Sutskever 2018 Radford et al.) (019) fine-tuning style but replaces its predictive loss with the contrastive loss; (2) Center loss, which introduces compactness of intra-class variations (Wen et al.. 2016) that is effective in recognition tasks. As reported in Table 3 trivially borrowing fine-tuning strategy GPT (Radford & Sutskever (018) or center loss brings tiny benefits, and is even harmful some datasets, e.g. CUB. Bi-tuming yields consistent gains all fine-tuming tasks of unsupervised representations, indicating that Bi-tuning benefits substantially from exploring the intrinsic structure.
Bi-tuning other unsupervised pre-trained representations. To justify Bi-tuning's s general efficacy, we extend our method to unsupervised representations by other pre-trrining methods. Bi-tuning is applied to MoCo (version 2) (He et al. 2020). SimCLR (Chen et al. 2020), InsDisc (Wu et al. (2018). Deep Cluster (Caron et al. 2018). CMC (Tian et al. 2019) on Car dataset with 100% training data. Table 4 is strong signal that Bi-tuning is not bound to specific pre-training pretext tasks.
Analysis on components of contrastive learning. Recent advances in contrastive learning, i.e. momentum contrast (He al., 22200) and memory bank (Wu al. (0188) can be plugged into Bi- tuning smoothly to achieve similar performance and the detailed discussions are deferred to Appendix.
Previous works (He al. 2020 Chen et al. 2020) reveal that a large amount of contrasts is crucial
to contrastive learning. In Figure ((a) we report the sensitivity of the numbers of sampling keys in Bi-tuning (MoCo) under 25% and 100% sampling ratio configurations. Figure[(a))shee ((a))shows that though
larger key pool is beneficial, we cannot expand the key pool due to the limit of training data, which may lose sampling stochasticity during training. This result suggests that there is trade-off between stochasticity and a large number of keys. Chen et al. (2020) pointed out that the dimension of the projector also has a big impact. The sensitivity of the dimension projector head is also presented in Figure 2(b) Note that the unsupervised pre-trained model (e.g., MoCo) may provide off fff-the-shelf projector, fine-tuning or re- initializing is almost the same (90.88 vs. 90.78 on Car when is 128).
Interpretable visualization of learned representations. As visualized by Fong & Vedaldi (2017) shown in Figure 3 Note that 3(a) is the original image, Figure ()b)) Figure ((c) and Figure 3(d respectively obtained from randomly initialized model, supervised pre-trained model
8

---

Under review as a conference paper at ICLR 2021
(a) Original
(b) Random
(c) Supervised
(d) Unsupervised
(c) Bi-tuning
Figure 3: Interpretable visualization of learned representations via various training methods.
ImageNet, and an unsupervised pre-trained model via MoCovl (He et al. 2020 We infer that supervised pre-training will obtain representations focusing on the discriminative part and ignoring the background part. In contrast, unsupervised pre-training pays uninformative attention to every location of input. Bi-tuning in ((e) captures both local details and global category structures. sategory-struccures..
5.3 COLLABORATIVE EFFECT OF Loss FUNCTIONS
Using either contrastive cross-entropy (CCE) or categorical contrastive (CCL) with vanilla cross- entropy (CB) already achieves relatively good results, shown in Table These experiments prove that there is collaborative effect between CCE and CCL loss empirically. It is worth mentioning that CCE and CCL can work independently of CE (see the fourth row in Table while we optimize these three losses simultaneously to yield the best result. As discussed in prior sections, we hypothesize that Bi-tuning helps fine-tuning models characterize the intrinsic structure training data when using CCE and CCL simultancously.
Table 5: Collaborative effect in Bi-tuning CUB-200-2011 using ResNet-50 pre-trained by MoCo.
Loss Function
Sample Rate
CE
CCE
CCL
25%
50%
100% 75%
X
38.57:0..13
58.97:10.16 16
69.55:10.18
04.35-10.18
V
X
4..4
64.33-10.28
71.56=00.0
75.82=1022
X
V
41.09=0.23
60.77+10.31
70.30=0.29
75.30
X
V
V
47.70:10.41
64.77-10.15
71.69+0.11
7..54
V
V
50.54
66.88
74.27-10.05
77.12
Table 6: Comparison different multi-positives contrastive losses on 100% data CUB.
Losses
Fine-tuning baseline
Bi-tuning with L InfONCE
Bi-tuning with Lproposed
Accuracy
80.74
81.20
82.93
CONCLUSION
In this paper, we propose general Bi-tuning approach to fine-tuning both supervised and unsuper- vised representations. Bi-tuning generalizes the standard fine-tuning with encoder for pre-trained representations, a classifier head and a projector head for exploring both the discriminative knowledge of labels and the intrinsic structure of data, which trained end-to-end by two novel loss functions. Bi-tuning yields state-of-the-a results for fine-tuning tasks on both supervised and unsupervised pre- trained models by large margins. Code will be released upon publication httpthtt/tttt whw.. com

---

Under review as a conference paper at ICLR 2021
REFERENCES
Becker, S. and Hinton, G. E. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):111-163, 1992.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. Simple Framework for Contrastive Learning of Visual Representations, arXiv preprint ar arXiv:2002.05709 2020.
Chen, X.. Wang, S., Fu, B., Long, M., and Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NeurlPS, pp. 1906-1916, 2019.
Deng. J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. leee, 2009.
Devlin, J., Chang, M.-.,, Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Doersch, C. and Zisserman, A. Multi-task sule-super visual learning. In ICCV, pp. 2051-2060, 2017.
Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430, 2015.
Fong, R. and Vedaldi, A. Interpretable explanations black boxes by meaningful perturbation. In ICCV, pp. 3429 3429-3437, 2017.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NeurIPS, pp.
2014.
Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Hadsell, R., Chopra, S., and Lecun, Y. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
He, K., Girshick, R. B., and Dollár, P. Rethinking imagenet pre-training. In ICCV, pp. 4917-4926. IEEE, 2019.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Huang. G., Liu, Z., van der Maaten, L., and Weinberger, Q. Densely connected convolutional networks. In CVPR. IEEE Computer Society, 2017.
Kang, Z., Lu, X., Lu, Y., Peng, C., and Xu, Structure learning with similarity preserving. arXiv preprint arXiv:1912.01199 2019.
Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L. Novel dataset for fine-grained image categorization. In CVPR, Colorado Springs, CO, June 2011.
Kornblith, S., Shlens, J., and Le, Q. V. Do better imagenet models transfer better? In CVPR, pp. 2661-2671, 2019.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In 3dRR, 2013.
Li, X., Grandvalet, Y., and Davoine, F. Explicit inductive bias for transfer learning with convolutional networks. In Dy, J. G. and Krause, A. (eds.), ICMI., 2018.
10

---

Under review as a conference paper at ICLR 2021
Li, X., Xiong, H., Wang. H., Rao, Y., Liu, L.,, and Huan, J. Delta: Deep learning transfer using feature map with attention for convolutional networks. ICLR, 2019.
Li, Z. and Hoiem, D. Learning without forgetting. TPAMI, 40(12):2935-29477 2017
Lin, T.-Y., Maire, M., Belongic, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Maaten, L. v. d. and Hinton, G. Visualizing data using t-sne. Journal of machine learning reseurch, (Nov):2579-2603) 2008.
Maji, S., Rahtu, E.., Kannala, J., Blaschko, M., and Vedaldi, A. Fine- Fine-grained visual classification of aircraft. Technical report, 2013.
Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84. Springer, 2016.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, pp. 3498-3505. IEEE, 2012.
Radford, A. and Sutskever, I. Improving language understanding by generative pre-training. In arxiv, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In CVPR, 2015.
Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. arXiv preprint arXiv: 1906.0990600849, 2019.
van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding, 2018.
Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine- fine-graine grained dataset collection. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pp. 595-604, 2015.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F,, Belongie, S., and Perona, P. Caltech-UCSD Birds 200. Technical Report CNS-TR- T-22000000, Califoria Institute of Technology, 2010.
Wen, Y., Zhang, K., Li, Z., and Qiao, recognition. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), ECCV, pp. 499-515, 2016.
discriminative feature learning approach for deep face
Wu, Z., Xiong, Y., Yu. S. X., and Lin, D. Unsupervised feature learning via non non-parait instance discrimination. CVPR, 2018.
Yang, 7., Dai, 7., Yang, Y., Carbonell, J. G., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp- 3320-3328, 2014.
Zhang, R., Isola, P., and Efros, A. A. Colorful image colorization. In ECCV, 2016.
11

---

Under review as a conference paper at ICLR 2021
A VISUALIZATION BY T-SNE
We train the t-SNE MMaten & Hinton 2008 visualization model on the MoCo representations fine-
tuned on Pets dataset Parkhi et al. (2012). Visualization of the validation set shown in Figure Note that representations in Figure 4[a) do not present good classification structures. Figure 4(c) suggests that forcefully combining the unsupervised contrastive learning loss as GPT (Radford et al. 2019) may cause conflict with CE and clutter the classification boundaries. Figure F(d) suggests Bi-tuning encourages the line-tuning model to learn better intrinsic structure besides the label information. Therefore, Bi-tuning presents the best classification boundaries well intrinsic structures.
S
(a) MoCo Representations
(b) Fine-tuning with CE (c) Fine-tuning with GPT
(d) Bi-tuning
Figure 4: T-SNE Maaten Hinton 2008) visualization of baselines on Pets (Parkh al. 2012)
B KEY GENERATING MECHANISMS
B. MOMENTUM CONTRAST
Momentum Contrast (MoCo) (He et al. 2020 is general key generating mechanism for using contrastive loss. The main idea in MoCo producing encoded keys on-the-fly via momentum- updated encoder and maintaining queue to support sampling operations. Thus, the memory cost in MoCo does not depend on the size of the training set (while memory bank Wu al. (2018) will store the whole dataset). In all our experiments (Section 5), Bi-tuning chooses the unsupervised MoCo as our default setting.
Formally, denoting the momentum updated encoder as fx with parameters 0g. Likewise, denoting the backbone encoder as fa with parameters Oq. 0g is updated by:
0g f- md + (1 mm)0g-
Here we set the momentum coefficient m 0.999. To the Bi-tuning approach, we reorganize the queues in MoCo for items in each category separately. Moreover, two contrastive mechanisms in Bi-tuning are performed on different the instance level and category level respectively, and we maintain two groups of queues correspondingly.
Table Top-1 accuracy (%) of Bi-tuning on CUB with memory bank as key generating mechanism (Backbone: ResNet-50 pretrained via MoCo).
Sample Rate
Key Generating Mechanism
25%
50%
75%
100%
MoCo (He et al.
2020)
49.25 49.25++022 2.
66.88+0..13 66.69:0.26
74.27+:0.05 74.22+0031
77.12+0.23
Memory bank Wu al. 2018
50.51+0.55
77.62+0.29
B.2 MEMORY BANK
Bi-tuning is a general approach, which is not bound to any special key generating mechanism (MoCo). The memory bank proposed by /w et al. (018) generates encoded keys via momentum-updatee snapshots of all items the training set. Keys for each mini-batch are uniformly sampled from the memory bank. Compared MoCo, maintaining memory bank is computation-efffcien with
12

---

Under review as a conference paper at ICLR 2021
more memory required.
Similar to Eq.
S snapshots here updated by:
z mz! + -
h + mh + (1 - m)h..
(10)
Notations follow Section 3. Here we set the momentum coefficient m 0.5 (Wu et al. 2018 are the same as Section 5. We evaluate Bi-tuning with memory bank on CUB (Welinder et al.,2010) with the same configurations in Section 4. The results in Table 7 show that the performance is close in both methods. Key generating mechanisms in Bi-tuning only have limited effects on the final performance the supervised paradigm. These suggest that the key generating mechanism in Bi-tuning can be implemented by some variants with similar performance. MoCo recommended regarding its scalability and simplicity.
Other
C ADDITIONAL EXPERIMENTAL RESULTS
C.I BI-TUNING SUPERVISED PRE-TRAINE TRE-NEAINED REPRESENTATIONS ON MORE BENCHMARK
In addition to the Table T in the main paper, further conduct experiments Stanford Dogs (Khosla
et al. 2011), Oxford-IIIT Pets (Parkhi et al.) 2012), Oxford 102 Flowers (Nilsback & Zisserman
2008
NABirds (Van Horn al. 2015). Results shown in Table g
Table Top-1 accuracy on more benchmark using ResNet-50 by supervised pre-training.
Sampling Rates
Dataset
Method
25%
50%
75%
100%
Fine-tuning (baseline)
86.97
88.47
88.84
89.44
Dog
Bi-tuning
87.07
88.72
88.99
89.52
Fine-tuning (haseline)
90.92
91.99
92.97
93.68
Pets
Bi-tuning
90.92
92.29
93.32
93.83
Fine-tuning (baseline)
84.84
92.50
94.91
96.08
Flower
Bi-tuning
86.55
94.18
96.21
97.12
Fine-ture min-ttning (baseline)
38.86
61.33
69.92
73.44
NABird
Bi-tuning
48.33
65.81
72.36
74.83
C.2 BI-TUNING UNSUPERVISED PRE-TRAINES REPRESENTATIONS ON MORE BENCHMARK
Similarly, we also conduct experiment with ResNet-50 unsupervised pretrained by MoCo on Stanford Dogs (Khosla al., 201T), OXford-IIIT Pets (Parkh et 2012), Oxford 102 Flowers Misback & Zisserman 2008), J, NABirds (Van Horn et al. 2015). Results are shown in Table 9 suppmentary to 3 to report results on more datasets using ResNet-50 unsupervisedly pre- by MoCo. Further, to clarify that Bi-tuning is not overfitting to pre-training methods, Table 10 and Table []] provide more promising results with different unsupervisedly pre-trained representations on CUB and Aircraft respectively, which reveal that Bi- Bi-Tuning consistly outperforms fine-tuning method.
13

---

Under review as a conference paper at ICLR 2021
Table 9: Top-1 accuracy more benchmark using ResNet-50 unsupervised pre-trained pre-trrrne by MoCo.
Sampling Rates
Dataset
Method
25%
50%
75%
100%
Fine-tuning (baseline)
54.95
68.38
72.51
75.57
Dog
Bi-funing
59.32
69.52
73.25
76.65
Fine-tuning (haseline)
70.89
80.68
84.40
86.30
Pets
Bi-tuning
74.14
82.94
86.09
87.79
sune-tuning (baseline)
77.96
89.69
92.91
95.25
Flower
Bi-tuning
81.01
90.70
94.13
95.43
Fine- tuning (baseline)
56.36
69.75
74.85
77.10
NABird
Bi-tuning
62.11
73.26
77.29
79.50
Table 10: Top-1 Accuracy CUB dataset with different unsupervisedly pre-trained representations.
Pre- Pre- tre-training Method
Fine-tuning (100% data)
Bi-tuning (100% data)
Deep Cluster (Caron et al.
2018
74.63
78.24
InsDisc (u
2018)
71.35
74.92
CMC (Tian al. 2019
68.71
77.21
MoCov2 (He et all
2020
75.75
77.24
SimCLR(IX)) (Chen et a
2020
72.21
77.44
SimCIR((2x)
CChen et al.
2020
74.35
78.43
Table 11: Top-1 Accuracy on Aircraft dataset with different unsupervisedly pre-trained representa-
tions.
Pre- traininn Method
Fine-tuning tuning (100% data)
Bi-tuning (100% data)
Deep Cluster (Caron et al.
2018
77.68
81.70
InsDise (Wu
2018)
84.16
86.08
CMC (Tian al. 2019
83.98
85.84
MoCov2 (He et al.
2020
88.04
89.85
SimCLR((X) CCen et a.
2020
86.52
89.10
SimCLR((2X) Chen et
2020
88.39
89.98
14


Reviews:
[{"review_id": "Tb26MnxH27n", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to augment the regular fine-tuning stage by two additional loss: a contrastive cross-entropy loss (L_CCE) and a categorical contrastive learning loss (L_CCL).  For each input example, L_CCE minimizes the relative distance between its representation and the softmax linear weight (compared to the distance between another image from the same class and the linear weight). L_CCl encourages representations from the same class to stay close with each other. Empirical results on CUB, Cars, Aircraft and COCO-70 show the advantage of the method. \n\nStrength:\n1. Exploring the intrinsic structure of the downstream task is an interesting direction to improve transfer learning performance.\n2. The proposed method shows consistent improvements across various datasets.\n\nWeakness:\nThe additional loss terms are not well-motivated and difficulty to justify. How do they help the model exploit the intrinsic data structure?\n  a. In Eq3, L_CCE computes the log ratio between \n        {the distance between the representation h_{i}^{q} and the weight of the true class w_{y_i}}\n      and \n        the sum of {the distance between the positive key set h_{j}^{k} and the weight of the true class w_{y_i}}\n  What does this log ratio term represent? The numerator can be bigger than the denominator in this case (and then the log term will be negative). Also if you just want to encourage instances from the same class to stay closer in the representation space, why don't you just explicitly minimize the L2 distance between the representation and the prototype?\n  b. The goal of L_CCL is to encourage examples from the same class to stay closer with each other. Since the positive and negative sets are constructed using the label information, I don't see how it is complimentary to the original cross entropy loss. Also, in the case where the size of the positive set |S_i| is bigger than one, equation 4 may not make much sense as there are positive examples in the denominator.\n\nMaybe I missed or misunderstood something, but based on these questions and concerns, I found it hard to justify the improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PMQYOdoBgH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper suggests to extend fine-tuning of pre-trained representations by adding additional contrastive losses to leverage the intrinsic structure of the downstream training data. The authors call the presented method Bi-tuning and evaluate it by comparing it to other variants of fine-tuning.\n\n### Quality, clarity, originality and significance\n\nThe paper describes its motivation well, and Sections 1 and 2 are well-written and clear. Sections 3, 4, and especially 5, I found less clear and in some parts a bit confusing.\n\nOne point that I found not entirely clear in the motivation is the focus on the conjunction of *fine-tuning* and *supervised contrastive learning*. Others have looked at the latter for general training, e.g. [arXiv:2004.11362](https://arxiv.org/pdf/2004.11362.pdf) or [arXiv:1808.04699](https://arxiv.org/pdf/1808.04699.pdf), so it is not clear to me why the Bi-tuning loss is restricted (in this study) to fine-tuning only. If the loss is universally applicable, shouldn't it also work for training in general? Or if there is a reason to not expect this (i.e. to expect that it should only work well with pre-training) this should be explained (or the explanation emphasized if I missed it). If the focus is on bringing contrastive learning to the fine-tuning stage, then maybe an inclusion of some more \"plain\" contrastive  losses on the way to the \"novel\" loss introduced here may lead to a better understanding, focusing specifically on the fine-tuning stage.\n\nThe experimental results look good at first glance, but overall I found it hard to evaluate how convincing they are because of several potential problems:\n* The experimental results are not compared to any results from the existing literature, all baselines and comparisons are entirely from this paper. This makes it hard to know how much careful tuning went into the proposed algorithm versus the baselines. I think at least *some* comparison to results from the literature should be possible or it should be carefully explained why such an overfitting to the proposed algorithm clearly did not happen. For example, the [SpotTune paper](https://www.cs.utexas.edu/~grauman/papers/CVPR19_spottune.pdf) contains numbers for CUBS and Cars that look comparable at first glance because they also use pre-trained ResNet-50 models, and they include numbers for L2-SP that are better than the ones in this paper. It may be very insightful to compare results to these or other similar results. There seems to be some overlap with methods/datasets to (Chen et al., 2019), but then again there are differences like the percentages of data samples.\n* There are a lot of seemingly random choices in the selection of the datasets, e.g. why CUB/Cars/Aircraft and not e.g. CIFAR, Dogs, Pets, Flowers, or Birds? I'm not saying any of these choices are inherently better, but the authors should explain *why* these datasets are chosen, to avoid the impression that the datasets may have been chosen because the proposed method works particularly well on these dataset combinations. One way to make the results stronger would therefore be to use combinations of datasets that other, previous work has already used, which would also enable a direct comparison (see above).\n* Again in a similar fashion, Table 4 evaluates only one target data set, why only this one?\n\n### Pros and cons\n* Pros: interesting exploration of fine-tuning with added contrastive losses\n* Cons: experimental results not very convincing, main message is not very clear\n\n### Minor details and comments\n* Novelty of the presented components seemed a bit overstated to me in some parts. E.g. words like \"improved\", \"novel\", \"newly-designed\" are used throughout the work when it is not entirely clear (at least to me) where exactly the novelty lies. \n* \" the best learning rate is selected by cross-validation under a 100% sampling rate and applied to all four sampling rates.\" - how exactly is the cross-validation performed here?\n* Sec. 5.1. does not seem to specify the pretraining dataset. I assume it's ImageNet, but giving some specifics (e.g. was a particular publicly available checkpoint used) might be helpful.\n* \"Previous fine-tuning methods mainly focus on improving performance under low-data regime paradigms.\" It seems recently there are several works using fine-tuning to ImageNet, which may count as large-scale, too? E.g. [PapersWithCode-ImageNet](https://paperswithcode.com/sota/image-classification-on-imagenet) has several approaches near the top-performing methods that would fall into that category I think.\n* Figure 2(b): performance seems not yet saturated going to higher dimensionality - would it make sense to go even higher?\n* Figure 3 is very hard to interpret I think and it is only explained extremely briefly by citing a reference. Why is the visualization uniform in case (d)? What does the visualization show? Which model and data were used? Also the conclusion \"Bi-tuning in 3(e) captures both local details and global category-structures\" is not evident from the figure alone, I think it may need more explanation and interpretation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "J3dkiG6Kr96", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a general Bi-tuning approach to fine-tuning both supervised and unsupervised representations to downstream tasks. The main contribution of this paper is integrating two heads upon the backbone of pre-trained representations: a classifier head with an improved contrastive cross-entropy loss and a projector head with a newly-designed categorical contrastive learning loss. Results are shown that Bi-tuning achieves state-of-the-art results for fine-tuning tasks of both supervised and unsupervised pre-trained models by large margins.\n\nPros: \n\n+ Overall, the paper is well written. In particular, the Introduction section has a nice flow and puts the proposed method into context. Despite the method having limited novelty (only improve the loss function on the basis of contrastive learning), the method has been well-motivated by pointing out the limitations in SOTA methods. \n\n+ The idea of designing a novel categorical contrastive loss $L_{CCL}$ on the projector head, by expanding the scope of positive keys to a set of instances instead of a single one, it is very interesting.\n\n+ The results section is well structured. It's nice to see analysis on components of contrastive learning and the collaborative effect of the loss functions. \n\n \nCons: \n\n- The key concern about the paper is the lack of enough novelty. It seems that the core idea of Bi-tuning is the last overall loss function: $L_{CE} + L_{CCE} + L_{CCL}$, but the author stated that \"the naive combination of the supervised cross-entropy loss and the unsupervised contrastive loss is not an optimal solution for fine-tuning\" in section 4.4. In my opinion, $L_{CE}$ and $L_{CCE}$ are from the classifier head, they are supervised cross-entropy loss, while $L_{CCL}$ is from the projector head, it is an unsupervised contrastive loss. Isn't adding these three values a \"naive combination\"? If the core idea is only improved the loss function, I think such novelty is limited.\n\n- Considering the comprehensive experiments, a deeper analysis of the proposed method would have been nice. The idea of novel categorical contrastive loss $L_{CCL}$ on the projector head, expanding the scope of positive keys to a set of instances, what kind of scope? What effect will the scope change have on the result? The author stated that \"resulting in more harmonious cooperation between the supervised and unsupervised learning mechanisms\", some form of theoretical/analytical reasoning behind the effectiveness would provide greater insights into the community and facilitate further research in this direction. \n\n- The claim of the 10.7\\% absolute rise on CUB with a sampling rate of 25\\% mentioned in the abstract has not been clearly addressed and pointed out in the results section of the paper. From Table 1, I don't know how to calculate the 10.7\\% **absolute rise**. Do you actually mean the **relative rise** including *the margin of error*?\n \n- Fig 3: Why interpretable? The author stated that \"The visualization shows that MoCo focuses only on local details.\", I don't understand how Fig 3(d) expresses local features. What's more, Fig 3 was quoted many times in the paper, such as sections 2.3 and 5.2, but I really don't understand the connection with Fig 3.\n \n- In section 5.3, the author stated \"It is worth mentioning that CCE and CCL can work independently of CE\", why not set up experiments for CCE and CCL is independent?\n\n \nMinor comments: \n\n* In Section 5.2, \"As reported in Figure 3\", the link of Fig 3 is an error.\n\n* The labels of the figures and tables in the paper should preferably appear and quote in order.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ICVy8Qh6gJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Summary ==\n\nThe paper proposes to use a contrastive cross-entropy loss during fine-tuning, for improving transfer accuracy of both supervised and unsupervised pre-training methods in image classification. The paper builds on the intuition that both class discriminative information and intrinsic structure of the downstream task are useful for fine-tuning, and existing fine-tuning approaches only use the former. The authors conduct experiments on four image classification datasets, using a modern ResNet-50 architecture.\n\n== Pros ==\n\n- The authors use two contrastive losses during fine-tuning, which has not been deeply explored, since most works typically use only class cross-entropy loss. One contrastive loss acts on the classification head, while the second, an extension of InfoNCE, acts on a projection head (on top of the representation layer).\n\n- The proposed method achieves very good results across the four image classification datasets (CUB, Cars, Aircraft and a custom version of MS-COCO), using both supervised and unsupervised pre-training. The authors perform 5 runs for each experiment and report the average as well as statistical significance metrics (although it's not clear if the provided interval is standard deviation or a confidence interval). \n\n- When using unsupervised pre-training, the authors explored 6 different pre-training algorithms, and Bi-tuning improves the standard supervised fine-tuning in all cases.\n\n- The authors also show that their method outperforms 4 baselines: standard (supervised) fine-tuning, BSS, DELTA, and L2SP by a significant margin across different data sizes available for fine-tuning (25%, 50%, 75% and 100% of the original dataset size).\n\n- Despite the fact that they have three terms in the final loss, no additional hyper-parameter needs to be introduced (in addition to choosing the number of keys, which depends on the amount of training data).\n\n== Cons ==\n\n- Despite the fact that contrastive losses have not been widely used for fine-tuning (as far as this reviewer is aware), the authors should probably tone-down statements such as \"Bi-tuning, a general learning framework to fine-tuning both supervised and unsupervised pre-trained representations to downstream tasks\". There's a plethora of works using multiple loss/regularization terms during fine-tuning, such as DELTA or BSS, which the authors compare to. The proposed work only proposes a different type of loss/regularization. One could even argue that \"Bi-tuning\" is only a particular instance of multi-task learning. \n\n- The authors could have run additional experiments with other modern deep neural network architectures alternative to the ResNet50 (Inception, EfficientNet, DenseNet, AmoebaNet, etc.) to show whether the benefits transfer to other architectures.\n\n- The fact that the authors coined the method \"Bi-tuning\" but the loss has three terms (CE, CCE and CCL) is confusing. Probably, the \"Bi\" is due to the two heads, but still.\n\n== Typos ==\n\nThe paper contains several typos and some parts are not clear or could be improved, please read it carefully, correct them. Some typos that I've found.\n\n- Not a typo, but you should indicate whether intervals in Tables are +/- std. deviation or confidence intervals.\n- Not a typo, but there are much earlier works showing that fine-tuning (usually) performs better than training from-scratch. In the introduction you cite He et al. (2019). A quick search on the Internet yields a survey from 2009 on the topic, for example.\n- Equation 1, bold q, k_+ and k_i, since these denote vectors.\n- Figure 1, missing h_0^k (according to equation 3, j starts at 0).\n- \"Previously, we propose\" -> \"Previously, we proposed\".\n- \"As shown in 1\" -> \"As shown in Table 1\".\n- \"Results in Table 2 reveal that Bi-tuning brings general gains for all tasks, even provided with sufficient training data\". I don't understand the phrase after the comma.\n- \"SimClR\" -> \"SimCLR\" in Table 4.\n- The result of Bi-training with MoCov2 (Table 4) is not statistically significant than the supervised baseline, thus it should not be bolded. \n\n== Reasons for score ==\n\nThe authors show that the proposed method achieves significantly better results than strong baselines across multiple datasets, data set sizes, pre-training paradigms (supervised vs. unsupervised) and algorithms. Despite introducing an additional head and two additional losses during fine-tuning, the proposed algorithm does not introduce more hyper-parameters than other alternatives, which makes it easier to be applied by other researchers. This reviewer believes that some claims are a bit overstated (e.g. claiming that the proposed approach is a \"framework\"), but there's no doubt that the proposed algorithm achieves excellent results and the experimental work is solid. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an alternative fine-tuning procedure by introducing a projection head and two new losses to be combined with the vanilla cross-entropy loss. The authors introduce and jointly optimize the standard cross-entropy loss, the contrastive cross-entropy loss for classifier head and the categorical contrastive learning loss for projector head in an end-to-end fashion. The authors empirically confirmed that this setup compares favorably to existing baselines.

The reviewers found the setting challenging and worth investigating. The idea of exploring the intrinsic structure of the downstream task to help with fine-tuning was deemed useful. The reviewers appreciated the thorough empirical validation. While the proposed approach was not yet explored in this specific context, most reviewers were concerned with the lack of novelty. In addition, there seems to be a large gap between the quality of exposition in the introduction and results section with respect to the rest of the paper which introduces confusion. 

As it currently stands, the paper is not yet ready for publication and I will recommend rejection. To improve the manuscript the authors should incorporate the received feedback and significantly improve the exposition and justification of the proposed loss. In terms of empirical results, the authors should also explore alternative neural architectures to validate whether the proposed approach is general and whether the need for hyperparameter tuning arises.

Decision: Reject