Head2Toe: Utilizing Intermediate Representations for Better OOD Generalization
Authors: Utku Evci
Vincent Dumoulin
Hugo Larochelle
Michael Curtis Mozer
Venue: ICLR 2022 Submitted (2022)
Keywords: efficient training
transfer learning
efficient transfer
fine tuning
computer vision
linear probe

Abstract: Transfer-learning methods aim to improve performance in a data-scarce target  domain using a model pretrained on a data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training a new classification head for the target domain. This strategy is outperformed by a more costly but state-of-the-art method---fine-tuning all parameters of the source model to the target domain---possibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers  might be directly exploited by linear probing. We propose a method, Head-to-Toe probing (Head2Toe), that selects features from all layers of the source model to train a classification head for the target-domain. In evaluations on the VTAB, Head2Toe matches performance obtained with fine-tuning on average, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning.

Under review as a conference paper at ICLR 2022
HEAD2T0E: UTILIZING INTERMEDIATE REPRESEN- TATIONS FOR BETTER OOD GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer-learning methods aim to improve performance in a data-scarce target do- main using model pretrained on data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training new classification head for the target domain. This strategy is outperformed by a more costly but state-of art method-finee finn-tuning all parameters of the source model to the target domain- poaniiilpposibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers might be directly exploited by linear probing. We propose a method, Head-to- Heae-ro-Toe probing (HEAD2TOE), that selects features from all layers of the source model to train classification head for the target-domain. In evaluations the Visual Task Adaptation Benchmark (VTAB), Head2Toe matches performance obtained with fine-tuning on average, but critically, for oft-o--distribution transfer, Head2Toe outperforms fine-tuning.
INTRODUCTION
Tranfer learning is a widely used method for obtaining strong performance in a variety of tasks where training data is scarce-see [Zhu al. (2020); Alyafeai al. (2020); Zhuang al. (2020) surveys. A well- wnoln recipe for transfer learning involves the su-
for recent application-specine pervised or unsupervised pretraining of a model on source task with large training dataset (also referred to upstream training). After pretraining, the model's output head is discarded, and the rest of the network is used to obtain a feature embedding, i.e., the output of what was formerly the penultimate layer of the network. When transferring to a targer task, new output head is trained on top of the feature extractor (downstream training). This approach makes intuitive sense: if linear combination of embedding features performs well the source task, we expect different linear combination of features to generalize to the target domain, provided the source and target tasks are similar.
This approach of training a new output head, referred to as linear probing (LINEAR), often yields significant improvements in performance the target task over training the network from scratch (Komblith et al. 2019). An alternative to LINEAR is fine-tuning (FINETUNING), which uses target- domain data to adapt all weights the feature extractor together with the new output head. This procedure requires doing forward and backward passes through the entire network at each training step and therefore its per-step cost is significantly higher than LINEAR. Furthermore, since the entire network is fine-tuned, the entire set of new weights needs to be stored for every target task, making FINETUNING impractical when working on edge devices or with a large number of target tasks. However, FINETUNING is often preferred over LINEAR since it consistently leads to better performance on a variety of target tasks even when data is searce (Zhai al.) 2019).
FINETUNING'S superior generalization in the low-data regime is counterintuitive given that the num- ber of model parameters to be adapted is often large relative to the amount of available training data. How does FINETUNING learn from few examples successfully? We conjecture that FINETUNING better leverages existing internal representations rather than discovering entirely new representa- tions; FINETUNING exposes existing features buried deep in the net for use by the classifier. Under this hypothesis, featurrss features needed for transfer are already present in the pretrained network and might be identified directly without fine-tuning the backbone itself.

---

Under review as a conference paper at ICLR 2022
Linear
natural
30
structured specialized
ALinear
xilSpr Loc
* ANORB-AAth xcNoRE ERY
20
Cleve Counh Regirore
Head2Toe
AH2T
10
AiTT Drs
CuroseT
&e wewwertt at
-40
-20
40
60
80
Domain Affinity
Figure 1: (left) HEAD2TOE selects the most useful features from the network and trains a linear head on top. (right) HEAD2ToE improves OOD generalization. Domain affinity refers to the difference between linear probe and scratch accuracies (defined in Section 2 and the y-axis represents the improvements (delta accuracy) made by HEAD2ToF over linear probe.
In this work, we propose and explore methods for selecting useful features embedded in internal lay- ers of a pretrained net, concatenating them to the embedding produced by the pretrained net, and then applying the LINEAR transfer approach to the augmented representation (Fig. [eft)) Our approach leads to significant improvements over LINEAR as shown Fig. T right. Our key contributions are follows:
1. We perform analyses to better understand two factors that affect the benefit of incorporating intermediate representations: feature selection and the degree to which a target domain is out-of- tit-of-distribution (OOD).
2. We introduce head-to-toe probing (HEAD2TOE), simple recipe for selecting relevant fea- tures from intermediate representations for few-shot generalization. On the VTAB collec- tion of data sets, we show that HEAD2Tos outperforms LINEAR and matches the perfor- mance of more computationally costly FINETUNING.
3. Critically, HEAD2ToE outperforms TUNING OOD target domains. If a practitioner can make an educated guess about whether a target domain OOD with respect to a source, using HEAD2ToE improves the state of the for transfer learning
2 PRELIMINARIES
Source domain and backbone models. In experiments, on ImageNet-2012 (Russakovsky et al. 2015), a large scale image classification benchmark with 1000 classes and over million natural images. We benchmark HEADžToR using convolutional (ResNet-50,, Wu et al. 2018) and attention-based (ViT-B/16, Dosovitskiy et al. (2021) architectures pretrained on ImageNet-2012.
models pretrained
Target domains. In this work, we focus on target tasks with few examples few-shot and use Visual Task Adaptation Benchmark- 1k (Zhai et al. (019) to evaluate different methods. Visual Task Adaptation Benchmark- 1k consists of 19 different classification tasks, each having between 2 397 classes and a total of 1000 training instances. The domains are grouped into three primary categories: (1) natural images (natural), (2) specialized images using non-standard cameras (specialized), and (3) rendered artificial images (structured).
Characterizing nnt-of-distribution domains. Consider the relationship between source and tar- get domains. If the domain overlap is high, then features extracted for linear classification in the source domain should also be relevant for linear classification in the target domain, and LINEAR should yield performance benefits. If the domain overlap is low, then constraining the target domain use the source domain embedding may be harmful relative to training model from scratch
2

---

Under review as a conference paper at ICLR 2022
Test Accuracy
80
60
40
Linear
20
Fine-tuning
Scratch
dspr-L.c
Clevr Clevr-Ditt
SNORB VOORBEElev Elev SVHN
structured
SNORB Azim
dSpr-Orrien OOiint
dipr- DMLab
Retinopathy
Coun-
EureSAT
Camelyon
specialized
.avg
KITTI-Dist
Resise45
Sun397
.natural
CIFAR-1 100
Flowers 102
Caltech101 Pets
Clev--
Figure 2: Characterizing out-of-distribution domains Generalization performance of various baselines on VTAB-IK benchmark using ResNer-50 architecture and 224 image size. The ar- chitecture is pretrained on ImageNet-2012 for the transfer learning (TL) baselines. Datasets (and the three groups of the benchmark) are ordered according to their Domain Affinity scores in as- cending order from left to right. Examples from the left- and right-most datasets are also shown corresponding sides.
the target domain. Therefore, we might quantify the source-target distribution shift in terms of how beneficial LINEAR relative to training model from scratch (denoted SCRAtCH):
DomainA,finitt- ACCLINEAR ACCSCRATCH-
In Fig. 2, we sort the 19 VTAB- Ik target tasks by their domain affinity ImageNet-2012. from low to high-for a pretrained ResNet-50 backbone. The Figure shows examples of the three target domains with the most and least distribution shift (left and right sides, respectively). These examples seem consistent with intuitive notions of distribution shift.
Baselines. Fig. 2 also presents transfer test accuracy of LINEAR, FINETUNING, and SCRATCH baselines. Consistent with the literature, FINETUNING performs as well or better than the other two baselines. For in-distribution targets (right side of graph). LINEAR matches FINETUNING; for OOD targets (left side of graph), LINEAR is worse than FINETUNING. With distribution mismatch, the source network may filter information available in lower layers because is not needed for the source task but is essential for the target task. Observing FINETUNING performs better than SCRATCHI in OOD tasks, hypothesize that intermediate features are key for the FINETUNING, since learning novel features was possible using limited data, training the network from scratch would work on-par with FINETUNING. Motivated by this observation, HEAD2TOF probes the intermediate features of a network directly and aims to eliminate the need for fine-tuning.
3 HEAD2ToE UTILIZATION OF PRETRAINED BACKBONES
3.1 YOUR REPRESENTATIONS ARE RICHER THAN YOU THINK
In this section, we conduct a simple experiment to demonstrate the potential of using representa- tions from intermediate layers. We concatenate the feature embedding of a pretrained ResNet-50 backbone (features from the penultimate layer) with features from one additional layer and train linear head on top the concatenated features. When extracting features from convolutional layers, reduce the dimensionality of the convolutional stack of feature maps using strided average pool- ing, with window size chosen so that the resulting number of features is similar to the number of embedding features (2048 for ResNet-50).
In order to estimate an upper bound on how much the performance can improved over LINEAR by including single intermediate layer, boost in test performance (for each target task separately). Percentage improvement over LINEAR using this ORACLE is shown in Fig. Helttt We observe negative correlation (-0.745, Spearman)
oracle to select the layer which yields the largest
3

---

Under review as a conference paper at ICLR 2022
Clevr-Dist
Pets
8
natural
Test Accuracy
Liner
Test Accuracy
1,00
aat
Aoracle ALinear
structured
0.06
Dee Oount Sount
speciallzed
0.90
2
2
Layer
Layer
Comiiton
Method
Avg
Spel
Stre
Natr
Lincar
56.53
80,70
36,36
65.79
-40
-20
0
20
40
60
80
Control 58.96
81.00
40.88
67.03
Domain Affinity
Oracle
60.15
81.52
43.44
67.03
Figure 3: (left) Accuracy gains when prelogits augmented with additional layer correlates negatively (-0.745, Spearman) with domain affinity. (right-top) effect of using features from inter- mediate layers for Clevr-Dist (low domain affinity) and Pets (high domain affinity) tasks (right- bottom) Test accuracies of various baselines on VTAB-1k. Linear uses only prelogits, Oracle averages are obtained by using the layer that gives best generalization (test) for each task. Con- trol experiment uses second feature embedding from second pretrained network trained using different seed. We use ResNet-50 models pretrained on ImageNet-2012.
between the domain affinity of a target task and the accuracy gain, As predicted earlier, adding intermediate representations does not improve in-domain generalization because feature embedding already contains the most useful features. In contrast, generalization on out-of-domain tasks are improved significantly when intermediate features are used.
In Fig. toop-rigtt we show test accuracy as a function of the ResNet-50 layer whose internal repre- sentation is used to augment the feature embeddings boost LINEAR transfer. Figures for remaining tasks can be found in Appendix D Different tasks on VTAB benchmark prefer inclusion of differ- ent layers obtain the best generalization accuracy, emphasizing the importance of selecting the right set of features for optimal generalization. Overall, the ORACLE that selects the layer with best test performance for each task yields an average of 3.5% improvement on VTAB- 1k benchmark. One possible explanation for the improvement in performance with the augmented representation simply that it has more degrees of freedom (4096 features instead of 2048). To demonstrate that the improvement is due to inclusion of intermediate layers and not simply due to increased dimen- sionality, we formed a 4096-clement feature representation by using a second feature embedding obtained from second ResNet-50 backbone pretrained on ImageNet-2012 (CONTROL). Note that this experiment bears similarity to ensembling (Zhou et al. 2002). which is known to bring sig- nificant accuracy gains its own (Mustafa et (020)). Utilziing second backbone doubles the amount of resources required, it falls 1% shy of the Oracle performance demonstrating the extent which using intermediate representations can be helpful for generalization.
3.2 HEAD2ToE
Motivated by our observations in the previous section, we hypothesize that we can attain or pos- sibly surpass-thh the performance of FINeTUNING without changing the backbone itself by using LINEAR augmented with well-chosen intermediate activations. However, this approach introduces the problem of selecting features. The number of possible features to include is large relative the amount of training data available, leading to the potential of overfitting. In this section, we explore various regularization and feature selection methods, which we then incorporate into our method, HEAD2TOE.
Notation. Our method applies to any network with any type of layers, but here, for simplicity and without loss of generality, we consider a network with L fully connected layers, each layer receiving input from the layer below:
2z h Th-iWe We ; h f(z))
(1)
4

---

Under review as a conference paper at ICLR 2022
where the subscript denotes a layer index, hg is the input, f is the activation function, We is the weight matrix of layer f, and &L is the logit vector used for classification. When transferring a pretrained network to target task using LINEAR, discard the last layer of the pretrained network and train a new set of linear weights, WT, such that predictions (logits) for the new task are obtained by 2 h W-
Head2Toe. Consider a simple scheme that augments the backbone embedding with activations from all layers of the network, such that:
= hall Wall ; hall - concat(a (hi)) ar (hL))
where ae(.) denotes fixed function to reduce the dimensionality of the activation vector at given layer f. Such functions become valuable when considering network architectures that generate large number of intermediate features. For example, for convolutional networks, where the representations at each layer are structured as multi-channel spatial arrays of detectors, we perform two-dimensional average pooling as explained in Section 3.1 and Section 5 Even with dimensionality reduction, hall can exceed a million elements, and Wolt is underconstrained by the training data, leading to overfitting. Further, without care Wall may become so large as to be impractical for deploying this We can address these issues by selecting a subset of features before training the target-
model[ | domain classifier. Equivalently, we can zero out the non-selected rows of W We will show that appropriate selection of subset of features leads to better generalization than using all features.
Feature selection based on group lasso. Group lasso Yuan & Lin 2006) is a popular method for selecting relevant features in multi-task adaptation settings (Argyriou et al. 2007 NNe et al. 2010) When used as regularizer on a weight matrix W, the group-lasso regularizer encourages the & norm of the rows of the matrix to be sparse, and is defined as:
W 2,1 - |8/1 Σ 1ss1
Σ
81
To determine which features are most useful for the task, the linear head is trained with group-lasso regularization on Wal. In contrast to the approach taken by /Argyriou et al. (2007) and [Nie et al. (2010), which use costly matrix inversions, we incorporate the regularization as a secondary loss and train with stochastic gradient descent. Following training, the l2 norms of the rows of Walt. 8, are used decide which features are most relevant. We refer to s; as the relevance score for feature We pick fraction F' of all features with the greatest relevance and train new linear head to obtain the final logit mapping. Feature selection alone provides strong regularization, therefore during the final training we don't 't use any additional regularization.
We make two comments on this. First, because the initial round of training Wall with the group-lasso regularizer is used only to rank features by importance, the method is quite robust to the regulariza- tion coefficient; simply needs to be large enough to distinguish the contribution of the individual features. Second, interpreting 8; as the importance of feature i depends on all features having the same dynamic range. This constraint will often be satisfied naturally due to the normalization layers found in the architecture itself (we read out representations after normalization). To cover the maining cases (where no normalization exists between two layers) we normalize the representations of each layer to a unit norm.
Selection of F and the cost of HEAD2TOE. The fraction F determines the total number of fea- tures retained. One would expect the optimal value to depend on the target task. We select F based on validation performance. This validation procedure is inexpensive compared to the cost of the ini- tial phase of the algorithm (i.e., training of Wau to obtain s) due to the reduced number of features in the second step. Overall, HEAD2ToE together with its validation procedure requires 18% more operations compared to training Wau alone (details shared in Appendix B
Verifying HEAD2ToF. In Fig. | left, we demonstrate the effectiveness of group lasso identi- fying relevant intermediate features of a ResNet-50 for transfer from ImageNet. We rank all features
For example, using a pooling size of 2, ResNet-50 generates 1.7 million features and storing W at requires 6.6e8 parameters (2.6GB for float32) for SUN-397.
5

---

predicts the importance of Under review as a conference paper at ICLR 2022
0.60
Oracle
3.0
Feature (Acc: 63.8%)
Oracle
Test Accuracy
0.58
0.58
Test Test Accuracy
Linear
0.56
Linear
0.56
0.44
0.52
Layer (AcC, 44.1%)
350
feat wiss
layer-oie viise
0.8
layer viice (group)
0
5000
10000 15000 20000 25000
0/66
10"
13a
Offset
#fieatures Selected
Figure 4: (left) Over all VTAB tasks, average accuracy of HEAD2ToE when selecting 2048 con- secutive features sorted by their relevance score, starting with an index specified by offset. (center) Average accuracy over all VTAB tasks as function of the number of features included. In this experiment, we only use features from the first layer of each of the 18 ResNet blocks and adjust pooling have around 2048 features for each layer, totalling 29800 features. Results show that selecting layers performs worse than selecting features when adapting to target domain. (right) Distribution of 2048 intermediate features retained from a ResNet-50 when using feature-wist wise and layer-wise scores on the SVHN transfer task.
by their relevance score, 8i, and we select groups of 2048 consecutive features beginning at a par- ticular offset in this ranking. Offset 0 corresponds to selecting the features with largest relevance. We calculate average test accuracies across all VTAB tasks. As the figure shows, test accuracy de- creases monotonically with the offset, indicating that the relevance including a feature in the linear classifier.
HEAD2ToR selects individual features independent of the layer in which they embedded. We compare this feature-wise strategy to selecting layers as whole (i.e., selecting all features a layer or none). One might expect layer-wise selection be superior because it involves fewer decisions and therefore less opportunity to overfit to the validation set that is being used for selection. Further, layer-wise selection may be superior because the relevance of one feature in layer may indicate the relevance of others. To obtain layer-wise relevance score. we compute the mean relevance score of all features selection algorithm, in which we group weights originating from the same layer together and select layers using the l2 norm of the groups (layer-wise (group)). Fig. Acenter compares feature wise and layer-wise selection methods, matched for number of features retained. Feature-wise selection performs better than layer-wise selection on average and the best performance is obtained when around 1000 features are kept. We share figures for all 19 datasets in Appendix E Fig. right shows the distribution of features selected over different layers for both strategies.
layer and then rank layers by relevance. We also run an alternative layer
Note that HEAD2ToE's use of a fixed backbone means that as we search for features include, the extracted features themselves are not changing. Consequently, we can calculate them once and re- as necessary, instead of recalculating every step, as required for FINETUNING. Furthermore, since the backbone is frozen, the only additional weight storage required for each target task is the output head. Such properties are key for transfer-leaming methods to be practical when using models with billions of parameters.
4 RELATED WORK
Transfer Learning is studied extensively in the literature and used widely in practice. Most similar to our work is ELMo (Peters et al., 2018). which averages intermediate representations of lan- guage model (2 LSTM embeddings) using a learned linear combination (a softmax). ELMo requires embeddings to be same size and is most similar to the suboptimal layer-selection baseline shown in Fig. center. Peters et al.] (219)) compared FINETUNING with LINEAR and reported a small gap in performance for ELMo and BERT models. Kornblith et al. (2019) showed that ImageNet-2012 performance correlates highly with LINEAR and FINETUNING. Using large pool of pretrained models, Renggli et al. (2020) examined various metrics to decide which backbone to transfer. Given the ever increasing size and never saturating performance of pretrained models, the importance of reducing the cost of FINETUNING models is stated in the literature regularly. Methods like feature- wise transformations (Bilen & Vedaldi 2017), residual adapters (Houlsby al. 2019: Rebuffi al.)

---

Under review as a conference paper at ICLR 2022
2017).
Diff-pruning (Guo et al. 2021) and selective fine-tuning (Guo et al. proposed in order to reduce the cost of storing fine-tuned models. Zhang et al. (022) showed that selecting sub-network of pretrained model can increase OOD generalization. However, none of these methods match the simplicity of training (and storing) a linear classifier. Teerapittayanon et al. (2016). Kaya et al. (211)) and (Zhou et al. (2020) studied intermediate representations reduce "overthinking" and thus provide better early-exit criteria. Similarly Baldock et al. (2021) showed a correlation between early classification of a sample and how easy its classification is.
2019 Fu et al. 202 1)
Alternatives to transfer learning exist to learn from few labeled examples. Meta-learning TThrun & Prati 1998, Schmidhuber 1987; Schmidhuber et al.) 1997) is used (among other things) to tackle the few-shot learning problem (i.e learning from limited data) from a learning-to-lean perspective; see Hospedales et al. (2020) for comprehensive survey. In-between the representation learning and learning-to-learn paradigms lie approaches that use multi-domain training as an inductive bias, either without (Dvornik et al. 2020 Triantafillou et al. 2021 Li et al. 202Tb a) or with (Liu et al.)
2021) meta-learning. However, in the large-scale setting FINETUNING remains a top-performing approach to few-shot classification (Dumoulin et al. 2021).
Feature selection aims to find the most relevant features from the input and is studied extensively in the Machhnn--laaringg literature. Approaches can be grouped according to whether labeled data is used- supervised (Nie et al. (2100) unsupervised (He et al., 2005 Balm et al. 2019 Atash-
gahi et al.] 2020) or what high-level approach is faken-filter methods (Blum & Langley 1997),
wrapper methods (Kohavi & John, 1997), or embedded methods (Yuan & Lin| 2006). Most rel-
evant to our work are embedded supervised methods as they have good scaling properties while achieving the best performance which is vital in our setting with over a million features. Embedded supervised feature selection methods use a cost function to iteratively refine the subset of features selected and popular approaches include forward selection (Viola & Jones 2001 [oobooudaiis & [1899]99yoo Guyon et al. 2004) based methods (Yuan Lin| 2006 Blum & Langley 1997; /Zhao
Tsmarddnoss, 2019), backward selection (pruning) (Mozer & Smolensky 1989;
and regularization/eetureeaaaa et al. 2010). Most relevant to our work is AAgyriou et al. (2007); Nie et al. (2010)), both of which uses 2, regularization to select features, however their approach requires matrix inversions which is not practical in our setting. We point interested readers to the survey Gf et al. (2017) and book
of Boln-Canedo et al. (015) for a detailed discussion on the topic.
5 EVALUATING HEAD2TOE
We evaluate HRAD2TOR on the VTAB- benchmark using two popular vision architectures, ResNet-50 (wu et al. 2018) and ViT-B/16 (Dosovitskiy et al.) 202T), both pretrained on ImageNet- 2012. Re consists of 50 convolutional layers. To utilize the intermediate convolutional features, we aggregate spatial information using average pooling non-overlapping patches, as ex- plained in Section 3 We adjust the pooling size to target a fixed dimensionality of the representation. For example, for feature map of shape 20 x 20 x 128 and target size of 512, we average-pool disjoint patches of size 5 x 5, resulting in 4 features per channel and 1024 features in total. This helps to balance different layers in terms of number features they contribute the concatenated embedding. We normalize features coming from each layer to a unit norm. This scaling preserves the relative magnitude of features within a layer while accounting for inter-layer differences and works better than normalizing each feature separately or not normalizing at all. ViT-B/16 consists of 12 multi-headed self-attention layers. When we aggregate the output of the self-attention lay- ers, we perform 1-D average pooling over the patch/token dimension choosing the pooling size to match a target number of features as before. Given that token dimension is permutation invariant, 1-D average pooling might not be the best choice here and better aggregation function can provide further gains. We share a detailed list of intermediate representations utilized for each architecture in Appendix H
HEAD2ToE selects a subset of features and trains a linear classifier without regularization on top of the selected features. We compare HEAD2ToE with regularization baselines that utilize all features without doing feature selection. These baselines are denoted as +All-f1, +All-&2 and +All-62,1 according to the norm they use.
We perform five-fold cross validation for each task and method in order to pick the best hyper- parameters. All methods search over the same learning rates and training steps (two values of
7

---

Under review as a conference paper at ICLR 2022
Specialized
Strictured
SNORB ANORB-Azim Azim
NNOBB. Elev
CIFAR 100
Caltechior
DID
Flovers
Pets
SVHN
San.39
Canelyon
EuroSAT
Resise45
Retinopathy
Clevr-Count
Cleve-Dist
DMLah
KITT-Dis
dSpr.ee
dSpr-On
Mean
48.5
36.0
19.6
25.5
+All-gg
44.7
67.8
86.1
31.9
56.3
+All-61
50.8
88.6
67.4
87.7
34.6
94,9
75.6
74.7
57.0
44.8
40.8
64.6
+2,1-f2,1
49.1
86.7
68.5
84.2
88.0
84.4
34,8
81.5
94.9
75.7
74.3
48.3
58.4
42.0
74.4
58.8
45.2
37.8
34.4
64.3
Head2Toe
47.1
88.8
67.6
85.6
87.6
32.9
94.3
76.0
74.1
58.3
59.5
43.9
72.3
64.9
51.1
39.6
43.1
68.8
Fine-runing"
54.6
89.8
65.6
88.4
89.1
86.3
34.5
79.7
95.3
81.0
72.6
41.
52.5
42.7
75.3
81.0
47.3
32.6
35.8
66.6
Scratch"
11.0
37.7
23.0
40.2
13.3
59.3
3,9
73.5
84.8
41.6
63.1
38.5
54.8
35.8
36,9
87.9
37.3
20.9
36,9
42.1
Natural
Specialized
Structured
CIFAR-100 100
Caltech101
DDD
Floworst02
Pets
SVHN
Sun397
Canelyon
EunSAT
Rosise45
Retinopatyy
Clov--Com Count
Clvi-Dist
DMLab
KITTI-iis
Spe-LLo
dSp-OO
SNORB
SNORB Elee
Mean
Linear
55.0
81.0
53.6
85.3
38.7
32.3
80.1
90.8
67.2
74.0
38.5
36.
33.5
55.7
34.0
31.3
18.2
26.3
52.8
+All
57.3
87.0
64.3
82.8
84,0
75.7
32.4
82.0
94.7
79.7
74.
474
57.8
41.4
62.8
46.6
33.3
31.0
38.8
61.8
+All-f1
58.4
87.3
64.9
83.3
84.6
80.0
34.4
82.3
95.6
79.6
73.6
47.9
57.7
42.
65.1
44.5
33,4
32.4
38.4
62.4
4All-f2.1
59.6
87.1
64.9
85.2
85.4
79.5
35.3
82.0
95.3
80.6
74.2
47.9
57.8
40.7
64.9
46.7
33.6
31.9
39.0
62.7
Head2Toe
58.2
87.3
64.5
85.9
85.4
82.9
35.1
81.2
95.0
79.9
74.1
49.3
58.4
41.6
64.4
53.3
32,9
33.5
39.4
63.3
Finc-tuoing
62.6
83.1
61.5
80.4
86.8
83.0
33.7
83.2
94.9
78.0
73.6
58.3
59.2
43.6
39.8
61.4
44.8
27.1
26.2
62.2
Fine-tuning+
65.2
82.1
60.5
85.2
86.9
86.7
36.9
82.4
95.2
81.3
73.0
68.2
60.6
41.7
73.9
69.3
28.0
25.7
36.1
65.2
Table 1: Median test accuracy over 3 seeds the VTAB-Ik benchmark using pretrained (top) ResNet-50 and (bottom) ViT-B/16 backbones. Regularization baselines that use all layers are indi- cated with the +All prefix. "*" indicates results obtained from Zhai et al. (2019). Fine-tuning results for ViT-B/16 are obtained using code and checkpoints provided by Dosovitskiy et al. (2021)
each). Methods that leverage intermediate features (i.e., regularization baselines and HEAD2ToE) additionally search over regularization coefficients and the size of the aggregated representation at each layer. The FINETUNING baseline searches over 4 seper-pprammttrs: thus the comparison of HEAD2TOE, which searches over 24 values, to fine-tuning might seem unfair. However, this was necessary due to fine-tuning being significantly more costly than training a linear classifier, even with intermediate features. We repeat each evaluation using 3 different seeds and report median values and share standard deviations in Appendix More details on papee-parrmmter selection and best values be found in Appendix
5.1 RESNET-50
The top half of pretrained ResNet-50 architecture. On average. HEAD2TOE slightly outperforms all other methods, including FINETUNING (see rightmost column of Table). HEAD2TOE, along with the regularization baselines that use intermediate layers, is far superior LINEAR, indicating the value of the inter- mediate layers. And HEAD2ToE is superior to the regularization baselines, indicating the value of explicit feature selection. Among the three categories of tasks. HEAD2TOE excels relative to the other methods for the specialized category, but does not outperform FINETUNING for the natural and structured categories.
results on the 19 VTAB-1k target domains when transferring from a
Does HEAD2ToE select different features for each task? Which layers are used more frequently? In Appendix[F F. intersection among features selected for different tasks and seeds. We observe a high variety among layers and features motivating the importance of performing feature selection for each task sepa- rately. HEAD2ToE exceeds FINETUNING performance, but requires only 0.5% of FLOPs during training on average. Similarly, the cost of storing the adapted model is reduced to 1 1 on average. We discuss HEAD2TOF's computational and storage costs in detail in Appendix
we show the distribution of features selected across different layers and the amount of
8

---

Under review as a conference paper at ICLR 2022
ResNet-50
VIT-B/16
0.70
Grapp
Group2
Validation Accuracy
Validation Accuracy
0.65
DMLab
CIFAR-
DTD
Clevr-Count
0.65
sNORB-Azim
dSpr-Orient
0.60
SVHN
Retinopathy
0.60
group
Groupt
0.55
Groupl
dSpe-Loc
Pcts
BureSAT
Resisc45
Group2
Grounz
ANORB-Eley
Flower102
0.5
rLayers
J.yeess
Camelyon
V/5
1/15
Caltech
1/4
1/4
Clevr-Dist
050
KITTI-Dist
6
5030
13000
15000
20000
25000
5000
10000 5000
70000253000 90000
Target Size
Target Size
Figure 5: Effect of increasing the number of intermediate features that HEAD2TOE uses from the (left) ResNet-50 backhone and (right) ViT-B/16 backbone. The abscissa of the graph indicates the dimensionality of the representation extracted from each layer of the backbone (target size). The tasks are split into two groups (see right side of Figure), which show different behavior. The solid, dashed, and dotted lines indicate the fraction of layers selected for forming the representation used by HEAD2ToE: 1/16, 1/4, and 1, respectively. Scaling curves for individual tasks can be found in Appendix G
5.2 VIT-B/16
Results for ViT-B/16 shared in the bottom half of Table 1 As with the ResNet-50 archi- tecture, HEAD2TOF achieves the best accuracy among methods that keep the backhone fixed: HEAD2ToR improves accuracy over LINEAR by about 10% on average (in absolute performance), and HEAD2ToE outperforms the regularization baselines that include intermediate features but that do explicitly select features. The story for HEAD2ToE versus FINE TUNING is bit more com- plicated. The FINETUNING recipe used by Dosovitskiy et al. (202T) includes many tweaks such as selective aata-auumentttion,, image upsampling. learning rate warm-up, and gradient clipping. In order to make a direct comparison with LINEAR, in the next-to-last row of the Table we report the performance of FINETUNING without all of these tweaks; HEAD2TOR obtains about a absolute accuracy improvement over FINETUNIE incorporated into FINETUNING- in particular gradient clipping- FINETUNING achieves a 3% im- provement over HEAD2TOE. Since it is linear classifier, adding gradient clipping HEAD2TOI doesn't 't improve the results. That being said, HEAD2TOE retains its competitive advantage in terms of resource costs and its accuracy would likely similarly benefit from the development of more sophisticated tricks (notably moving beyond the naive I-D pooling of ViT features), much like FINETUNING has gained over time.
in this setting. However, when
of the tweaks
5.3 SCALING HEAD2TOE
In Fig. 5 we vary the number of intermediate features used for each of our pretrained backbones. We observe that including layers always performs better on average. However when varying the number of target features for each layer, we observed 2 distinct sub-group of target tasks that behave differently as the number of features increases. This observation informed our discussion to include both small and large target sizes in our validation hyper parameter search.
6 CONCLUSION
In this work, we introduced HEAD2ToE, an approach that extends linear probing (LINEAR) by se- lecting the most relevant features among pretrained network's intermediate representations. We show that doing so greatly improves performance over LINEAR and allows the approach to reach performance competitive with-and some cases superior to TO-FINETUNING. Our findings chal- lenge the conventional belief that FINETUNING is required to achieve good performance on OOD tasks. While more work is needed before HEAD2ToE can realize the full computational benefits of linear probing, our work paves the way for applying new and more efficient feature selection approaches and for experimenting with HEAD2ToE probing in other domains such as regression, video classification, object detection, reinforcement learning, and language modelling domains.

---

Under review as a conference paper at ICLR 2022
REFERENCES
7aid Alyafeai, Maged Saeed AIShaibani, and Irfan Ahmad. A survey on transfer learning in natural language processing. arXiv preprint arXiv:2007.0422 040099 2020.
Andreas Argyriou, Theodoros Eygeniou, and Massimiliano Pontil. Multi-task feature learning. In B. Schölkopf, Platt, and Hoffman (eds.), Advances in Neural Information Processing Sys- tems. MIT Press, 2007.
Zahra Atashgahi, Ghada Sokar, Tim van der Lee, Elena Mocanu, Decebal Constantin Mocanu, Ray- mond N. J. Veldhuis, and Mykola Pechenizkiy. Quick and robust feature selection: the strength of energy-efficient sparse training for autoencoders. ArXiv, abs/2012.00560, 2020.
R. Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. ArXiv, abs/2106.09647,, 2021.
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint 17Xiv:1701.07275, 2017.
Avrim L. Blum and Pat Langley. Selection of relevant features and examples in machine learning. Artificial Intelligence, 1997.
Vernica Boln-Canedo, Noelia Snchez-Maroo, and Amparo Alonso-Betanzor Feature Selection for High- Riigh-Dimensional Data. Springer Publishing Company, Incorporated, 1st edition, 2015. ISBN 3319218573.
Giorgos Borboudakis and Tsamardinos. Forward-backward selection with early dropping. J. Mach. Learn. Res., 20:8: 1-::9, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference Learning Representations, 2021. URL https: //openreview www.tett...uaa d=Y cbFdNTTy
Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches on a unified few-shot classi- fication benchmark. arXiv preprint urXiv:2104.026333 2021.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi- domain representation for few-shot classification. European Conference on Computer Vision, pp. 769-786. Springer, 2020.
Cheng Fu, Hanxian Huang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. Learn-to-share: A trrdware-friendly transfer learning framework exploiting computation and parameter sharing. In ICML, 2021.
Jie Gui, Zhenan Sun, Shuiwang Ji, Dacheng Tao, and Tieniu Tan. Feature selection based on struc- tured sparsity: A comprehensive study. IEEE Transactions on Neural Networks and Learning Systems, 28:1490-1507, 2017.
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL/JCNLPP 2021.
Yunhui Guo, Humphrey Shi, Abhishek Kumar, Kristen Grauman, Tajana Simunic, and
Rogério Schmidt Feris. Spottune: Transfer learning through adaptive fine-tuning. 2019 IEÉE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4800-4809, 2019.
10

---

Under review as a conference paper at ICLR 2022
Isabelle Guyon, Jason Weston, Stephen D. Barnhill, and Vladimir Naumovich Vapnik. Gene lection for cancer classification using support vector machines. Machine Learning, 46:389 422, 2004.
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In Proceedings of the 18th International Conference on Neural Information Processing Systems, 2005.
Timothy Hospedales, Antreas Antoniou, Paul Micaclli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.054333 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In ICMI., 2019.
Yigitcan Kaya, Sanghyun Hong, and T. Dumitras. mitigating network overthinking. In JCML, 2019.
Shallow-deep networks: Understanding and
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artif. Intell., 1997.
Simon Korbllith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2656- 2666, 2019.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Improving task adaptation for cross-domain few-shot learning. arXiv preprint arXiv:2107.00358, 2021a.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. arXiv preprint 1...:22033.384. 2021b.
Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal rep- resentation transformer layer for few-shot image classification. In International Conference Learning Representations, 2021.
Michael C Mozer and Paul Smolensky. Skeletonization: technique for trimming the fat from network via relevance assessment. In Advances in Neural Information Process- ing Systems, 1989 URL http:/ppss/pppcceeissseeeeee http:/pps...c/ppp////////////
c7edcd7dca 9a 77
pdd
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, andAndr'e e Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. ArXiv, abs/2010.06866, 2020.
Feiping Nie, Heng Huang, Xiao Cai, and Chris Ding. Efficient and robust feature selection via joint
f2, forms minimization. In J. Lafferty, C. Williams, J. Shawe-Taylee Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, 2010.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune not to tune? adapting pretrained representations to diverse tasks. In RepLANLP@ACL, 2019.
Sylvestre-Allvis Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. arXiv preprint arXivv 2017.
Cédric Renggli, André Susano Pinto, Luka Rimanic, J. Puigcerver, Carlos Riquelme, Ce Zhang, and Mario Lucic. Which model to transfer? finding the needle in the growing haystack. ArXiv, abs/2010.06402,, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-mela-.. hook. PhD thesis, Technische Universität München, 1987.

---

Under review as a conference paper at ICLR 2022
Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering, Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self- colf-improveeett Machine Learning, 28(1): 105-130, 1997.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: exiting from deep neural networks. 2016 23rd International Conference on Pattern Recognition (ICPR). pp. 2464-2469, 2016.
Fast inference via early
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pp. 3-17. Springer, 1998.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. In International Conference on Machine Learning, 2021.
Paul A. Viola and Michael Jones. Rapid object detection using boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, 1:1-1, 2001.
Songtao Wu, Shenghua Zhong, and Yan Liu. Deep residual learning for image steganalysis. Multi- media Tools and Applications, 2018.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of The Royal Statistical Society Series B-statistical Methodology, 68:49-67, 2006.
Xiaohua Zhai, J. Puigcerver, Alexander Kolesnikov, P. Ruyssen, Carlos Riquelme, Mario Lu- cic, Josip Djolonga, André Susano Pinto, Maxim Neumann, A. Dosovitskiy, L. Beyer, Olivier Bachem, M. Tschannen, Marcin Michalski, O. Bousquet, S. Gelly, and N. Houlsby, The visual task adaptation benchmark. ArXiv, 0/bs/1910.0486 2019.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron C. Courville. Can subnetwork structure be the key to dittr--distiibution generalization? In JCML, 2021.
Zheng Zhao, L. Wang, and Huan Liu. Efficient spectral feature selection with minimum redundancy. In AAAI, 2010.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses pa- tience: Fast and robust inference with early exit. ArXiv, abs/2006.00 0ttps/2006.441555 04152, 2020.
Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: Many could better than all. Artificial Intelligence, 2002. URL
atticle//11/5000170000000
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprinl MXXi:2009.007880 2020.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
12

---

Under review as a conference paper at ICLR 2022
A VALIDATION PROCEDURE FOR DIFFERENT ALGORITHMS
We perform hypereparameter validation for each VTAB task separately. For all methods, we chose the learning rate and the total number of training sleps using the grid Ir 0.1, 0.01 and steps = 500, 5000, following the lightweight hyper-parameter sweep recommended by the VTAB benchmark (Zhai al. 2019).
For regularization baselines 11. l2 and 22 search for regularization coefficients using (0.00001, 0.0001, 0.001). We include an extra value in this setting in order to account for the over- head introduced by HEAD2TOE.
For HEAD2TOE we choose 221 regularization coefficients from (0.001, 0.00001) and tar- get feature sizes from (1024, 16384, 40000) for ResNet-50 and (768. 15360, 32448) for ViT-B/16. After calculating feature scores HEAD2ToE validates the following fractions: (0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1) and thus requires 18% more operations com- pared to other regularization baselines. Note that this is because initial training to obtain feature scores s; performed once and therefore searching for optimal number of features has small overhead. Hyper parameters selected by HEAD2ToE for each VTAB task are shared in Table and Table Next we explain how this overhead is estimated.
Dataset
Target Size
F
LR
Steps
f2,1 Coef.
Caltech101
8192
0.010
0.01
5000
0.00001
CIFAR-100
512
0.200
0.01
500
0.00001
Clevr-Dist
8192
0,001
0.01
500
0.00100
Clevr- Count
512
0.005
0.10
5000
0.00100
Retinopathy
8192
0.200
0.01
500
0.00001
DMLab
8192
0.020
0.01
500
0.00001
dSpr-Orient
512
0.200
0.01
500
0.00001
dSpr-Loc
8192
0.005
0.10
500
0.00100
DTD
24576
0.005
0.01
5000
0.00001
EuroSAT
512
0.100
0.01
500
0.00001
KITTI-Dist
8192
0.020
0.01
500
0.00001
Flowers 102
512
0.100
0.01
5000
0.00001
Pets
8192
0.002
0.01
5000
0.00001
Camelyon
512
0.020
0.10
500
0.00100
Resise45
8192
0.020
0.01
500
0.00001
SNORB-AR Azii
24576
0.002
0.01
500
0.00001
sNORB-Elev
8192
0.050
0.01
500
0.00100
Sun397
512
0.100
0.01
5000
0.00100
SVHN
24576
0.005
0.01
500
0.00001
Table 2: Hyper parameters selected for the VTAB-Ik benchmark tasks when using pretrained
Re backbone. F: fraction of features kept, LR: learning rate, Steps: Training Steps.
B COST OF HEAD2ToE
We evaluate different values of F and pick the value with best validation performance. Cost of
HEAD2ToE consists of three parts: (1) Cr: Cost of calculating the representations using the pre- trained backbone. (2) Cau: cost of training the initial head Wall (in order obtain 8; s,'s) (3) Cr=f: total cost of validating different values of F. Cost of validating a fraction value f. assuming equal number of training steps, is equal to C = Cau * f. Therefore relative cost of searching for F is equal the sum of fractions validated (in comparison to the initial training of (s)).
In Table 4l compare cost of running HEAD2ToE adaptation with FINETUNING. HEAD2TOE uses the backbone once in order to calculate the representations and then trains the Watt. whereas FINETUNING requires a forward pass the backbone at each step. Therefore calculate the cost
13

---

Under review as a conference paper at ICLR 2022
Dataset
Target Size
LR
Steps
f2,1 Coef.
Caltech101
768
0.050
0.01
5000
0.00100
CIFAR-100
768
0.020
0.01
500
0.00001
Clevr-Dist
15360
0.002
0.01
500
0.00100
Clevr-Count
768
0.050
0.10
5000
0.00001
Retinopathy
768
0.010
0.01
500
0.00100
DMLab
32448
0.005
0.01
500
0.00001
USpr-Orien
768
0.100
0.01
5000
0.00001
dSpr-Loc
32448
0.002
0.10
500
0.00100
DTD
768
0.100
0.01
500
0.00100
EuroSAT
768
0.100
0.01
500
0.00100
KITTI-Dist
32448
0.050
0.01
5000
0.00100
Flowers
768
0.020
0.01
500
0.00001
Pets
768
0.020
0.01
5000
0.00100
Camelyon
768
0.100
0.01
500
0.00100
Resisc45
768
0.050
0.01
5000
0.00001
SNORB-Azim
32448
0.010
0.01
500
0.00001
sNORB-Elev
15360
0.200
0.01
500
0.00100
Sun397
768
0.050
0.01
5000
0.00100
SVHN
32448
0.005
0.01
500
0.00001
Table 3: Hyper parameters selected for the VTAB-Ik benchmark tasks when using pretrained ViT- B/16 backbone. fraction features kept, LR: learning rate, Steps: Training Steps.
FLOPs
Size (vs FINETUNING)
Size (vs LINEAR)
Dataset
F
N
C
(vs FINETUNING
Caltech101
0.010
467688
102
0.013932
0,020750
2.353167
CIFAR-100 100
0.200
30440
100
0.002876
0.025743
2.977301
Clevr-Dist
0.001
467688
6
0.002808
0.000741
1.417419
Clevr-Count
0.005
30440
0.000270 8
0.000092
0.132278
Retinopathy
0.200
467688
0.002673 6
0.020531
47.099634
DMLab
0.020
467688
0,002808
0,003011
5,756287
dSpr-Orient
0.200
30440
0.002140 16
0.004183
3,001686
dSpr-Loc
0.005
467688
0.004154 16
0.002212
1.587624
DTD
0.005
1696552
0.023153 47
0.019157
4.692396
EuroSAT
0.100
30440
0.002088 10
0.001336
1.532776
KITTI-Dist
0.020
467688
0.002539
0.002215
6.350983
Flowers 102
0.100
30440
102
0.001094
0,013146
1.490882
Pets
0.002
467688
0.005181 37
0.002089
0,649417
Camelyon
0.020
30440
0.002018 2
0.000092
0.5291 14
Resisc45
0.020
467688
0.008058 45
0.018474
4.725480
SNORB-Azim
0.002
1696552
0.010791 18
0.004851
3.094923
SNORB-Elev
0.050
467688
9
0,003212
0.009578
12,210897
Sun397
0.100
30440
397
0.003678
0.049781
1.487498
SVHN
0.005
1696552
10
0,006884
0.005865
6.730334
Average
0.005282
0.010729
5.674742
Table 4: Relative cost of HEAD2ToE when compared with FINETUNING and LINEAR. F is the fraction of features kept, N is the total number of features and C' is the number of classes. On average HEAD2ToE requires 0.5% of FLOPs required by FINETUNING during the adaptation. Cost of storing each adapted model is also small: requiring only 1% of the FINETUNING and only 5.7x more than LINEAR. See main text for details on how the numbers are calculated.
of finetuning for t steps as C₁ t. Similarly, cost of HEAD2ToE is calculated as C, + Caul tt The overall relative cost of Call increases with number of classes C and number of features considered
14

---

Under review as a conference paper at ICLR 2022
N. As shown in Table topp HEAD2ToE obtains better results than FINETUNING, yet it requires 0.5% of the FLOPs needed on average.
After adaptation, all methods require roughly same number of FLOPs for inference due to all meth- ods using the same backbone. The cost of storing models for each task becomes important when the same pre-trained backbone is used for many different tasks. In Table 4 we compare the cost of storing different models found by different methods. finetuned model requires all weights to be stored which has the same cost as storing the original network, whereas LINEAR and HEAD2Tor requires storing only the output head: Winnerr HEAD2TOE also requires to store the indices of the features selected using a bitmap. Even though HEAD2TOE considers many more features during adaptation, it selects small subset of the features and thus requires much smaller final classifier (on average 1% of the FINETUNING). Note that hyper parameters are selected maximize aceu- racy, not the size of the final model. We expect to see greater savings with an efficiency oriented hyper-parament selection.
C STANDARD DEVIATIONS FOR TABLE
In Table we share the standard deviations for the median test accuracies presented in Table IT On average we observe LINEAR obtains lower variation expected, due to the limited adaptation and convex nature of the problem. Head2Toe seem have similar (or less) variation then the other regularization baselines that use all features.
Natural
Specialized
Structured
SNORB Azim
CIFAR- 100
Flowors 102
Retinopathy
Clevp-Coutt Count
KITT-Dist
NORR Bev
Caltech10
DTD
Pass
SVHN
Sun
Canelyon
EaroSAT
Resise45
CleveDiit
DMLab
dSp-Loc
dSp-Ori
Mean
Linear
0.09
0.08
0.14
0.06
0.08
0.06
0.06
0.03
0.12
0.08
0.42
0.1
0.03
0.21
0.14
0.07
0.21
+All-f2
0.09
0.78
0.44
0.29
0.22
0.02
0.04
0.02
0.06
0.05
0.02
0.09
0.2
0.1
1.31
0.32
0.08
0.03
0.35
0.24
+All-f1
0.14
0.11
0.11
0.11
0.11
0.12
0.03
0.06
0.04
0.2
0.02
0.47
0.38
0.18
0.2
0.34
0.07
0.06
0.44
0.17
+All-&g,1
0.09
1.0
0.13
0.1
0.1
0.18
0.23
0.15
0.07
0.09
0.05
0.03
0.08
0.11
0.41
0.51
0.24
0.16
0.07
0.2
Head2Toe
0.14
0.25
0.08
0.08
0.24
0.24
0.16
0.23 0.06 0.06
0.03
0.18
0.23
0.13
0.43
0.3
0.06
0.4
0.08
0.18
Natural
Specialized
Structured
CIFAR- 100
Cattech101
Flower1022
SVHN
Sun397
Camelyoo
EuroSAT
Resise45
Retinopathy
Clev--Cout
Clew-Dist
DMLab
KITT--ist
dSpcLoc
dSpr-Ori
SNORD-AZii Azim
SNORD- Hlev
DTD
Pets
Meun
Linear
0.1
0.23
0.19
0.16
0.06
0.06
0.08
0.09
0.08
0.06
0.0
0.05
0.02
0.07
0.92
0.21
0.08
0.08
0.03
0.14
+All-f2
0.13
0.17
0.0
0.66
0.36
0.04
0.56
0.02
0.59
0.01
0.04
0.23
0.13
0.12
1.24
0.28
0.09
0.84
0.2
0.3
+All-f1
0.06
0.11
0.15
0.08
0.19
0.31
0.08
0.08
0.06
0.12
0.05
0.28
0.17
0.23
0.87
0.34
0.07
0.22
0.28
0.2
+Al1-62.1
1.55
0.03
0.12
0.04
0.06
0.41
0.13
0.18
0.08
0.1
0.04
0.09
0.13
0.15
0.87
0.66
0.1
0.23
0.22
0.27
Head2Toe
0.29
0.16
0.26
0.5 0.19
0.14
0.07
0.13 0.04
0.09
0.09
0.04
0.44
0.14
1.34
0.21
0.01
0.31
0.08
0.24
Table 5: Standard deviation of test accuracy over 3 seeds on the VTAB-1k 1k benchmark using pre- trained (top) ResNet-50 and (bottom) ViT-B/16 backbones. The mean column averages the standard deviations for each dataset.
D ADDITIONAL PLOTS FOR EXPERIMENTS USING SINGLE ADDITIONAL LAYER
Test accuracies when using a single additional intermedieate layer from a pretrained ResNet-50 backbone are shown in Fig. 6 natural datasets (except SVHN) are highly similar to upstream dataset (lmageNet-2012) and thus adding an extra intermediate layer doesn't 't improve performance much. However performance OOD tasks (mostly of the tasks in structured category) improves significantly when intermediate representations are used, which correlates strongly with datasets in which HEAD2ToF exceeds FINETUNING performance.
15

---

Under review as a conference paper at ICLR 2022
C1
C CIFAR-100
Clevr-Oist
Clevr-Count
Retinopathy
Toat Aocuraty
Apcurayy
Tea Accuraty
Test Agcu Accuraer
Co
DMLab
dSpr-Orient
dSpr-Loc
DTD
EuroSAT
Test Accurayy
Test Accuracy
Test Accuracy
Tesk Accurasy
Test
8
KITTI-Dist
Flowers102
Pets
Camelyon
Resisc45
Text Arcurary
Test Accurayy
Test Accurccur eag
Accuraty
Test- www.Aucureeo
90
3NORB-Azim
SNOR3-Elev
Sun297
SVHN
Test Accuracy
Test
Figure 6: Test accuracies when using single additional intermedieate layer from pretrained ResNet-50 backbone.
E ADDITIONAL PLOTS FOR LAVER/FEATURE-WIS SELECTION COMPARISON
We compare layer-wise selection strategy discussed in Section B to HEAD2TOE in Fig. 7 For allmost all datasets, feature-wise selection produces better results. Retinopathy and Flowers- are the only two datasets where the layer-wise strategy performs better.
F ADDITIONAL PLOTS FOR RESNET-50 RESULTS
Improvement of HEAD2ToE over fine-tuning test accuracy for ResNet-50 backbone is shown in Fig. g Similar to earlier plots, observe a clear trend between being OOD and improvement in accuracy: HEAD2ToE obtains superior few-shot generalization for most of OOD tasks. We also share the distribution of features selected for each task in Fig. TO Since different tasks might have different number features selected, we normalize each plot such that bars add up to 1. Overall, fea- tures from later layers seem to be preferred more often. Early layers are preferred, especially for OOD tasks like Clevr and sNORB. We observe diverse set of distributions for selected features, which reflects the importance of selecting features from multiple layers. Even when distributions match, HEAD2ToE can select different features from the same layer for two different tasks. There- fore, next, we compare the indices of selected features directly to measure the diversity of features selected across tasks and seeds.
Transfer across Tasks
In Fig. left we investigate how features selected using task performs when evaluated on a task j. Each pixel represents the average accuracy over 3 seeds. For each column we subtract the diagonal term (i.e. self-transfer, i = j) obtain delta accuracy for each
16

---

Under review as a conference paper at ICLR 2022
Orient
Sur
Retingpathy
Test Accuracy
225
GA
SNOR3-Flev
DML
PuroSAT
Clevr-Count
Flwers1
0.50
04
63
62
0/2
Clevr-Dist
CTD
dSpr-Loc Loc
Resisc45
SVHN
Test Accurayy
0.
94
66
04
03
0.2
Camelyon
CIFAR1100
Pets
specialized
Caltech101
ponertt www.
Test Testrccurayy
8:00
0.2
6725
2.0
0.1
10
0.
12
12
structured
average
natural
Accuracy
type
feature-wise
1.0
layer wise
643
layer-wise (group)
19
12
12
TP
#Festures Selected
Figure 7: Test accuracies when varying the number of features selected for HEAD2TOE using a pretrained ResNet-50 backbone.
Count
10
NNOR--Aam
Azmm
natural
AH2T - AFinetuning
CRe
MY
structured
5
specialized
SVeins Tembb
Caltect101 101
0-
Sunag
Pe
Resisc45
-5
CIFA--100
-10
pr--Loc
-15
x
-40
-20
0
20
40
60
80
Domain Affinity
Figure 8: Accuracy improvement of HEAD2TOE compared to FINETUNING.
dataset. For most tasks, using a separate task for feature selection hurts performance. Results in Flowers-102 102 and Pets get better when other datasets like sNORB-Elee -eeee- is used. Crucially no single dataset (rows of Fig, leftt) seems to get best transfer, which highlights the importance of doing the feature selection during the adaptation (not beforehand). Note that, in practice (and in the literature) evaluation for each task is done in isolation, i.e. the data from other tasks are not available.
17

---

Under review as a conference paper at ICLR 2022
Transfer of Selected Features
Intersection of Features
Cattcc101
CITAR-100
Caltech111
Clev Dis
CIFAR-100
Feature selection
Cleyr-Count Relingpathy
Cleyr-Diet
Cee Counk
Retinopathy
0.8
DSir-Orient
DMLab
d5p Loc
dSpr-Or lent
DTD
dSpr-
EUroSAT
0.6
EuroSA
-3
KITT Dist
Pets
Flowers102 0102
Carne yon
Pets
ResisA5
Camelyon
SNORG Agm
SNORB-ER Eley
SNORS Azim
Sun
SNOR Bey
SMHN
Sun397
Catedhf
CIRIR do
Clew Dost
Cleer Cowr-Court Count
Rotinana by
DM hy
eSpr-Oner
dSp CnD
KI EuroSA
Camelyon
SNORS Reneiii
SNOWS hari sureer Swin
SVHN
Clev Dat
Clevn-Count
Relimonsaiy
DMiLa
dSer-D 15pr-Crient Crient
dSpi-Loc
1112
EuroS4i os4
kITh Dest
niewers power110z 3102
Pels
sINCRB AECRR-AU Auin
ENOME- Sine--Eee Sun Saa
Flowe
CIFAR-
Evaluation
Figure 9: (left) Change in accuracy when features selected for a different task used for adapta- tion. Most tasks get their best accuracy when the same task is also used for feature selection. (right) Intersection of features when selecting 2048 features from 29800 (same settings in Fig. 4 The intersection calculated as the fraction of features selected in two different runs. Values are aver- aged over pairs of runs (3 seeds for each datasets), except the diagonal terms for which remove identical pairs resulting in pairs.
Similarity features selected In Fig. right we investigate the intersection of features selected by HEAD2ToE. We select 2048 features for each task from the pool of 29800 features (same ex- periments as in Fig. center). For each task HEAD2ToE selects a different subset of features. calculate the fraction of features shared between each subset. For each target task we run seeds resulting in sets of features kept. When comparing the similarity across seeds (diagonal terms), we average over possible combinations among these sets. Similarly. when comparing two different datasets we average over possible combinations. Results show that features selected by comple- mentary tasks like dSprites-Loe and dSpr-OOreet (or sNORB tasks) have high overlap. Both dSprites and sNORB images are generated artificially and have similar characteristics. This is reflected in the heat-map (about 40% overlap). Similarly, features selected for CIFAR-100 and Caltech101 tasks seem to have high overlap.
Features selected also vary across different runs (seeds). This variation seem be higher for highly in-domain datasets like Flowers and Pets, which seems to correlate with HEAD2ToE's poor performance. Similarly dSprites-Loc seems to have higher variance across different seeds, which correlates with the outlier performance of HEAD2TOF on this task when compared with FINETUN- ING (see Fig. [8. We believe that understanding and reducing the inconsistency among different runs can be promising direction to improve head-to-toe utilization performance.
Apart from small fraction datasets, most datasets seem to share less than 20% the features, which highlights the importance of doing the feature selection for each target task separately.
Effect of Training Data In Fig. we compare the performance of HEAD2TOF, with other base- lines using reduced training data. Fraction (dy)=1 indicates original tasks with 1000 training sam- ples. For other fractions we calculate number of shots for each class as int(1000 fa/C) where C the number of classes and then sample new version of the task accordingly. For SUN-397 task, number of shots are capped at and thus fractions below 0.75 lead to 1-shot tasks and thus results are all the same. Overall we observe the performance of HEAD2ToE improves with amount of data available, possibly due to the reduced noise in feature selection.
G ADDITIONAL PLOTS FOR SCALING BEHAVIOUR OF HEAD2TOE
Scaling behaviour of HEAD2ToE when using different feature target size and number of layers over 19 VTABB- 1 tasks is shown in Fig. 12
18

---

Under review as a conference paper at ICLR 2022
KI KTTTID Dist
SNORB-Azim
SNORB-Elev
0.4
0.0
Clevr-Dis Dist-Dist
DMLab
dSpr-Loc
dSpr-Orient
0.4
0.0
EuroSAT
Resisc45
Retinopathy
Clevr-Count
0.2
Pets
Sun397
SVHN
Camelyon
0.4
0.0
Caltech101
CIFAR-100
DTD
Flowers1
0.4
0.0
Figure 10: Distribution of selected features over different ResNet-50 layers for VTAB- 1k tasks for results presented in Table We group the layers in each block (group of 3 layers) to reduce numbers of bars.
H DETAILS OF INTERMEDIATE REPRESENTATIONS INCLUDED
ResNet-50 has 5 stages. First stage includes single convolutional layer (root) followed by pool- ing. We include features after pooling. Remaining stages include 3,4,6 and bottleneck units (v2) each. Bottleneck units start with normalization and activation and has 3 convolutional layers. We includes features right after the activation function applied, resulting in intermediate feature sets per unit. Output after 5 stages are average-pooled and passed to the output head. We include features before and after pooling. with the output of the final layer (logits), total number of locations where features are extracted makes 52.
VIT-B/16 models consists of multiple encoder units. Each attention head consists of self attention module and followed by 2 MLPs. For each encoder unit, we include features (1) after layer-norm (but before self-attention) (2) features after self-attention (3-4) features after MLP layers (and after gelu activation function). Additionally we use patched image (i.e. tokenized image input), pre-logits and logits.
19

---

Under review as a conference paper at ICLR 2022
1a
CIAR-1100
Clevr-Dist
Clevr-Count
Recinopathy
0.00
56
krcurayy
0.8
04
0.6
05
0.4
5.
0.8
DMLab
dSpr-Onen Oninn
dspr-Loc
EurosAT
Teat Accurary
0.5
56
0.4
0.4
8400
85
0.80
02
KisTi-Dist
Flowers' 02
Pets
Camelyon
Resisc45
Arcurary
025
0.20
030
05
05
0.59
04
0,8
16
SNORB-AZim
SNORB-Elev
Sun397
SVHN
Test Accuracy
tye
Head2Toe
12,1
033
032
0.0
Linear
Fraction of Data
Figure 11: Effect of data available during training to the accuracy. Fraction=1 indicates original tasks with 1000 training samples. Overall we observe the performance of HEAD2ToE improves with amount of data available, possibly due the reduced noise in feature selection.
20

---

Under review as a conference paper at ICLR 2022
C1
CIFAR-100
Clevr-Diss
Clevr-Cound Count
Retinopathy
045
155
188tt Size
6.97
0/4
0.0
555
030
545
079
042
0.0
0750
10
DMLab
DSpr-Crient
d'sr--Loc
DTD
EuroSAT
5.5
Target Size
0915
060
6.60
0.0
6.42
0595 122
040
103
10P
30
10
KITTI- Dist
Flowers 102
Pets
Camelyon
Resisc45
Target Size
6.36
0.5
0.2
0.70
2000
C15
0.72
16
SNORB-Azim
SNCBB- Elev
Sun397
SVHN
D0
n_aayers
029
Target
1/16
030
028
027
1/4
05
028
all
123
# #Target Features Per Layer
Caltech101
CIFAR-100
Clevr-Dis
Clevr-Count
Retinopathy
025
Terget Size
0.575
8678
5.00
0.52
6400
034
0255-
0/25
073
0.500
0.07
LM
d dSpr-Oree Orient
dSpr-Loc
DTD
EuroSAT
0.30
3.30
030
Target
030
0.20
030
10
10
0.24
0.35
V24
100
TM
0.2
[64
KistI-Dist
Flowers 102
Pets
Camelyon
Resisc45
020
884
0.80
0.2
Target
882
025
0.55
0.53
0.5
030
10'
10"
1CT
127
SNORB-Azim
SNORB-Elev
Sun397
SVHN
081
Size
Size
0.38
033
87.
layers
1/16
66
1/4
10
0.26
86
184
all
# #Tagget Features Per Layer
Figure 12: Scaling Behaviour over 19 VTAB-Ik tasks when varying feature target size and number of layers utilized for (top) ResNer-50 and (bottom) ViT-B/16
21


Reviews:
[{"review_id": "ajXHtaoG5T7", "reviewer": "Reviewer_VhK8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, an interesting paper. The proposal is indeed interesting, although utilizing intermediate representations has been seen in the past.\nMy main concern regarding authors claim \"We conjecture that FINETUNING\nbetter leverages existing internal representations rather than discovering entirely new representations; FINETUNING exposes existing features buried deep in the net for use by the classifier\". Even if Head2Toe manages to reach comparable performance to finetunning, that doesn't proves that existing good representations where present in intermediate layers and where \"brought up\".  An analysis on the resulting weights of both methods might be a better way to prove that.\nIn addion, Table 1 shows a gap between the best performance reached by Head2toe and the best performance using finetunning. That result seems to imply that is more than a clever usage of existing representations what finetunning the whole model is doing.\n", "summary_of_the_paper": "The main focus of this paper is the study the use of intermediate layers in a deep pretrained model on downstream tasks.\nAuthors argue that the the fact that the good performance while fine tuning on downstream tasks even if data scarce is due to the prior existence of useful representations deep in the model which are brought up during training.\nAuthors propose a new approach. Head2toe, which consist in utilizing intermediate representations, while freezing the weights of the network for training for downstream tasks\nExperimentation is carried out starting with a ResNet-50 and ViT-B/16 models on a variety of datasets collected on the VTAB collection, showing that the approach where Head2toe  outperforms linear finetunning (training a classification head on top of the model, while freezing the rest of the weights) and matches the performance of finetunning the whole model.", "summary_of_the_review": "An interesting idea,\nNumbers hold\nClaims are not completely justified with experimentation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZKIs-5CAz2v", "reviewer": "Reviewer_kz8L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Evaluation\n\nThis is a well-written paper with a simple but clearly motivated idea. The preliminary experiments and discussion are helpful in motivating the approach. I am not familiar with the benchmark chosen for experimentation, but the experiments appear to be sound as far as I can assess. There is a comparison with several reasonable baselines. \n\n\n# Questions and comments \n\n1. While I'm not very familiar with the transfer learning literature in image classification, the idea seems common enough (as discussed in the related work) and pretty straightforward. This isn't necessarily a shortcoming, but I think it would make sense to discuss similar work in more detail, and perhaps draw more direct comparisons. Others have studied intermediate representations -- have they used the exact same approach? A different one? How would the results of the present approach compare to others that use intermediate representations? What exactly is the novelty? \n\n2. The name Probing is not so intuitive. I'm familiar with this term from the interpretability literature, especially in NLP, where a probing classifier is used on frozen features to assess their quality w.r.t some property. In the present context, however, the goals are different, and a more familiar term (at least to me) is to refer to frozen transfer, or feature-based transfer learning as opposed to fine-tuning. \n\n3. In figure 2, what is Scratch trained on? If it's a randomly-initialized ResNet-50 trained on low-resource datasets, it seems like a problematic case of over-parameterization. Would a simpler network perform better? the comparison with Linear is maybe problematic here. \n\n4. The Control reported in section 3.1 is helpful in assessing the extent of improvement that can be expected from adding intermediate representations. Oracle is than 1% greater than Control on average. \n\n5. The group Lasso regularization is applied once, and then a second final classifier is trained on the remaining features. I'm confused as to why this retraining is necessary, and why the group-lasso-regularized classifier can't be used as is. The experiments show that this second step works better, but why would it be so? Is it basically an optimization problem? \n\n6. In group Lasso, it's important to a-priori select the groups. In one case, features are grouped by layer, but I'm confused about the \"feature-wise strategy\". What are the groups there? And, would it make sense to consider other groupings? \n\n7. Related to the above, [1] performed feature selection with intermediate representations in an NLP context, and found different layers and different features inside a layer. They ahd a different kind of regularization, the ElasticNet. Even though their motivation is different (analyzing redundancy), they still get some pretty good results, so it would be interesting to compare the approaches. \n\n8. In table 1, I would have liked to know which differences are statistically significant and which aren't. The text mentions standard deviations as appearing in Appendix 1, but I couldn't find them. \n\n9. Could you discuss the different performances on the three groups (natural, structured specialized), of Head2Toe vs fine-tuning, in ResNet and ViT cases? Do the differences make sense in light of the nature of these categories? \n\n10. From an interpretability standpoint, I would have liked to see more discussion of which features are selected, in which layers, and how this relates to different tasks and what we might expect. \n\n[1] Dalvi et al., 2020. Analyzing Redundancy in Pretrained Transformer Models. \n", "summary_of_the_paper": "# Summary\n\nThis paper proposes a method for using intermediate representations from a pre-trained model for better transfer to other tasks. A linear classifier is trained on features from multiple layers. A feature selection strategy is chosen, where first a linear classifier is trained on all features with group-lasso regularization, and then another classifier is trained only on features whose regularization score exceeds some threshold. This approach is evaluated in a low-resource transfer scenario of image classification, and compared both to training on all features without the two-step feature selection but with regularization, and to the standard fine-tuning approach. The proposed method works slightly better than fine-tuning with a ResNet model, and slightly worse with a ViT model, although the comparison is not completely fair, as discussed. There are different performances on different categories of images. \n\n", "summary_of_the_review": "# Evaluation\n\nThis is a well-written paper with a simple but clearly motivated idea. The preliminary experiments and discussion are helpful in motivating the approach. I am not familiar with the benchmark chosen for experimentation, but the experiments appear to be sound as far as I can assess. There is a comparison with several reasonable baselines. See the main review for more questions and comments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GPUsaX7-9q9", "reviewer": "Reviewer_yqBZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well-written and easy to follow. \n- The overall idea of Head2Toe for transfer learning is interesting.\n- Competitive performance on VTAB benchmark.\n\nWeaknesses: While the main idea and experiments presented in the paper are very interesting, there are several important weaknesses in the paper that need to be throughly addressed to improve the quality of the work (mainly missing experiments, analysis and comparisons).\n\n- While authors empirically demonstrate the utility of intermediate features in experiments, there are no feature visualizations to support the claims. How about visualizing the selected intermediate features per target task? Is it possible to analyze the weight matrix W across different target datasets to better understand what type of features being selected for transfer learning? Are the selected features complementary go each other?\n\n- In current experiments, features are selected specific to one target task. How about selecting a common set of intermediate features for all the tasks? How is that global baseline comparable to individual selecting selection of features per task?\n\n- I would suggest the authors to rearrange or rewrite the main contributions of the paper. Points 1-5 in introduction section are not really the contributions. In other words, authors should merge them into 2-3 key contributions along the directions such as, problem definition, methodology and experiments.\n\n- How the proposed method comparable to finetuning on other tasks besides image classification, e.g., video classification? I believe the main conclusions will still be the same but a small experiment on this will strengthen the paper.\n\n- Authors compare with layer wise selection by computing the mean relevance score of each feature in a layer. Can it be not done through the L21 optimization directly where group of features belong to one layer will be made zero?\n\n- Authors mention that one of the properties of the proposed method is key for transfer-learning methods to be practical with billions of parameters such as GPT-3. I wonder how is the proposed method applicable to transfer learning in NLP. Experiments and analysis on text classification or other standard NLP tasks should be included in the paper to validate the effectiveness of the proposed method beyond image classification.\n\n- Is it possible to include some of the tweaks used in ViT paper to further improve the performance of Head2Toe (say Head2Toe+), e.g., different aggregation schemes as authors mentioned in the main paper?\n\n- How is the proposed method comparable to Linear probing and Finetuning with different percentages of labeled data in the target domain? Is it still effective in low data regime, e.g., 1% of labeled samples?\n\n- How is this method comparable to existing transfer learning methods besides Finetuning on VTAB benchmark, e.g., selective finetuning or some of the methods mentioned by the authors in the related work?\n", "summary_of_the_paper": "This paper explores the utility of intermediate layers for linear probing in transfer learning. Specifically, authors proposed Head-to-Toe\nprobing (HEAD2TOE), that selects features from all layers of the source model to train a classification head for the target-domain. Experiments on the VTAB benchmark shows that Head2Toe matches performance obtained with fine-tuning on average, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning.", "summary_of_the_review": "Given the interesting idea and competitive performance, my initial recommendation is borderline accept. I would like to see authors response on new experiments and discussions during the rebuttal as I think the experiments are limited and somewhat unconvincing in the current version of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DA5FmAnIhQ", "reviewer": "Reviewer_nxiA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper investigates how to use pre-trained models for better transfer learning, which is an important problem.\n2. The paper is generally well-written and easy to follow. \n3. The proposed method is simple, technically sound, and makes sense. \n4. The authors conducted extensive experiments on VTAB benchmark with both ResNet and ViT backbones, and conducted comparisons with several baselines. The overall performance of Head2Toe shows some improvement compared to linear prob and even fine-tuning.\n5. The preliminary experiments and analysis is informative and helps the readers understand to motivation of the paper.\n\nWeaknesses:\n1. The idea of combining intermediate representation from a pre-trained model to improve fine-tuning (transfer) performance is already exploited by ELMo. They use a learnable weight to control the relative importance of features from different layers. The key difference between Head2Toe and ELMo's approach is that Head2Toe selects a subset of intermediate features after learning the weights. However, I'm not sure this makes the novelty, or technical contribution, enough.\n2. The experimental results is not convincing enough. In the VTAB benchmark, in natural and structured category, the performance of Head2Toe is only comparable with linear prob but worse than fine-tuning. The overall improvement of Head2Toe is because it performs well on Specialized category and it has more tasks. It will be better if some analysis on why Head2Toe perform particularly on Specialized category but not on others.\n3. The claim in the title and abstract that Head2Toe improve OOD generalization are not well supported by the experimental results. The main experiments only show that Head2Toe improves transfer performance. However, the fine-tuned models' performance on OOD samples is not tested.", "summary_of_the_paper": "This paper propose Head2Toe, a method that exploits intermediate representations of DNNs to improve performance, and OOD generalization of transfer learning. The key idea of the Head2Toe is to augment traditional linear probing with intermediate representations. It uses group lasso as regularization to learn weights of different features, and then select features based on the learned weights and validation performance.Experiments on VTAB benchmark show that Head2Toe outperforms linear probes and is competitive compared with fine-tuning. An important finding is that Head2Toe improves OOD generalization.", "summary_of_the_review": "The paper is generally well-written and the method makes sense. However, given that the novelty of the method and some concerns about experimental results, I think the paper, in its current form, is slightly below the bar. I believe this paper can be improved largely if theconcerns about experiments are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the usefulness of intermediate layers for linear probing, aiming at improving out-of-distribution transfer with significantly less cost than fine-tuning. Two reviewers recommended borderline acceptance, while two others recommended borderline rejection as final rating. The main concerns raised by the reviewers were the limited novelty of the proposed method (e.g., compared to Elmo), unconvincing results in the natural and structure categories of VTAB, and lack of experiments to justify the claims, as well as the demonstration of the method in other tasks beyond image classification. The rebuttal has clarified several other questions. The AC really likes the simplicity of the approach, and also finds the problem of improving the efficiency of transfer learning very important. In addition, the paper is very well-written and easy to follow, as acknowledged by all reviewers. However, the AC agrees with R2 and R3 that the paper, in its current form, does not pass the bar of ICLR, unfortunately. First, the novelty is limited, as pointed out by R1, R2, and R3. In addition to the related works mentioned in the reviews like Elmo, note that the idea of selecting intermediate features, concatenating them, and running a linear classifier for OOD transfer has also been explored in [Yunhui Guo et al, A broader study of cross-domain few-shot learning, ECCV 2020]. Second, while the approach has advantages in terms of efficiency, the accuracy drop (compared to fine-tuning) for in-domain tasks limits its applicability. Finally, even though the AC agrees with the authors this is not a requirement, a more comprehensive set of experiments on more tasks would make the paper stronger, especially given that the novelty is incremental. The authors are encouraged to improve the paper for another top conference.
Decision: Reject