Towards Estimating Transferability using Hard Subsets
Authors: Tarun Ram Menta
Surgan Jandial
Akash Patil
Vimal K B
Saketh Bachu
Balaji Krishnamurthy
Vineeth N. Balasubramanian
Chirag Agarwal
Mausoom Sarkar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transfer Learning
Transferability
Hard Subsets

Abstract: As transfer learning techniques are increasingly used to transfer knowledge from the source model to the target task, it becomes important to quantify which source models are suitable for a given target task without performing computationally expensive fine-tuning. In this work, we propose HASTE (HArd Subset TransfErability), a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. By leveraging the model’s internal and output representations, we introduce two techniques – one class-agnostic and another class-specific – to identify harder subsets and show that HASTE can be used with any existing transferability metric to improve their reliability. We further analyze the relation between HASTE and the optimal average log-likelihood as well as negative conditional entropy and empirically validate our theoretical bounds. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE-modified metrics are consistently better or on par with the state-of-the-art transferability metrics.

Under review as a conference paper at ICLR 2023
TOWARDS ESTIMATING TRANSFERABILITY USING HARD SUBSETS
Anonymous authors Paper under double-bling review
ABSTRACT
As transfer learning techniques are increasingly used to transfer knowledge from the source model the target task, becomes important to quantify which source models are suitable for a given target task without performing computationally expensive fine-tuning.
this work, we propose HASTE (HArd Subset TransfErabillty), new strategy to estimate the transferabilityy of source model to a particular target task using only a harder subset of target data. By leveraging the model's internal and output representations, we introduce two techniques - one class-agnostic and another class-specific to identify harder subsets and show that HASTE can be used with any existing transferability metric to improve their reliability. We further analyze the relation between HASTE and the optimal average log-likelihood as well negative conditional entropy and empirically validate our theoretical bounds. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE-modified metries are consistently better or on par with the itate-of-the transferability metrics. Our code is available here
INTRODUCTION
Transfer learning (Pan Yang. 2009 Torrey & Shavlik 2010 Weiss al. 2016 aims improve
the performance of models on target tasks by utilizing the knowledge from source tasks. With the increasing development of large-scale pre-trained models (Devlin et al.) 2019: Chen al. [220aa bj Radford et al., 202 and the availability of multiple model choices (e.g model hubs of Pytorch, Tensorflow, Hugging Face) for transfer learning, it is critical to estimate their transferability without training on the target task and determine how effectively transfer learning algorithms will transfer knowledge from the source to the target task. To this end, transferability estimation metrics (Zamir et al. 2018b Achille et al. 2019 Tran et al.) 2019b] Pándy et al., 2022 Nguyen et al.) 2020 have been recently proposed to quantify how easy it is to use the knowledge learned from these models with minimal to no additional training using the target dataset. Given multiple traitrained source models and target datasets, estimating transferability is essential because is non-trivial to determine which source model transfers best to target dataset, and that training multiple models using all source-target combinations can be computationally expensive.
Recent years have seen few different approaches (Zamir et al., 2018b) Achille al. 2019 Tran et al. 2019b; Pándy et al. 2022, Nguyen et al.) 2020) for estimating a given transfer learning task Trom source model. However, existing such methods often require performing the transfer learn- ing task for parameter optimization (Achille et al. 2019 Zamir et al.] 2018b) or making strong assumptions on the source and target datasets (Tran et al. 2019b 2 Zamir 20188b) In addition, they are limited to estimating transferability
specific source architectures (Pándy et al. (022) or
achieve lower performance when there are large domain differences between the source and target dataset (Nguyen et al., 2020). This has recently led to the questioning of the applicability of such metrics beyond specific settings (Agostinelli et al. 2022a).
Prior works in other contexts KKaan al. 2018: Agarwal et al. 2022 Zhang al. 2021b Khan et al. 2018 Soviany et al., 2022 D souza et al. 2021 show that machine learning (ML) models find much harder. In this work, observe and leverage a similar phenomenon in transfer learning tasks (Figure Ia), where images belonging to the harder subset of the target dataset achieve lower prediction accuracy than images from the easy subset. The
some samples easier to learn while others

---

Under review as a conference paper at ICLR 2023
70
Accurac
HARD EASY
30
Reshet-18 VGG-19
30
STOP
LACO STOP
STOP
[Hardest)
[Fasies))
Bins
(a) Transfer Accuracy
(b) Top- 10 images from Easy and Hard subsets
Figure 1: Analyzing the impact of hard subsets in transfer learning. Column (a): Results show the accuracy of different bins a target dataset (Caltech101) based on their hardness. Across two source models (VGG-19 and ResNet-18) trained on the ImageNet dataset, we observe that the accuracy for images in the hardest subset (B1) is lower as compared to the easier subset (B5). Column (b): Top-10 10 images from hard and easy subsets show that harder subsets comprise images (cliparts) that are dut-of-distribution when compared to the source dataset images. See Figures for more qualitative images for different souree-target pairs.
key principle is that easy samples do not contribute much when comparing the performance of a pre-trained model on multiple datasets or ranking the performance of different models on given dataset. Additionally, in Figure Tb we observe qualitatively that easy examples of the target dataset (Caltech 101) comprise images that are in-distribution as compared to the source dataset (ImageNet), whereas images from the harder subset contain -fiit--iittrittiion clip art images that are not present in the source dataset and, hence, may be more challenging in the transfer learning process.
Present work. In this work, we incorporate the aforementioned observation and propose a novel framework, HASTEL (HArd Subset TranstErability), to estimate transferability by only using the hardest subset of the target dataset. More specifically, we introduce two complementary techniques class-agnostic and class-specifer model's internal and output representations (Section 4.1). Further, we theoretically and empirically show that HASTE transferability metrics inherit the properties of its baseline metric and achieve tighter lower and upper bounds (Section 4.2)
to identify harder subsets from the target dataset using the
We perform experiments across a range of transfer learning tasks like source architecture selection (Section 5.1). target dataset selection (Section (.2). and ensemble model selection (Section 5..4.
as well as on other tasks such as semantic segmentation (Section [5.3) and language models (Sec- tion (5.5). Our results show that HASTE scores better correlate with the actual transfer accuracy than their corresponding counterparts (Nguyen et al. 2020 Tran et al., 2019a Pándy et al. 2022)
Finally, we establish that our findings are agnostic to the choice of source architecture for identify- ing harder subsets, scale to transfer learning tasks for different data domains and that utilizing the hardest subsets can be highly beneficial for estimating transfcrability.
RELATED WORK
This work lies at the intersection of transfer learning and diverse metrics to estimate transfcrability from a source model to a target dataset. We discuss related works for each of these topics below.
Transfer Learning (TL). It can be organized into three broad categories: i) Inductive Transfer han et al. 2010 Yosinski et al. 2014), which leverages inductive bias, ii) Transductive Transfer,
which is commonly known as Domain Adaptation ( (Wang & Deng 2018 Wilson & Cook] 2022).
and iii) Task Transfer [Zamir et al. 2018a Pal & Balasubramanian 2019), which transfers between different tasks instead of models. Amongst this, the most common form of transfer learning task is fine-tuning a pre-trained source model for a given target dataset. For instance, recent works have demonstrated the use of large-scale pre-trained models such as CLIP (Radford et al. 2021a) and VirTex (Desai & Johnson, 2021) for learning representations for different source tasks.
Transferability Metries. Despite the development of plethora of source models, achieving an optimal transfer for a given target task is still a nascent research area as it is non-trivial to identify the source model or dataset for efficient TL. Transferability metries are used proxy scores to
Cooe: http://aamyyymooss.oooesssscccrr
2

---

Under review as a conference paper at ICLR 2023
estimate the transferability from source to target task. Prior works have proposed diverse metrics to estimate TL accuracy. For instance, NCE (Tran et al. 2019a) and LEEP (guuyen et al. 2020)
utilize the labels in the source and target task domains to estimate transferability. Further, metrics like H-Score (Bao et al.j 2019), GBC (Pándy et al. 2022) and TransRate Huang et al. (2022) use the embeddings from the source model to estimate transferability. In contrast the above metrics that focus on a single source model, Agostinellli et al. (2022b) explored metrics to estimate the transferability for an ensemble of models and introduced two metrics MS-LEEP and E-LEEP for identifying a subset of model ensembles from the pool of available source models.
3 PRELIMINARIES
Notations. Let a transfer learning task comprise of pre-trained source model fg trained on a source dataset D.= (-=(Dumim, Dir"), }. and a target dataset (D Distt } for transfer learning. We define target model /ū which is initialized using the source model weights and fine-tuned the target dataset Di The performance of the target model fØ is quantified using the target model accuracy AA when evaluated on the unseen target test dataset Di Despite fine-tuning the source model, training target models is computationally expensive. Hence, we define a transferability met- ric 7 which correlates with the target model accuracy A and gives an efficient estimation of how the transfer learning will unfold for given pair of source model and target dataset.
Probability Estimations. Let the source model fÿ output softmax scores over the source dataset label space Z. Next, we construct a "source label distribution" of the target dataset over the source label space Z by passing them through f and use it to build an empirical joint distribution over the source and target label spaces, i.c., P(y. 2) 1 softmax score of an instance x; for class = E Z. Finally, the empirical marginal distribution and conditional distribution can be computed using P(=)=Ene>> P(y, 2) and P(yl=)= P(u.z), where y P(+)
Eyy fă (x,)2. where fØ (xx)) represents the
denotes the target label space for dataset Dr.
Transferability Metric. Following prior works, the performance of a transferability metric is eval- uated by measuring the correlation between T and A Further, we focus on the fine-tuning style of transfer learning. Here, the final source classification layer of the source model fā is re- placed with the target classification layer, and the whole model is trained on the target dataset task.
OUR METHOD: HASTE
Next, we describe, HASTE, a meta-transferabilicy metric, which improves the transferability es- timates by leveraging the harder subsets of the target data. We first discuss two complementary techniques (class-agnostic and class-se,,ific; Sec. [4.] identify harder subsets and show that they can be applied to any of the existing transferability metrics. Next, we theoretically and empirically show that HASTE transferability metrics inherit the properties of their baseline metric (Sec. (.2).
Problem Statement (Transferability Metric). Güven a source model få. source dataset DD., and target dataset Dr, a transferabiliiyy metric aims to output a score T that correlates with the accuracy -- of the target model fā
4.1 CALCULATING HARDNESS
Here, we define the methods for identifying harder subsets in the target dataset, where one method uses the overall data distribution, and the other controls individual samples/classes to provide more representation of the dataset.
Class- Class-Agnostic Method. The Class-Agnostic method uses the representation similarity between the samples in the source and target dataset to compute hardness scores. In particular, it employs embeddings from multiple layers of the source model and compares them for the source and target samples using cosine similarity, i.c.
e(x; = Σ E(x)) &(x)),
(1)
3

---

Under review as a conference paper at ICLR 2023
Algorithm HASTE
Require:
Source model fÿ. Source dataset Dz, Target dataset Dr, hard subset variant he Transfer-
ability Metric
k + Number of samples in hard subset
if h "ca' then
Collect source dataset activations (x;)
P Class-Agnostic Case
Collect target dataset activations (xx))
Compute similarity matrix Sij < (x x
D As per Eqn.
Compute Hardness (x))) for each target image using Sij
else if h, "cs' then Collect target dataset activations (x))
Class-Specific Case
for class с E DD do
Compute Mei Ee end for
D As per Eqn.
Compute Hardness (x))) for each target image using Mahalanobis Distance from μic.
end Dhard 4- (k hardest samples ordered by H())
return T(ā Dhar)
where & represents the similarity between a pair of source x; and target x} sample, Et(-) is the intermediate layer output from the 1-th layer of fä, and L the total number of layers in fā Next, calculate activation similarity matrix S RMxN, where Sx= x5). M- and 8 -= Using the pairwise similarity matrix S, we compute the hardness score for a target image, where samples closer to the source dataset obtain lower hardness scores, and vice-versa.
H(x))ca=1
1 M
Σ S
Class-Speeific Method. In contrast to the class- strategy, which does not utilize label infor- mation of the target dataset, we introduce a Class-Specifer technique to identify harder subsets by controlling the target classes (as they provide more representation of the dataset). Following Pannyy jet al. (2022), we model each target class e as a normal distribution in the embedding space of Jă and define the mean and covariance of the distribution as:
(e(()) E. Me No b:y/=c
(( po)(((a.))) -Me)) 1:y;=c
No
where -c, and Nr is the number of samples in class c. For each target sample, the hardness is defined as the Mahalanobis distance of the sample from the mean of the corresponding class distribution (Equation 3 i.e.
U(x))o== V =z) ( pe)
(4)
Next, we use the above-mentioned techniques identify hard subsets from the target dataset.
HASTE. To identify hard subsets, we sort the target dataset samples using either of the hard-
ness scores, as defined in Equations 4 We denote the indices of the sorted samples using (g) (un44,.... .2,,...N The hard subset is then defined as: Diard ((x yi ....),, (x! where the hardness of each sample follows H (x )> )2 ... H (x For using our HASTE modification with the existing metrics, we propose the use of only these identified harder subsets Dhard an input to these metrics for estimating transferability, i.e.
y }) < N.
(5) where T--) denotes any existing transferability metric. See Algorithm[ details getting harder subsets using Class Specific or Class Agnostic methods. Finally, we show the t-SNE (van der Maaten & Hinton! 2008) embeddings (Figure 2 of the entire target dataset and their harder subsets. We
HASTE T(J Dhard).
4

---

Under review as a conference paper at ICLR 2023
40
48
30
30-
30-
20
10
10-
10-
-10
-10
-10
-30
-30
-30
-20
-2
-1
20
30 40
-30
-20
-10
20 30 40
30
10 20 20 40
(a) Dogs Full Dataset
(b) Hard Subset CA
(c) Hard Dataset CS
Figure 2: t-SNE embeddings of the entire target dataset and its hardest subset using a ResNet- 50 source model trained on ImageNet. We show the embeddings from five random classes from Stanford Dogs as the target dataset, using both the class-agnostic (b) and class-specific (c) methods. We observe that the embeddings from the harder subset are more entangled than the entire dataset.
observe that the embeddings from the entire dataset (Figure 2a are well segregated, but embeddings of samples in the harder subsets are highly entangled (Figure 2b 2c These findings align with our findings in Figure [ where images from harder subsets achieve lower transfer accuracies, i.e., the source models struggle to find the decision boundaries between these harder samples.
4.2 THEORETICAL PROPERTIES
Here, we show that HASTE-modified metrics inherit the theoretical properties of their baseline met- ric. Note that showing theoretical bounds for all transferability metrics is outside the scope of this work. Hence, we take one representative metric (LEEP) and show that HASTE-LEEP retains its theoretical properties.
LEEP. Let source model fä predict the target label y by directly drawing from the label distribution ply|x; x; fă. D E.Ez ((v)=),f(x)== The LEEP score is then defined as average fog-likelihood.
1 f=1
LEEP T(ā. Dr
log
((x))))((x).)
#E2
Average log-likelihood. We fix the source model weights 0 and re-train the classification model using maximum likelihood and the target dataset Dirain to obtain new classifier Jă i.e.,
fø arg max I(0,k:)
kEK
where 1(0, k)) the average likelihood for the weights and on the target dataset Dirain and is selected from space of classifiers K.
Lemma 1. HASTE-LEEP a lower bound of the optimal average log-likelihood for the hard subset.
)uond T(Jā, Dhand < 1(w, k. hond < I(w, k*)
Proof. This proof true by definition as Dhuni Ditain represents the hard subset of the target dataset.
Note that l(w, k*) is the maximal average log-likelihood over lo E K, and T T(. Dirain) ) is the average log-likelihode in K. From Nguyen et al. (202)) we know T(f;, Dirain) < l(w, k*) and by definition of Dhard, T(J;, Dhard) S I(w, k*). In addition, the model struggles to learn the samples in the hard subset, and hence I(w, )arh < [(w,k")
Lemma 2. HASTE-LEEP is an upper bound of the NCE measure plus the average log-likelihood of the source label distribution, computed over the hard subset, i.e.,
T(fā Dhard) 2 H-NCE(Y ()) DDhund
log fo fx1)
(9)
Proof Sketch. This proof extends from the property of LEEP. See Appendix A.l for detailed proof.
5

---

Under review as a conference paper at ICLR 2023
HASTE LEEP
LEEP
NCE Average Log Likelhead
Optimal Log Ukelhood
Hard-NC Hard Uikellood
Optimal Ukelhood- Hard
-0.6
-12
RN-152
Architectures
Figure 3: Empirical results on the StanfordDogs target dataset show no violations of our theoretical bounds. Empirically calculated HASTE-LEEP (in blue) and our theoretical upper (in purple) and lower (in green) bounds from Equations 8 across seven source model architectures trained on the ImageNet dataset, where RN ResNet, MN MobileNet, DN DenseNet.
Empirical Analysis. We analytically evaluated the upper and lower bounds for HASTE-LEEP by computing the RHS of Equations 9 Figure 3 our results show HASTE-LEEP and its corre- sponding theoretical upper and lower bounds, confirming that, across seven source model architec- tures, none of our theoretical bounds are violated. In addition, we empirically demonstrate that our bounds are tighter than LEEP.
EXPERIMENTS
Next, we present experimental results to show the effectiveness of HASTE modified transferability metrics for different transfer learning tasks, including source architecture selection (Sec. (Sec.5.1)) tar- get dataset selection (Sec. 5.2). semantic segmentation task (Sec. 5.3 ensemble model selection (Sec. 5.4), and language domain transferability (Sec. 5.5).
Evaluation metrics and Baselines. We use the Pearson Correlation Coefficient (PCC) for corre- lation between T and A (see Tables 11 for results of other correlation coefficients). For baselines, we use LEEP, NCE, and GBC for single model transferability tasks, and MS-LEEP and E-LEEP for ensemble model selection. See Appendix A.2 for more details.
5.1 SOURCE ARCHITECTURE SELECTION
Experimental setup. As detailed inpándy et al. 2022). the target dataset is fixed and the T are computed over multiple source architectures. The correlation scores are computed between 7 and the transfer accuracies _4=+t__ We consider seven target datasets for our source architecture ex- periments: i) Caltech101 (Fei-Fei al. 2004). ii) CUB200 (Welinder al. 2010), iii) Oxford-IIIT
Pets (Parki et al. 2012), iv) Flowers 122 (Nilsback & Zisserman 2008) v) Stanford Dogs (Khosla al. 2011), vi) Imagenette (Howard), and vii) PACS-Sketch (Li et al. 2017).
Model architectures and Training. We consider seven source architectures pre-trained on Ima- geNet (Russakovsky et al. 2013) dataset, including ResNet-50. ResNet-101, ResNet- (He al.) 2016). 121, DenseNer-169. DenseNet-201 (Huang et 2017). and MobileNetV? (San-) dler et al. 2018). All models were set using the publicly available pre-trained weights from the Torchvision library (Marcel & Rodriguez, 2010). For each source architecture,
utilize the ResNet-50 model to calculate the hardness ranking, and the hard subset used to compute HASTE scores. Following Pándy et al. (2022), we calculate the target accuracy A by fine-tuning the source model on each target dataset. We fine-tune the source model for 100 epochs using an SGD optimizer with momentum of 0.9, learning rate of 10-1, and batch size of 64.
Results. On average, across seven target datasets, results show an improvement in correlation scores of + 1+12,7749 for LEEP, +29.38% for NCE, and -0.07% GBC, using HASTE-modifies metrics (Ta- ble [. Interestingly, for most target datasets, both CA and CS variants of the HASTE metrics out- perform the baseline scores.
5.2 TARGET DATASET SELECTION
Experimental setup. Here, the source model is fixed and the transferability metric computed over multiple target datasets (Nguyen et al. 2020). We construct 50 target datasets by randomly selecting

---

Under review as a conference paper at ICLR 2023
Table 1: Results on source architecture selection task. Shown are correlation scores (higher the bet- ter) computed across all source architectures trained on ImageNet. Results where HASTE modified metries perform better than their counterparts are in bold.
LEEP
HASTE-NCE
HASTE-GBC
Target (Dr)
LEEP
NCE
GBC
CA
CS
CA
CS
CA
CS
CUB200
0.534
0.405
0.667
0.330
0.040
0.500
0.790
0,811
0.785
StanfordDogs
0.926
0.943
0.931
0.930
0.924
0.955
0.784
0.944
0.834
Flowers 102 102
0.504
0.508
0.723
0.382
0.390
0.388
-0.012
-0.013
-0.02
Oxford-IITT
0.921
0.952
0.927
0.846
0.851
0.916
0.668
0.867
0.745
1altech101
0.416
0.439
0.458
0.204
0.461
0.504
0.810
0.793
0.821
Imagenette
0.950
0.950
0.962
0.927
0.940
0.889
0.709
0.723
0.711
PACS-Sketch
-0.029
0.196
0.253
-0.129
0.160
-0.208
0.612
0.637
0.601
Table Results on target task selection using the tunint method for Caltech101 source models.
Shown are correlation scores (higher the better) computed across all target datasets. Results, where HASTE modified metrics perform better than their counterparts, are bold. See Table for results on CUB200 source models.
HASTE-LEEP
HASTE-NCE
HASTE-GBC
Target (Dt)
LEEP
NCE
GBC
CA
CS
CA
CS
CA
CS
CUB200
0.948
0.950
0.948
0.944
0,948
0.944
0.916
0.917
0.916
Flowers 102
0.769
0.820
0.761
0.762
0.823
0.758
0.743
0.742
0.727
StanfordDogs
0.884
0.901
0.884
0.885
0.899
0.886
0.873
0.876
0.856
Oxford-III
0.899
0.907
0.905
0.899
0.905
0.908
0.845
0.854
0.858
BACS-Sketch
0.940
0.943
0.944
0.939
0.940
0.941
0.692
0.852
0.894
subset of classes from the original target dataset. The PCC is computed between T--t and A across all 50 target tasks, where each target subset contains 40% to 100% of the total classes, and for each class, all train and test images are included in the subset. We consider six target datasets including Caltech101, CUB200, Oxford-IIIT Pets, Flowers102. Stanford Dogs, and Seeehh.
Model architectures and Training. We consider two source models: ResNet-18 18 pre-trained on CUB200 and ResNet-34 pre- pre-trained on Caltech101. We train the transferred models for 100 epochs using SGD with a momentum of 0.9, learning rate 10- 3, and a batch size of 64.
Across two source datasets and four target datasets, HASTE-LEBE achieves the highest correlation for the target selection task, and outperform their respective baseline methods (Table In particular, we observe an improvement in correlation scores of +0.99% for LEEP, +1.15% for NCE, and +5.1 15% for GBC.
Results.
5.3 SEMANTIC SEGMENTATION
Experimental setup. We follow the fixed target setting described in Pándy et al. (2022) and report the correlation between meanloU and T for each target dataset. We consider a Fully Con- nected Network (FCN) Long al. (2014) with ResNet-50 backbone pre-trained
subset of
COC02017 (in et al. al. 2009). BDD100K Yu al., 2018). IDD (aarma et al. 2019), PascalVOC (Li al. 2020) and SUIM (Islam et al. (2000) datasets. Among them, we consider the target datasets CityScapes (Cordts et al. 2016), Cam Vid (Brostow al., 2009). BDD100k (Yu al. 2018) and SUIM (Islam
2014). We consider CityScapes (Cordts et al. 2016), Cam Vid (Brostow)
et al. 2020). Note that we use the CA variant of HASTE for semantic segmentation as segmentation does not have class labels.
Model architectures and Training, We train an FCN Resnet50[Long et al. (2014) model for each source training dataset (except the target) and individually fine-tune them on Dicain We train the individual models for 100 epochs using SGD with momentum of 0.9, weight decay of 10-4, batch size 16, a learning rate of and reduce it on plateau by a factor of 0.5. Each model is fine-tuned on the target dataset independently, using SGD with momentum of 0.9, learning rate of 10-3, and a batch size of 16.
Results. On average across four target datasets, results show that HASTE-modifiee metrics out- perform their bascline methods (Table In particular, we observe an improvement in correlation scores of +182.23% for LEEP, +33. +3%.34% for NCE, and + 1449300% for GBC.

---

Under review as a conference paper at ICLR 2023
Table 3: Results on the semantic segmentation source architecture selection task. Shown are cor- relation scores (higher the better) computed across all source architectures. Results where HASTE modified metrics perform better than their counterparts are in bold.
Target (D,)
LEEP
HASTE-LERE
NCE
HASTE-NCE NCE
GBC
HASTE-GBC
BDDI00K 0.147
0.197
0.731
0.743
0.645
0.660
Cam 0.063
0.374
0.573
0.583
0.334
0.796
SUIM 0.823
0.980
0.204
0.461
-0.218
0.784
CityScapes 0.045
0.127
0.524
0.545
0.952
0,923
Table 4: Results on the ensemble model selection task. Shown are correlation scores (higher the
better) computed across all ensemble candidates. Results where HASTE modified metrics perform
better than their counterparts are bold. See Appendix Table for results using K=3.
HASTE-MS-LERE
BASTE-E-LEEE
Target (D+)
MS-LEEP
E-LEEP
CA
CS
CA
CS
Flowers102
0.230
0.368
0.251
0.271
0.314
0.244
Stanford Dogs
0.400
0.378
0.400
0.503
0.522
0.506
CUB200
0.334
0.411
0.324
0.402
0.403
0.434
OxfordPets
0.112
0.148
0.133
0.276
0.338
0.281
Caltech
0.462
0.502
0.467
0.520
0.513
0.518
5.4 ENSEMBLE MODEL SELECTION
Experimental setup. Given pool with P number of source models, this task aims to select the subset of models whose ensemble yields the best performance on a fixed target dataset (Agostinelli et al. 2022b). Since evaluating every ensemble combination of the P source models is very expen- sive, the ensemble size K (i.e., number of models per ensemble) is fixed, which yields (K) candidate ensembles. The PCC is then computed between the T and A across all candidate ensemble. We use K and P=11 in our experiments and consider the target datasets from Section 5.2
Model architectures and Training. We include source models pre-trained on the above datasets as well as ImageNet. Each ensemble of model architectures consists of one or more models from the pool of ResNet-101, VGG-19 (Simonyan & Zisserman 2013), and DenseNet-201 with each model pre-trained on the mentioned datasets. For a given candidate ensemble, each member model fixed target train dataset Dirinn and, finally, the ensemble prediction
is fine-tuned individually is calculated as the mean of all individual predictions. Each model is fine-tuned on the target dataset independently, using SGD with momentum of 0.9, a learning rate of 10 and batch size of 64.
Results. Our empirical analysis in Table T shows that, on average across five target datasets, HASTE- modified metrics outperform their baseline methods. In particular, we observe an improvement in correlation scores of +14.43% for MS-LEEP and +4.10% for E-LEEP.
5.5 ADDITIONAL RESULTS ON LANGUAGE MODELS
Experimental setup. We now evaluate the performance of HASTE for the sentiment classification transfer learning tasks and show results in the target dataset selection setting. We consider three target datasets, including TweetEval (Barbieri et al. 2018), IMDB Movie Reviews (Maas er al., 2011) and CARER (Saravia et al. 2018) for our language experiments.
Model architecture and Training. We include source models trained using a classification head on pre-trained BERT (Devlin al., (019) model on CARER (Saravia al. (018)) and AG- News (Zhang et al. 2015) datasets. We fine-tune the entire source model, including the BERT layers for 3 epochs using the Adam optimizer, with a learning rate of x 10-5 and batch size of 8.
Results. Table Shoow show that HASTE-modified transferability metrics outperform their baseline coun- terparts. On average across four source-target pairs and two techniques, we observe an improvement of 133..33% for LEEP, +33.40% for NCF, and +57.24% for GBC using HASTE metrics.
5.6 ABLATION STUDIES
We conduct ablations on two key components of HASTE modified transferability metrics: i) size of the hardest subset and ii) correlation estimates using different buckets. We also study the impact
8

---

Under review as a conference paper at ICLR 2023
Table 5: Results on target task selection for sentiment classification. Shown are correlation scores (higher the better) computed across all target candidates. Results where HASTE modified metrics perform better than their counterparts are bold.
Source Target Pair
LEEP
LESTE-LEEP CA
NCE
HCS
GBC
HASTE-GBB
CS
CA
CS
CA
CS
Emotion IMDB
-0.172
0.115
0.06
-0.192
0.073
0.050
-0.097
0.141
0.109
Emotion TveetEval
0.884
0.892
0.885
0.884
0.892
0.885
0.828
0.834
0.824
AGNews Emotion
0.939
0.943
0.944
0.940
0.947
0.944
0.808
0.808
0.808
AGNews TweetEval
0.776
0.779
0.784
0.884
0.892
0.885
0.549
0.549
0.549
LEEP score
0.8
LEEP corr score
0.8
Pets (CS)
Owford Pets (CA)
0.6
Pets Baselline
0.7
Oxford Pats (CS)
Oxford Pets (CA)
0.4
Pets Baseline
0.6
0.01
0.03
0.1
0.2
0.25
0.33
0.5
Bucket
Bucket 2 Bucket 3 Bucket Bucket Bucket number
Fraction size
(a) Bucket size analysis
(b) Hard Easy Buckets
Figure 4: Ablation results for HASTE. Left: It demonstrates the LEEP results (y-axis) on varying the size of the hard subset as a fraction of the complete dataset (x-axis) and shows that 25%-33% gives the best LEEP score. Right: It shows the LEEP results (y-axis) for hard-to-easy buckets (x- axis) and shows that the transferability scores are the highest for Bucket (hardest).
of different architectures for computing hardness on the performance of the HASTE metrics (see Appendix A.4).
Bucket Size Analysis. Here, we follow the experimental setup from the source architecture selee- tion (Section (.1) and choose different sizes of the hardest bucket (or simply hardest subset). We vary the bucket size b by using different fractions of the entire dataset and report the results for (0.01, 0.03, 0.1, 0.2, 0.25, 0.33, 0.5). We find that, on average, the correlation performance in- creases as we increase the bucket size (Figure 4a). Further, bucket sizes in the range 6= 0.4] generally give the best transferability estimates. This is intuitive because the influence of true hard samples might decrease in the light of easier samples for large bucket sizes, thus, going against the notion of HASTE, while for very small bucket sizes, the number of samples might not be enough for metrics like LEEP to show their effectiveness.
Transferability along Hard to Easy Buckets. A key question in HASTE is understand the effect of different subsets (depending on their easiness or hardness) on estimating transferability. Here, we follow the experimental setup from source architecture selection (Section 5.1). calculate HASTE- LEEP using different buckets, and compare it with the baseline LEEP score using the entire dataset. Results show that transferability estimates the best for harder subsets and gradually degrade while moving towards easier subsets (Figure 4b), confirming the core hypothesis of HASTE.
6 CONCLUSION
We propose and address the problem of estimating transferability from a source to the target domain using examples from the harder subset of the target dataset. To this end, we introduce HASTE (HArd Subset Trans/Erability) which leverages class-agnostic and class-specific strategies to identify harder subsets from a target dataset and can be used with any existing transferability metric. We show that HASTE-modified transferability metrics outperform their counterparts across different transfer learning tasks, data modalities, models, and datasets. In contrast to the findings in Agostinelli et al.) (202bb), i.e., one metric doesn't it work for all transfer learning tasks, we show that HASTE metries achieve favorable results across diverse transfer learning settings (Sec. 5). Hence, we anticipate that using HASTE could open new frontiers in estimating transferability and pave way for several exciting future directions, like developing new techniques to identify harder subsets and extending HASTE analysis to other transfer learning tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6430-6439, 2019.
Chirag Agarwal, Daniel D' souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10368-16600 10368-10378, 2022.
Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? arXiv preprint arXiv:2204.01403, 2022a.
Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metrics for selecting source model ensembles. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7936-792 18667996, June 2022b.
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An coormation-theoretee approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2309-2313, 2019. doi: 10.1 11110///T122011888372222
Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Multi- lingual emoji prediction. In Proceedings of The 12th International Workshop Semantic Eval- uation, pp. 24-33, 2018.
Gabriel Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognit. Lett., 30:88-97, 2009.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057009 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong sumi-supee learners. arXiv preprint arXiv:2006.10029, 2020b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annota- tions. In CVPR, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1...8653//1/N19-1423, URL https:
//aclanthology. wwtph//oooo..ooooo//12111222
Daniel D' D'souza, Zach Nussbaum, Chirag Agarwal, and Sara Hooker. tale two long tails. ICML Workshop on Uncertainty and Robustness in Deep Learning, 2021.
Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12387-12396, 2019.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statisties, pp. 201-208. JMLR Workshop and Conference Proceedings, 2010.
10

---

Under review as a conference paper at ICLR 2023
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training amples: An incremental bayesian approach tested on 101 object categories. In 2004 Con- ference Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004. doi:
N0.1109CCVP.22044.35.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10.1109/CVPR.2016.990
Jeremy Howard. Imagenette. URL |ttp://////ggttooooo///////aa/a/
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269 2017. doi: 10. 109/CVPR.2017.222
Loi Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transfer- ability estimation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 9201-9223 PMLR, 17-23 Jul 2022. URL https::///// ptttpp:://proceeinngg www..ress..er..ee.omm....e.....e prwss/ 621
https//v1//huunneeee hwtp:/uann22....cm html
Md Jahidul Islam, Chelsey Edge, Yuyang Xiao, Peigen Luo, Muntaqim Mehtaz, Christopher Morse, Sadman Sakib Enan, and Junaed Sattar. Semantic Segmentation of Underwater Imagery: Dataset and Benchmark. In /EEE/RSJ International Conference Intelligent Robots and Systems (IROS).
JEEE/RSJ, 2020.
Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A. Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions on Neural Networks and Learning Systems, 29(3)33533-3587 2018.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
D. Li, Y. Yang, Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In Los Alamitos,
2017 IEEE International Conference Computer Vision (ICCV), pp. CA, USA, oct 2017. IEEE Computer Society. doi: 10. (0.1109//CC222017.9999 URL https: //doi httpa//eecccottrroooooot.ooooooo
2017
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang. FSS-1000: A 1000- class dataset for few-shot segmentation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2020. doi: 10. 111100/<:24200002000000290 URL https://dos wwwg/orgr.0..o 1ww.110Fvv 42 2020. 00294
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0313 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. CoRR, abs/141 1.4038, 2014.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: 7/ w/www. ac www..b.......................... org/ www.thooooy//P11011..
Sébastien Marcel and Yann Rodriguez. Torchvision the machine- vision package of torch. In Pro- ceedings of the 18th ACM International Conference on Multimedia, MM '1, pp. 1485-1488, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605589336 doi: 10.1145/1873951 18742254. htR..ttss:/// UKl.:tttess//aoo.oorr/...11111/. /1t.0//01.11111111311153331 18744
11

---

Under review as a conference paper at ICLR 2023
Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference Machine Learning, (0. JoLR.org, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, ((10))1345-1355 2009.
Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Trans- ferability estimation using bhattacharyya class separability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9172-9182, June 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bun Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 Curran Associates, Inc., 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021a. URL. sttps::/pproeeddnng: htttss//poooeeddi.o......prsssss rad33//2ddoodd22... html
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-88888 8763. PMLR, 2021b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1 115(3)2111-553 2015. doi: 1007// 1263-015-08166.y.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Con- textualized affect representations for emotion recognition. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Bel- gium, October- November 2018 Association for Computational Linguisties. doi: 10.18653/v1/ 1404. URLLURLUUUE https: / www. ac ac www.www.bb..m org/ httpr//anthologyyyo /
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http: /ttp/v.iivv..///////ee//aaaaalv/ org. wwwgg/ab///ie 1ate:.119......
Jie Song, Yixin Chen, Xinchao Wang. Chengchao Shen, and Mingli Song. Deep model transferabil- ity from attribution maps. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song. Depara: Deep attribution graph for deep knowledge transferability. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3921-3929, 2020. doi: 10.1 N0111000CVP42260000000009999999
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal Computer Vision, 130, 2022.
Xinyi Tong, Xiangxiang Xu, Shao-Lun Huang, and Lizhong Zheng. A mathematical framework for quantifying transferability in multi-source transfer learning. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 26103- 26116. Curran Asso- neurips. wttppp/pe////22//111eee
ciates, Inc., 2021. URL
db
Lisa Torrey and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp. 242-264. IGI global, 2010.
Anh Tran, Cuong Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. 2019 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 1395-1405, 2019a. doi: 10.1 10.1109/CCV2200990000
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVF Internutional Conference on Computer Vision, pp- 1395-1405, 2019b.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9(86):25792260 2605, 2008. URL http: jmlr. wttp://orggaaaap///.//
waardermaatennnaa.nen html]
Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and C.V. Jawahar. Idd: A dataset for exploring problems of autonomous navigation in unconstrained envi- ronments. pp. 1743-1751, 01 2019. doi: 10.1 10.100////2.2201.000000
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocom- puting, 312:135-153, 2018. ISSN 0925-2312. doi: http:///oi.orgg//00/00001 neucom.
2018.05.088 .033. URL wttss//www.cieeedrreee.......... wwwpett.ccccsctieeo wwtp://ciince/arriieeee/eoo
509252312 6686
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Be- longie, and Pietro Perona. Caltech-uesd birds 200. Technical Report CNS-TR-201, Caltech, 2010. URL
11teeerrttaaaaaeaaaaaeaaallaee1e pdf, , http:/ www. wisionn caltech. edu http://chhhheeiiiieeeeaaaaee----
-CC- htmll
Garrett Wilson and Diane J. Cook. A survey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol. 1 (1(5). jul 2020. ISSN 2157-6904. doi: 10. 14/1444/3400066.
https:
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Asso- ciates, Inc., 2014. URL
meweeeeeidiinnn.eneeeseeeeeeeeee neurips. http://cccc................epeaa Paper.ppp.... pdf
cc/paper:201///iieee
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. BDD100K: diverse driving video database with scalable annotation tooling. CORR, 0bs//880,.0 (006004677.22 2018.
Amir R. Zamir, Alexander Sax. William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Con- ference Computer Vision and Patlern Recognition (CVPR), June 2018a.
13

---

Under review as a conference paper at ICLR 2023
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE con- ference computer vision and pattern recognition, pp. 3712-3722 2018b.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving trans- ferability in domain generalization. Advances in Neural Information Processing Systems, 34: 10957-10970, 2021a.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighhted adversarial training. In International Conference Learn- ing Representations, 2021b.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF FOR LEMMA
Lemma 2. HASTE-LEEP is an upper bound of the NCE measure plus the average log-likelihood of the source label distribution, computed over the hard subset, i.e.,
T(aa Diand) Z H--NCE(Y Z)
log (xi)
(10)
D
Daan
Proof. Let 2j be the dummy labels obtained when computing NCE and y; be the true labels.
T(ā; D.urd
log
Σ P(ul)))((xx))
Dhund
JEZ
log
(Monotonicity of Log)
DDhand
log Pivilzz) +
log fø (x.)2.
Diuurd
DDhurd
= -H-NCE(Y 2) +
log fö (x:)...
DDuund
:=1
A.2 EXPERIMENTAL SETUP
Implementation details. All experiments were run using the PyTorch library (Paszke et al. with Nvidia A-100/V- GPUs.
2019
Model Architectures. We use variety of model architectures (VGG, ResNet, DenseNet), trained on different source datasets across our experiments. For each model architecture, we utilize embed- dings from the final layer for the class-specific method (Eqn. B For the class-agnostic method, we utilize embeddings from intermediate layers for the similarity computation (Eqn. []. Particularly, we utilize the embeddings from the final layer of each block of convolutions (Ex: Output of each residual block in ResNets). We only include 3/4 layers for any architecture and do not consider embeddings from the first block.
Similarity Computation for Large Source Datasets. For the experiments with models pre-trained on ImageNet the source, when using the class-agnostic method, it is infeasible to use the entire ImageNet dataset for the similarity comparison to generate the similarity matrix (using Eqn. []] Instead, we use a random 10% subset of the ImageNet dataset, uniformly sampled from each class, as the source dataset to compute the similarity matrix. We do not observe any performance drop due to this sub-sampling, and this can be extended other datasets as well, for further computational speedup. Additionally, we do not re-do the similarity computation for each source architecture. Instead, we only compute the similarity matrix using the ResNet-50 model and use the hard subset obtained from this to compute HASTE modified transferability metrics for all model architectures pre-trained on ImageNet. We repeat this in the model ensemble setting, utilizing a single similarity matrix for all model architectures trained on the same source dataset.
GBC Implementation. In all experiments, for computing GBC and HASTE-GBC, we use a spheri- cal covariance matrix, as we found this to yield better results, even for the base GBC score.
Size of Hard Subset. The size of the hard subset in any experiment a hyperparameter that can be tuned. Due to variations in dataset sizes, the exact value differs significantly. Instead of fixing size, we set the size the hard subset to be k% of the size of the target dataset. In general, we found value of 10%-25% to work well.
Subset for Ensemble Selection. The hard subset for the Class Agnostic method is dependent on the source dataset. As result, the experiment setting described in Section 5.4 has different hard subsets depending on the source dataset and model for a single target dataset. Since MS-LEEP is simply the addition of LEEP scores, the calculation of MSSTT-MS-LEERT is trivial. But in the case of
15

---

Under review as a conference paper at ICLR 2023
HASTE-E-LEER, we need single common subset it involves the calculation of adding empirical probabilities followed by log and mean. To this end, we take the union of the hard subsets obtained from different sources and then proceed with further calculations. We report the results in Table 4 following the same.
Models used in Ensemble Selection. We the following pool of source models for ensemble selection experiment described Section 5.4 i) DenseNet-201.. ResNet- 101, MobileNetv2 trained on on Imagenet, ii) DenseNet-201, ResNet-18, VGG-19 trained Stanford Dogs, iii) DenseNet- 201, ResNet-101 trained on Ocford IIIT Pets, iv) DenseNet-201, ResNet-101 trained on Flowers v) ResNet-11, VGG-19 trained on CUB200, and vi) ResNet-34 trained on Caltech101.
A.3 ADDITIONAL RESULTS
Computational cost. To provide indicative reference, we compare the times of several transferability metrics, as well as our HASTE modification using ResNet-18 trained on CUB200 the source model (on single GPU), and Oxford-IIIT Pets the target dataset. Including the target dataset feature-extraction stage (shared by all metrics, as well as the class-agnostic method of HASTE), LEEP runs in 7.24s, NCE runs in 7.26s, and GBC runs in 8.33s. The class-agnostic variant on the HASTE modification takes additional 24.98s, and the class-specific variant takes an additional 0.56s.
EASY SUBSET
HARD SUBSET
Figure 5: The 5 x 5 grid shows the top-25 images from the easy (left) and hard (right) subset of the target dataset using the alass-agnostic technique for the ImageNet-StanfordDi Images with higher hardness scores tend to feature cluttered images with atypical vantage points, whereas images with lower hardness scores mostly comprise dogs an uncluttered background.
source-target pair.
Table 6: Results target task selection using the fine-tuning method for CUB200 source models. Shown are correlation scores (higher the better) computed across all target datasets. Results where HASTE modified metrics outperform their baselines are bolded.
H-LEEP
H-NCE
H--BC
Target (D.)
LEEP
NCE
GBC
CA
CS
CA
CS
CA
CS
11
-0.035
0.709
0.098
0.081
0.742
0.249
0.507
0.562
0.516
Flowers102
0.612
0.617
0.613
0.593
0.579
0.618
0.535
0.526
0.568
StanfordDogs
0.929
0.936
0.929
0.928
0.927
0.929
0,909
0.914
0.913
Oxford IIIT
0.863
0.871
0.860
0.812
0.826
0.814
0.859
0.860
0.872
PACS Sketch
0.947
0.965
0.960
0.949
0.958
0.950
0.819
0.909
0.883
16

---

Under review as a conference paper at ICLR 2023
EASY SUBSET
HARD SUBSET
Figure 6: The x grid shows the top-25 images from the easy and hard subset of the target dataset using the class- aggnosic technique for the ImageNet-OxfordIII higher hardness scores tend to feature cluttered images with atypical vantage points, whereas images with lower hardness scores mostly comprise dogs and cats in an uncluttered background.
OxfordIIIT Pets source- ttrget pair. Images with
EASY SUBSET
HARD SUBSET
Figure 7: The x grid shows the top-25 images from the easy (left) and hard (right) subset of the target dataset using the class-specific technique for the ImageNet-Caltechh pair. Images with higher hardness scores tend to feature classes that are typically harder classify (since they might have very less distinguishing features), whereas images with lower hardness scores mostly comprise classes that are easily distinguishable.
source-target
Table 7: Results on the ensemble model selection task for K = (higher the better) computed across all ensemble candidates. Results where HASTE modified metrics outperform their baselines are bolded.
3. Shown are correlation scores
Target (Dt)
MS-LEEP
H-MS-LEEP
E-LEEP
H-E-LEEP
Flowers
-0.288
-0.376
-0.323
-0,319
Stanford Dogs
0.390
0.264
0.477
0.494
CUB200
0.345
0.391
0.405
0.405
Oxford-IIIT
0.115
0.189
0.253
0.343
Caltech101
0.430
0.479
0.480
0.478
17

---

Under review as a conference paper at ICLR 2023
EASY SUBSET
HARD SUBSET
Figure 8: The 5 x grid shows the top-25 images from the easy (left) and hard (right) subset of the target dataset using the class-specific technique for the SrngeNetSStaffrdDDor pair. Images with higher hardness scores tend to feature classes that are typically harder to classify (since they might have very less distinguishing features), whereas images with lower hardness scores mostly comprise classes that are easily distinguishable.
tource-target
EASY SUBSET
HARD SUBSET
Figure 9: The 5 x 5 grid shows the top-25 images from the easy (left) and hard (right) subset of the target dataset using the class-specific technique for the ImageNet-Oxfon OxfordIIII Pets source-target pair. Images with higher hardness scores tend to feature classes that are typically harder to classify (since they might have very less distinguishing features), whereas images with lower hardness scores mostly comprise classes that are easily distinguishable.
A.4 ABLATION ON HARDNESS SOURCE ARCHITECTURE
HASTE aims to achieve better transferability estimates irrespective of the source of the hardness scores, i.e., the source architecture we use to calculate harder subsets in Class Agnostic way or Class Specific way. We follow the experimental setup from the target task selection (Section 5.2) experiments. We calculate HASTE-LEEP scores on harder subsets identified using i) ResNet18 8 and VGG19 trained CUB200, and ii) ResNet50 trained on ImageNet. Results show that HASTE- LEEP outperforms LEEP (baseline calculated using the entire dataset) across all three architectures (Table 8
18

---

Under review as a conference paper at ICLR 2023
Table Results on target task selection task different source model architectures. Shown are corre- lation scores (higher the better) computed across the target dataset using HASTE-LEEP. Irrespective of the architecture used for finding the hard subset, HASTE-LEEP outperforms the baseline LEEP score.
1altech101
Oxford-IIIT Pets
Hardness Source Model
CA
CS
CA
CS
ResNet18
0.360
0.014
0.896
0.901
VGG19
0.200
0.267
0.881
0.894
ResNet50
0.630
0.196
0.896
0.898
Bascline (LEEP Scorc)
-0.03
0.863
A.5 EASIER SAMPLES ADDED WITH STOCHASTICITY
A natural question which may arise when using HASTE is whether to completely neglect the easier samples. To this end, we conduct an experiment where we add casier samples stochastically to hard subsets and then compute the respective metric correlation score. We follow the same experiment setting as Section [ We obtain results by itcrating the addition of these casier samples 10 times followed by taking the mean. We observe that addition of these easier samples do not particularly enhance the results. In fact, the results come out to be worse than when using only hard subsets. Results for the same shown Table g
Table 9: Results source architecture selection task for subset obtained by stochastically adding casier samples to the hard subset. Shown are correlation scores (higher the better) computed across all source architectures trained on ImageNet. Results where HASTE modified metrics outperform their baselines are in bold.
H-LEEP
Stochasticity
Target (D;)
LEEP
CA
CS
1%
2%
3%
4%
5%
10%
Caltech101
0.416
0.439
0.475
0.474
0.468
0.472
0.472
0.472
0.458
Flowers 102
0.534
0.405
0.626
0.616
0.596
0.579
0.576
0.575
0.539
CUB200
0.504
0.508
0.723
0.719
0.714
0.714
0.723
0.728
0.679
A.6 RESULTS WITH ALTERNATE CORRELATION METRIC
HASTE approach is agnostic to the choice of correlation coefficient. We provide additional results using i) Kendall Tau and ii) Weighted Kendall Tau correlation coefficients the Source Architecture Selection experiment.
Table 10: Results on source architecture selection. Shown are Kendall Tau correlation scores (higher the better) computed across source architectures trained on ImageNet. Results where HASTE modified metrics outperform their baselines in bold.
HASTE-LEEP
HASTE-NCE
HASTE-GBB GBC
Target (Dr)
LEEP
NCE
CA
CS
CA
CS
CA
CS
CUB200
0.238
0.142
0.714
0.142
0.238
0.619
0.619
0.619
0.714
StanfordDogs
0.809
0.809
0.809
0.809
0.714
0.809
0.619
0.904
0.714
102
0.333
0.619
0.523
0.238
0.428
0.238
0.047
0.047
0.238
Oxford-III
0.904
1.000
0.904
0.523
0.619
0.714
0.523
0.523
0.523
1oltech101
0.390
0.390
0.390
0.097
0.292
0.195
0.683
0.683
0.683
Imagenette
0.714
0.714
1.000
0.619
0.714
0.683
0.619
0.619
0.714
PACS-Sketee
0.000
0.097
0.097
0.000
0.097
0.195
0.487
0.487
0.585
19

---

Under review as a conference paper at ICLR 2023
Table 11: Results on source architecture selection task. Shown are Weighted Kendall Tau correlation scores (higher the better) computed across all source architectures trained on ImageNet. Results where HASTE modified metrics outperform their baselines are bold.
MASTE-LEED
HASTE-NCE
HASTE-GBE
Target (Dr)
LEEP
NCE
GBC
CA
CS
CS
CA
CS
CUB200
0.258
0.108
0.638
0.113
0.247
0.659
0.591
0.591
0.805
StanfordDogs
0.865
0.865
0.865
0,865
0.672
0.865
0.746
0.952
0.805
Flowers 102
0.376
0.705
0.644
0.389
0.611
0.400
-0.119
-0.119
0.031
Oxford-IIIT
0.925
1.000
0.925
0,587
0.678
0.721
0.692
0.530
0,530
Caltech101
0.535
0.535
0.535
0.345
0.482
0.238
0.723
0.723
0.723
Imagenette
0.672
0.672
1.000
0,558
0.758
0.693
0.799
0.808
0.851
-0.145
0.026
0.095
-0.063
0.044
0.232
0.567
0.406
0.651
Table 12: Results on source architecture selection task with LogMe as the baseline. Shown are correlation scores (higher the better) computed across all source architectures trained on ImageNet. Results where HASTE modified metrics perform better than their counterparts are in bold.
HASTE-Loggh
Target (Dr)
LogMe
CA
CS
CUB200
-0.310
0.082
0.365
StanfordDogs
0.921
0,953
0.943
Flowers 102
-0,210
0.483
0.614
Oxford-IIIT
0.940
0.973
0.930
Caltech101
0.358
0.712
0.792
Imagenette
0.928
0.930
0.971
PACS-Skete
-0.423
0.677
0.117
A.7 RESULTS WITH LOGME BASELINE
We include results on LogME as an additional baseline metric. The results cover two experimental settings Source Architecture Selection (Table [2), and Target Task Selection (Section [5.2). On average, HASTE-LogME shows an improvement of 120.53% in the source architecture selection experiment, and improvement of 236.166 16% in the target task selection experiment.
A.8 DISCUSSION ON TASK TRANSFERABILITY WORKS
Our work focuses on the problem of estimating transferability of a source model on the target dataset prior to fine-tuning, Formally, this seeks to address two downstream tasks i) of all the source mod- els, find the most suitable to perform transfer learning on given target dataset, ii) of all the target datasets, find the most suitable to perform transfer learning on given source model. Thus, we can note that fine-tuning all the possible models or datasets is not a possible solution here, stark con- trast, some recent transferability works (Zamir et al. (2018b); Dwivedi & Roig (2019;; Song et al. 2020 2019) ) consider models that are pre-trained on one or more tasks, and some further transfer these models to another task, requiring the expensive fine-tuning process. These works only discuss task transferability, i.e., transferring across computer vision tasks such as classification to semantic segmentation, semantic segmentation depth prediction, etc. More precisely, these works establish task relatedness or how similar is one task to the other and do not propose a transferability metric, which is not the focus of our work. Further, these works are not generalizable they either perform fine-tuning from scratch or have computational costs similar to fine-tuning, which renders these approaches infeasible for our problem. In addition this, Zhang et al. (2021a). quantified transfer- ability for the task of Domain Generalization, while Tong et al. (2021) discussed transferability for multi-source transfer, both of which operate in a setting quite different from ours.
A.9 MOTIVATION BEHIND HARDNESS METRIC
Our approach for measuring hardness motivated by recent works that have shown that the process of transfer learning shows maximum gains when the images from the source and target tasks in the similar domains. Further, these recent works measure domain similarity as function of the distance between source and target samples. In addition to these findings from previous works,
20

---

Under review as a conference paper at ICLR 2023
Table 13: Results target task selection using the fine-tuning method for Caltech 101 source mod- els. Shown are correlation scores (higher the better) computed across all target datasets. Results, where HASTE modified metrics perform better than their counterparts, are in bold.
HASTE-LEEP
Target (D:)
LEEP
CA
CS
CUB200
-0,951
0.945
0.943
Flowers
-0.759
0.795
0.723
StanfordDogs
-0,887
0.847
0.842
Oxford-IIIT
-0,899
0.852
0.476
PACS-Sketch
0.044
0.035
0.416
our empirical analysis also confirms that hardest samples (i.e., with lowest similarity) obtain lower transfer accuracies than the rest of the samples (as shown in Figure [}. confirming the efficacy of our hardness measure.
A.10 DEFINITION OF TRANSPERABILITY
Following previous works [ao et al. 2019) Nguyen et al. 2020), we define transferability as to when transfer may work, and what extent
21


Reviews:
[{"review_id": "o0x3YJGnrdk", "reviewer": "Reviewer_bePf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to address the problem of estimating transferability from a source to the target domain by using examples from the harder subset of the target dataset. The authors introduce class-agnostic and class-specific techniques to identify harder subsets and show that the proposed method can be used with existing transferability metric to improve their reliability.", "summary_of_the_review": "The paper is well organized and the proposed method is well provided. However, the motivation of the proposed method is not very strong, and the performance is sensitive to the number of hard samples. Some highly related works and the comparisons with them are missing in the paper, which makes the superiority of the proposed method unclear.", "strength_and_weaknesses": "Pros:  \n+ The work paper is easy to follow. The authors clearly present the proposed method and make detailed descriptions about the experiments.\n\n+ The idea of using hard samples to estimate transferability is interesting. \n\nCons:  \nMy main concerns about the work are two-fold:  \n- The motivation of the proposed transferability estimation method using hard samples are not well described. The observation that hard examples are more out-of-distribution and challenging than the easy samples in the transfer learning process is not a surprise. However, why using hard samples delivers higher transferability estimation performance, especially provided that the target data is usually filled with both easy and hard samples, or only hard samples (considering the case where source and target domain difference is extremely large), is not explained in the paper. Furthermore, the number of hard samples appears a very important hyperparameter in the proposed method. The ablation study show that the performance of the proposed method is very sensitive to this hyperparameter. How do we set this hyperparameter in our own problem?\n\n- Many highly relevant works are missing from the work, which leaves the superiority of the proposed method unclear. Some are listed as follows:    \n[1] Deep Model Transferability from Attribution Maps, NeurIPS2019.  \n[2] DEPARA: Deep attribution graph for deep knowledge transferability, CVPR2020.  \n[3] Logme: Practical assessment of pre-trained models for transfer learning. ICML2021.  \n[4] Representation similarity analysis for efficient task taxonomy & transfer learning. CVPR2019.    \n[5] Taskonomy: Disentangling task transfer learning, CVPR2018.    \nThe authors are encouraged to make discussions on these highly relevant works and clarify the differences between the proposed method and these prior works. Furthermore, any experimental comparisons with these works are especially appreciated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A04rWFScZ7", "reviewer": "Reviewer_nxzz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to identity hard-subset samples in the target dataset for transferability estimation. The main idea is to find the hard examples from the target set by computing the similarities of the source features and target features. The hard examples are used with LEEP, NCE and GBC to improve the performance of transferability estimation of these methods. ", "summary_of_the_review": "The authors propose a method for finding hard examples for transferability estimation. The assumption is however too strong which makes the method not widely applicable. \n\nPost rebuttal:\n\nAfter reading the rebuttal, I still tend to maintain the score. The main concern is the applicability of the method in a more broad setting for transferability setting. The assumption of having all the samples available is too strong for practical application of the method.", "strength_and_weaknesses": "Strength:\n\n1. The paper is clearly written and easy to understand. \n2. The idea of using hard examples for transferability estimation is interesting. \n\nWeaknesses:\n\n1. The major concern is the setting of the method. The authors assume all the source samples and target samples are available. However, in practice, in many cases we just have the pre-trained models and a subset of the target samples. This makes the method not applicable in many real-world settings.\n\n2. The method of calculating hardness is kind of ad-hoc. It is more about the similarity to the source data rather than the hardness of the samples. \n\n3. Also, many recent baselines are not considered, such as LogMe (ICML 2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5TKBMEtCsik", "reviewer": "Reviewer_acad", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes HASTE for estimating the transferability of a source domain to a target domain. Two techniques are introduced, one is class-agnostic and another is class-specific. The techniques achieve state-of-the-art compared to other concurrent baseline metrics.\n\n--post rebuttal--\n\nDear authors, thank you for going through my comments and trying to address them. However, I still think the current draft is not ready for submission. As Reviewer bePf also pointed out, the current method (HASTE) is not clearly motivated, and comparison with other transferability metrics is not properly made. Overall, this paper is not clearly written. I would suggest the authors keep polishing the draft for the next round.", "summary_of_the_review": "In summary, this work proposes an interesting approach using the hard subset to measure the transferability. However, some parts are not clearly written, and the theoretical analysis doesn't fully explain the experiments (it only covers LEEP).", "strength_and_weaknesses": "Strengths:\n1) The problem of estimating the transferability is interesting and important for transfer learning, and the proposed method is well-motivated. \n2) Good literature survey and comparison with three baseline algorithms. The improvement is noticeable.\n3) The authors test the proposed transferability metric on a large variety of benchmark datasets. The comparison looks quite comprehensive.\n4) Some theoretical analysis is discussed. For example, Lemmas 1 and 2 show that Haste is upper bounded by the average log-likelihood of the hard subset, and lower bounded by the NCE measure plus the log-likelihood of the source label distribution computed over the hard subset.\n\nWeaknesses:\n1) What is the definition of transferability? I think even this question is still a hot research topic, e.g., in [1], [2], and these metrics have not been discussed in the paper. The transferability definition is not clearly given either.\n2) the activation volume $\\mathcal{E}_l$ is not well defined.\n3) How would eq. (2) and eq. (4) relate to each other? They use the same notation but seem to be different definitions.\n4) In eq. (5), the authors say $\\mathcal{T}$ can be any existing transferability metric, but the main goal of the current paper is to propose a new transferability metric. Does that mean HASTE is like a \"meta\"-transferability metric that can work on top of any existing ones?\n5) The connection between the class-agnostic and the class-specific methods are not clearly shown. They seem to be two separate methods.\n6) For the theoretical analysis, the discussion of Lemmas 1 and 2 are lacking: why would showing the lower and upper bounds of HASTE be interesting and what is the meaning of it?\n7) For the experiments Table 1 and Table 2, the improvement over baselines are quite minor (~0.002 sometimes), while sometimes it helps a lot (like Table 5 and Emotion-IMDB). What is the reason for this? Why for example, does HASTE not improve over GBC at all?\n\n\n[1] Zhang, Guojun, et al. \"Quantifying and improving transferability in domain generalization.\" Advances in Neural Information Processing Systems 34 (2021): 10957-10970.\n[2] Huang, Long-Kai, et al. \"Frustratingly easy transferability estimation.\" International Conference on Machine Learning. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject