Understanding the Transferability of Representations via Task-Relatedness
Authors: Akshay Mehra
Yunbei Zhang
Jihun Hamm
Venue: NeurIPS 2024 poster (2024)
Keywords: Transfer Learning Analysis
Distribution Shift
Primary area: safety_in_machine_learning
TL;DR: We rigorously analyze transfer learning in terms of relatedness between tasks and show that task-relatedness accurately predicts transferability in various practical scenarios.

Abstract: The growing popularity of transfer learning due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in a cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks.Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels of the target task and its high correlation with the model's accuracy after end-to-end fine-tuning on the target task makes it a useful metric for transferability estimation. Our empirical results of using task-relatedness on the problem of selecting the best pre-trained model from a model zoo for a target task highlight its utility for practical problems.

Understanding the Transferability of Representations via Task-Relatedness
Akshay Mehra, Yunbei Zhang, and Jihun Hamm
Tulane University
famehra, yzhang111, jhamm3)ttulane edu
Abstract
The growing popularity of transfer learning, due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models be transferred obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose novel analysis that analyzes the transferability of the representations terms of their relatedness to given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. Our experiments using state-of-the-an pre-trained models show the effectiveness test-relaeddness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels the target task and its high correlation with the model's accuracy after end-to-end fine-tun tunttging the target task makes a useful metric for transferability estimation. Our empirical results using relatedness the problem of selecting the best pre-traine pre-trained model from a model zoo for a target task highlight its utility for practical problems.
pre-trained models to downstream tasks
Introduction
is powerful tool for developing high perfoomaae machine learning models, especially in current times when large models [45] 12,1111881 pre- pre-trained on huge amounts of data are being fine-tuned for various downstream tasks. While large pre-trained models achieve impressive performance on downstream tasks even in the zero-shot inference setting 45). their performance can often be improved by fine-tuning them on data from target tasks. However, our understanding of when representations from these models lead to classifiers that achieve high performance (i.e., high transferability) to downstream tasks still lacking.
Transfer learning (TL)
42 59
Analytical works based on domain adaptation 8 tasks (i.e., when only features/label priors change across tasks) but in the TL setting, label sets can also change (i.c., cross-task setting). Recently, [56] showed that the relatedness between the label sets of the two tasks measured using conditional entropy can explain the difference in their transferability, However, [36] focused only on the cross-task setting, and analysis for transferability in a general cross-domain cross-task setting is not addressed. Apart from these analytical works, another line of work focuses on proposing transferability metrics that correlate well with performance on downstream tasks after end-to-end finettunin. We refer these works as score-based transferability estimation
36 32
37 38
can only explain cross-domain
19 [1.. These works focus on developing scores for selecting pre-trained model from a model zoo, that achieves the best transferability on a target task. While these works address practical problem, they do not focus on providing an analysis of transferability.
(SbTE) metrics [6]] [55
40
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Food
Thus, we first rigorously analyze the transferability of the representations in producing high- performnnnn classifiers and propose a novel approach that studies transferability in terms of its relatedness to a refer- ence task (see Fig. T_ This is in line with previ- ous analytical works 2 56 55 which studied
Trainable
Referonce Task "... maagaler
High
Pre-trained Encoder eg
Task-relatedness
Transferability
Downstream Task e.p. Pees
the model's performance
target tasks in terms of the source task in different settings such as domain cnapttation/geeralizzaen
and recently TL. However, there's ' a crucial difference: we study transferability in terms of reference task instead of the source task since it is impractical to assume the knowledge of the source task used to train large models such CLIP 45 or GPT, commonly used for TL.
Digits
X-mays
Textures
Airorotts
Flowers
Figure 1: Given a pre-trained encoder (e.g., CLIP (45)). how does the performance after fine-tuning it on a reference task (c.g.. Ima- geNet) relate to the performance after fine- tuning it on other tasks? Through a rigorous bound on transferability (Theorem (3) in terms of the relatedness between a reference and a target task, we show that tasks related the reference task achieve provably better perfor- mance after fine-tuning.
Our approach works by transforming the distribution (and classifier) of a reference task, by transforming its class-prior distribution, label set, and feature space to obtain a new distribution that is similar to that of the target task (Fig. (2) Based these transforma- tions, we show that transferability can be provably explained (and is tightly upper bounded) using three interpretable terms. A weighted reference loss term appearing due to the class prior distribution difference between the tasks, a label mismatch term appearing as conditional entropy between the label distributions of the tasks, and a distribution mismatch term appearing as the Wasserstein distance between the transformed reference and target distributions (Theorem 3 We define task-relatedness as the sum of these three terms (a smaller value implies higher relatedness). We propose an opti- mization problem (Eq. 4 and an algorithm (Alg. [T to learn the transformations compute it. Using state-of-the -f--th--rttrr art (SOTA) pre-trate pre-trained models, with different architectures, trained with various training methods on computer vision (CV) and natural language processing (NLP) tasks, we show that task- relatedness achieves a small gap to transfcrability (Sec. 4.1. Our analysis also leads to new insights into learning in the TL setting such as to improve the transferability of an encoder on downstream task, one can improve the encoder's transferability on related reference tasks (Sec. 4.2). This is particularly useful when practitioners intend to develop encoders that achieve high transferability to proprietary (and potentially inaccessible) datasets.
We also demonstrate the utility of task-relatedness in estimating the accuracy of the model after end-to-end fine-tuning. While the TL setting assumes access to target labels, the high computational cost of end-to-end fine-tuning of a pre-trained model on target task calls for developing metries that are efficiently computable and highly correlated with end-to-end fine-tuning accuracy. To this end, we propose to use relatedees computed in the penultimate layer of the pre-trained model as our transfcrability estimation metric. To further improve the computational efficiency of we only measure the difference between the class-wise means and covariances of the distributions in lieu of the Wasserstein distance as required in Theorem 3 This enables the computation of ress-relatedsess with only the statisties of the reference/target tasks. Our empirical results (Sec. 4.3) attest that task-relatedness achieves high correlation with the model's accuracy after end-to-end fine-tuning on the target task making it an effective metric for selecting a pre-trained model from model zoo that achieves the best accuracy on the target task. Moreover, unlike previous SЫТE metrics, relatedeessssss can be estimated even without labeled target data, making it suitable for unsupervised transfcrability estimation, highlighting the advantage of reference task as used in our analysis. Our main contributions are:
We rigorously analyze transferability for classification tasks. Our analysis, to the best of our knowledge, leads the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting.
We propose an optimization problem to efficiently compute task-relatedness, using small amount of target labels and show that can even predict performance after end-to-end fine- tuning without requiring target labels.
2

---

Using SOTA models and CV/LLP tasks, we show that task-relatedness accurately predicts transferability and show that transferability unseen tasks can be improved by improving transferabiliity to known (related) tasks.
2
Related Work
has been studied widely and consists of various settings including transductive transfer, inductive transfer, and task transfer learning. The transductive setting also referred to as domain adaptation 88] 7 focuses reducing the shift between two domains. The task transfer setting focuses on identifying the relationship between tasks, regardless of the model, explain the transfer performance (see Appendix for more details). Lastly, the inductive transfer setting focuses on using an inductive bias such as fine-tuning a pre-trained model (trained via adversarial training (88, self-supervised learning [1T 10 [2] or by combining language and image information [45)) to improve the performance on a target task. Our work focuses
Transfer learning (TL): TL 42
59
18
49
46
13
14
on the inductive transfer learning setting and proposes an upper bound on transferability of the representations of pre-trained models to downstream tasks.
Analytical works for learning under distribution shifts: Prior works 8
52
36
32
37
38
analytically explained learning under distribution shifts using distributional divergence between the marginal distributions and a label mismatch term. However, these results are applicable under assumptions such as covariate or label shift which need not be satisfied in TL where both the data distribution and the label spaces can be different (see App. Bor A detailed comparison). Recently, [56] proposed an upper bound on transferability in a restrictive setting of the same features for both tasks, however, our analysis does not require such assumption. Other works [ 4 analyzed the representation for the multi-task learning setting. These works showed that when tasks are weakly related, single representation space (model) may not perform well for all tasks. However, the TL setting differs from both of these and our work aims to analyze transferability in this setting.
Score-based transferability estimation (SbTE): These works [5] 40 29 61 55 39] use data from the target task and produce a score correlated with transferability. Such a score is useful for selecting the model from a model zoo that leads the best transferability to target task. (56) proposed the
Negative Conditional Entropy (NCE) score that predicts transferability using the negative conditional entropy between labels of the tasks but requires the two tasks to have the same input instances. [ estimates transferability by solving the HGR maximum correlation problem and using normalized Hscore, in the same setting as [56). 440] proposed the LEEP score and computed NCE using soft (pseudo) labels for the target task from a pre-trained model. OT-CE (55] combined Wasserstein distance and NCE whereas (5] 6 estimate likelihood and the marginalized likelihood of labeled target examples to estimate transferability. (33] proposes a model-agnostic approach that also relies on optimal transport compute the distance between the tasks similar OTDD [3]. In contrast, we focus on analyzing transferability in terms of task-relatedness theoretically along with demonstrating its effectiveness as a transferability estimation metric for the pre-trained model selection problem.
3 Analysis of TL using task-relatedness
Problem setting and notations: Let PR(x,, y) and Pr y) denote the distributions of the reference and the target tasks, defined on Xx VRR and X7 x 3/r respectively. We assume that the feature spaces are common (XR= = Xy = X) such as RGB images, but the reference label set D/R = 21, ... R and the target label set 3 {1, 2, Kr} }' can be entirely different. We assume the number of reference task classes (KR) are greater than or equal to the number of target classes (Kz). In dataset with or
the TL setting, encoder (feature extractor) g X - Z is pre-trained without labels depending on the training method (c.g., supervised vs. self-supervised). We denote the resultant push-forward distributions of R and T on the encoder output space as PR(=,,) v) and Pr, v)- With a fixed encoder J. a classifier (linear or non-linear), 11())= : 2 + A, that outputs a probability vector is learned for the reference (hn) and the target (hv) separately, where A R/T is PM p,, [((h(2gg)) y) and hy arg minkew Px [((h(z;g) g)))) where 7 is the set of classifiers and ((h(2), y) log(h(=),)
KR/ Kr simplex for R/T. The classifier ha arg minsen B(a,y)
the cross-entropy loss. Table 3 in App. A summarizes the notations used in our work. Next, we define transferability commonly used in the literature.
3

---

Reference
R¹
R
R
Target
Imagenet)
(eg. CIFAR-10
Prior
Feature Transform
Distribution mismatch
Transform
Transform
Cat
Dog
Dog
Wolf
(x)
(xr(()
Dog
Cat
hg (4)
Pn(2.y)
(2.))
(x)()))
Py(x.))
Figure 2: Overview of our task transformation model: A series of transformations are applied to the reference distribution PR(Z, y) and classifier hg to produce the transformed distribution PR and classifier fu to explain transferability to the downstream target task. Class-prior transformation (R + I2') changes the class prior of the reference distribution (e.g,, an irrelevant Bee class in It now has smaller prior) followed by label set transformation (R' R") (e.g., match [Lion, Wolf) with {Cat, Dog]), followed by feature space transformation (R" + R"1) to match the feature distribution of the target task (2.
Definition 1. (Transferability). Transferability of the representations from an encoder g on a target task T for classifiers in H is defined as Pr
(((e))eep(((r(eeeeeeee (((br 9).
In the next section, we show the analysis with H as the class of linear classifiers for ease of explanation and discuss its extension to non-linear classifiers in App. A.5 Proofs for See. 33 are in App. A
3.1 Our task transformation model
The reference and the target tasks share the same encoder but do not share label sets or data distributions. Therefore, to relate the two tasks, we propose chain of three simple transformations: 1) prior transformation (from R to R'), 2) label transformation (from R' to R"), and 3) feature transformation (from Rt to R""). The N', It", R!" are intermediate domain names after each of the transformations are applicd. The corresponding classifier in each domain is denoted by hR' hg", and hæ as illustrated in Fig. 2 The distribution after the transformations (PR") has the same feature ZRm = ZT = Z and label sets VR.. = YT as the target task T,, and consequently, the loss of the transformed classifier hn and the target classifier fy can be related.
Class-prior transformation (R - R'): Since the reference task has more classes than the target task (K 2 Ky), many of the reference task classes are likely irrelevant for transfer to the target classes, e.g., while transferring from ImageNet to CIFARIO 10, only a small portion ImageNet classes are relevant to CIFARIO classes. The prior transformation accounts for the relative importance of the reference classes. This is illustrated in Fig. where changing the class prior of R reduces the prior of the Bee class and increases the priors of Wolf and Lion classes (shown by the changed size of classes Wolf and Lion in R²). While transforming the prior of R. we keep the conditional distribution and the classifier the same i.c., PR (2/y) = PR (ll)) and h (z) (2)... Lemma I in App. A.2. shows that the expected loss of the classifier hR on R' is a re- reigheehted version of the loss of hR on R.
Label transformation (R' + R"): Next, we use a label transformation to match the label sets of the new domain R" and that of the target domain. To this end, we specify the conditional distribution Bi := P(yR y j) (Bij E [0, IJ, V j. Σ, Bij 1, vj). The label yR² of an example from the domain R²" is obtained via B BP(RR) This generative process doesn't require the feature, i.e., PR (y yyre a column is 1) models a deterministic map from VRR to Vr; B with dense entries models a weaker association. This process is illustrated in Fig. which shows the map from (Bee, Wolf, Lion] C Vv to (Dog. Cat} C yr after using B. Under this model, a reasonable choice of classifier for R" is hez (z) Bh) Lemma 2 in App. A.2.2 shows that the expected loss of h depends on the loss hi and the conditional entropy between the label sets the tasks R' and R" and Corollary | shows the conditions for optimality of hgr..
Pre (y (yrr) B with sparse entries (i.e., only one entry of
Feature transformation (R" + R""): The final step involves changing the feature space of the distribution R" We apply an invertible linear transformation A to the distribution in R" to obtain the new distribution R"" After the transformation, the classifier associated with the new domain R"
hg (2) = h - This is illustrated in Fig, 2 after feature transform using A. Lemma 3 in

---

App. A.2.3 shows that a linear transform of the space and classifier does not incur any additional loss and Corollary [ showw that the optimality of ne implies optimality of hn... Using these, we get Theorem by defining conditional entropy as follows
H H(V (Dr)))))
Σ Σ
Pie (yw B Bg log B (gg)
(1)
Mx EDR R VR
Pw) KR
Theorem 1. Let C
be vector of probability ratios, B be KT x KR matrix with
Pn(v)
Bi = P(yR" = ilyRR = j). A: y=1
- Z be an invertible linear map of features. Let the classifiers (A)(()))))))))))) (z)). Assuming E is the cross-entropy (2) := hn(z), hRn (2) Bh (z), hRme (2) == hg (A-1AAAA))) loss. we have
(2.)) (a (((hrro (=), S Pr(cew)) (C)))((n(())))))) + H('RR (w)
Re-weighted reference loss
Label mismatch
Theorem ! provides an upper bound on the loss of the final transformed classiferrddistiiutiinn in terms of the loss of the reference classiferrdditriiuttinn. The re-weighted reference loss shows that the performance of the transformed classifier on the new domain is linked to the label-wise re-weighted loss of the reference classifier on R. This implies that one can use only the relevant reference classes to contribute to the bound. The label mismatch term shows that the performance of the distribution R" and R depends on the conditional entropy ()rr DR: B) between the label distributions of the domain R" and Rt. A high value of H implies that the labels of the reference task are unrelated leading to lower transferability, whereas a low H implies higher transferability. Corollary 3 in App. A.2.4 shows when the bound in Theorem becomes equality.
3.2 Distribution mismatch between PR" and Pr
After the three transformations, the transformed reference PRI y) can be compared with the target Pr (z, y). However, these are only simple transformations and PRI" cannot be made identical to Pr in general. This mismatch can be measured by the Wasserstein Optimal Transport distance
44 58). Since our goal is to match two joint distributions defined on Z x y we use
d((2, v), (=',,'))) = ||> 10==21p ='112 + 0o 1yzy'
with 2, Z and y. v/ E y our base distance [53] to define the (type-1) Wasserstein distance inf ((((,.))) (<'t:)) [d((2, y), (z', y')) Qa(P,Q)= =EI(P.Q)
Using Eq. 2 the Wasserstein distance between the joint distributions is the weighted sum of the Wasserstein distance between conditional distributions (P(zly)) (Lemma 4 in App. A Theorem below explains the gap between the losses due to the distribution mismatch.
Assumption 1. 1) The composition of the loss function and the classifier l h is a- Lipschitz function w.r.t to || ||2 norm, i.e., [((h(z) - f(h(zz), v)1 S F||z - z'll2 for all y E y, 2, E Z where & E H. 2) Pr( PRI
The assumption 2), can be satisfied since we have full control on the prior PR (y) via B and C. Theorem 2. Let the distributions T and R" be defined on the same domain Z x y and assumption 7 holds, then
Epr(a.) (er(e.))((((()).)))))))))))))))) v)] Pp (a.0)
[((h(=). (
S Wa(P Pr).
Distribution mitmatch
with as in Eq. 2
Theorem shows that when fo is Lipschhit then the performance gap between the R" and T is bounded by the type-1 Wasserstein distance between the two distributions. The Lipschitz coefficient of the composition can be bounded by r, by penalizing the gradient norm w.r.t at training time. Thus, for linear fine-tuning, we train the classifiers lR and for with an additional gradient norm penalty max (0, (V.(((())) x). y) } to make them conform to the Lipschitz assumption (see App. C.3) Note that constraining the Lipschitz constant restricts the hypothesis class. The trade-off between the Lipschitz constant and the performance of h is empirically evaluated in App. C.3.I
5

---

Rewnigited
RNSO
ADV(2) RN50
SIMCLR RN50
SWAV RN50
DistilRoBERTA
CIMNS COARISS
CIARI aprlioadd CHARtoD Airo oft
oro
cranto areata
pro pynzione GPARION CHARIOO Airraft
CPARO CIARTIO
DrAnL Cnar Airrant
Pets
CIPARLOP
Airrrant
Ao. Newe
YELPS
Figure 3: Task-relatedness (decomposed into its components) produces a small gap to transferability (blue bars). As the task-relatedness between the reference (ImageNet (for CV), DBPedia (for NLP)), and the target tasks (x-axis) increases, the transferability improves. (Note: the label mismatch term is zero in our figures as B is fixed to sparse matrix, see Sec. 3.4
3.3 Bounding transferability using relttdelasssness
Here, we combine the results obtained in Theorem D and Theorem 2 The final bound proposed
in Theorem R is one of our main contributions which explains transferability as sum of three interpretable and measurable gaps.
Theorem 3. Let be the cross-entropy loss, then under assumptions of Theorems and Epr(s.)) [erre.)](((h(()))) (((hr(2),
& (EPa(en) (()eemm)(())(((a)))))))) H( (Yw) + W Pr).
Re-weighted reference
Label mismatch
Distribution mismatch
The theorem shows that transferability can be decomposed into the loss incurred while transforming the class prior distribution, label space, and feature space of the reference distribution (first two terms) and the residual distance between the distribution obtained after transformations and the actual target distribution (last term). Based on the terms in the upper bound we define reltteeatssssess follows.
Definition 2. (Task-relatedness). The relatedness between target and reference task is defined as Ep (()))(()))))) + ()<)) + T Wa(PR Pr).
smaller value of the task-relatedness measure implies higher relatedness of the reference and the target tasks. In particular, when the target task is a transformation of the reference task then there exist transformations A, B, and C such that the distribution R" perfectly matches the distribution of the target task (i.c.. Wa(PR!", Pr) 0). Moreover, when labels are deterministically related (Corollary our bound becomes an equality.
Lastly, while we presented an analysis for linear fine-tuning here (for simplicity of presentation), our bounds hold for non-linear classifiers and non-linear feature transformations well (see App. A.5).
3.4 Estimating task-relatedness
The optimization problem for learning the transformations A, B, and C' to compute task-relatedness in Theorem B presented below. We use two new variables: inverse of the transformation A, denoted by A = A and a transformed reference prior distribution denoted by D(y) = CR())
D(y) (hx(2))v)) HHRRR (Dr: B, D) + 7 Wa(P""" Pr; A, B) PR(V)
min (P((..) A.A.B.D
s.t. AA = AA= = 1, Pr(y)= = BD, BB = Vj. Σ VEYR
D(g) =
(4)
Bij E [0,1] Vi and D. E [0,1] Vi.
Alg. shows how we solve Eq. (see App. D for additional details of the algorithm). Fig. g in App. C.1.2 shows how the upper bound is reduced as the optimization proceeds. Computationally. a single epoch of Alg. ttakes mere 0.17 seconds on our hardware for transfer from ImageNet to Pets for the ResNet-18 18 model (we ran Alg. [ for 2000 epochs). App. C.T. we show the effectiveness of learning the transformations using Alg I on small-scale transfer tasks. Our results show that when

---

Algorithm Minimization of the bound in Theorem
Input:
Reference task samples and labels (ZR, YR). Target task samples (Zr), Target task labels (Yr) (optional).
Output: Estimate of task aaskreelaeddness using the learned transformations A, D.
Init: A = := :1 = Pn(y), random E R
Randomly sample ne points (# v/R) (ZR, YR) as per the class prior D.
2: if Yr is available then
3: Randomly sample nr points (2) f11) (Zr, Yr).
4: else
5: Randomly sample no points (2/,) ~ (Zr).
6: # CComuute pseudo-labels for the target samples 27.
7: 1/4- arg maXyeD1r Bh 2TT for j 1, NT.
8: end if
9: Compute (2/2",, VR (Az), Ri arg maxy Be(yR)) for 1,, ,R
10: Assign Vv2 = Ya and yn == Vr.
11: Compute the optimal coupling n² between the distributions R" and T by minimizing Wa(Pne", i...
min xel((Pn,.PP)
Σ Tay Tzyd((zhem Fn (24.
1 Vi,
s.l.
Σ Jaj
Faj
Vj.
n.R.
11
12: Using n², solve for A, A, B, D using mini-batch SGD
Σ The [d((= v ) (of vf
min A.A.B.D
D(y')
Σ
Pr(V*) E(h (()((=)) w') + HD (R))
nR
+ PP - BDI 3 + (IIAA T|E + 1I I(]r).
13: Repeat 1 12 until convergence.
the reference task has classes semantically related to the target task, Alg. Dlarns transformations that achieve the smallest gap to transferability. However, since finding data semantically related the target task may not always be possible we choose reference task with the same number of classes as the target and fix that matrix to random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.
4 Empirical Analysis
Here, we empirically demonstrate the effectiveness of task-relatedness in explaining transferability in various settings. We present additional results in App. C and dataset/experrmmental details App D Our codes can be found at
com/ httpa//aappaaaeee/ssaaaatttaaaaa
4.1 Task-relatedness achieves small gap to actual transferability
Task relatedness tightly upper bounds transferability across various architectures, pretraining meth- ods, and datasets. We demonstrate this by using various pre- pre-trained models with architectures such as Vision Transformers (ViT) (0), ResNeet18/50/100//
27), DostilRoBERTa (3 trained with
various pretraining methods including supervised training, adversarial training [48] SimCLR (1]. MoCo 26), SwAV [00, and MAE [25]. We also consider wide range target datasets including, CIFAR 100, Aircraft, Pets, DTD, AG-News, Yclp-5, and SST-5 whose details in App. D
7

---

Target Loes
Reweighted Reference Loss
Distribution Mismatch
Task-relatedness
Transferability
MNIST
SVHN
Refer Referencc Backs USPS FRENIST MNIST
USPS PRINST NIN55
1.23
Reference Tarks
2.23
3.76
3.53
1.77
1.64
3.32
3.65
MIIITT FMNIST USPS
MAIST FMNEST USPS
PE PFE
PE
FFE
PE FFE
PC
FFC
Target Tasks
Target Tasks
USPS
MNIST-M
USPS
MNIST-M
(a)
(b)
Figure 4: (a) Task-relatedness and transferability are highly correlated across various reference-targee
pairs. (b) Improving the transferability of an encoder on reference task (in the plot title) leads to
improved transferability of all related target tasks (x-axis). (e.g., compared to the original pre-trained CLIP encoder (PE), end-to-end fine-tuned CLIP encoder (FFE) on the reference task achieves higher transferability all related tasks.)
For this experiment, we fix the reference task be ImageNet [17 for image classification and to DBPedia for sentence classification tasks and use Alg. to estimate task-relatedness. The results in Fig. and [ (in the Appendix) show that our bound achieves small gap actual transferability. As the task-relatedness between the reference and the target tasks improves, transferability also improves showing that relkedelated and transferability are strongly correlated. Task- reak-reelasdness is also strongly correlated with the accuracy of the end-to-end fine-tuned classifiers on the target task. In Fig. [i the Appendix). we show high Pearson correlation coefficients (2 -0.57) for task-relatedness and accuracy after fully fine-tuning various pre-trained encoders using data from various target tasks.
4.2 Effect of the reference task on task-relatedness
Highly related reference- target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks. To understand how reference task affects task-relatedness and eventually transferability, we consider two experiments using convolutional and CLIP-trained models with various character recognition tasks such as MNIST, Fashion-MND (FMNIST), SVHN, MNIST-M, and USPS. Of these datasets, SVHN and MNIST-M contain colored images while the rest contain gray-scale images. In the first experiment, we train convolutional models on MNIST, FMNIST, and USPS and measure pairwise transferability. Here we use the reference task to be the same task that used for training the models. The results in Fig. 4a) show that transferability to those target tasks is higher for which lask-relatedness metric's 's value smaller. Specifically, USPS achieves the best transferability (1.23) and the smallest meskerelatedness (2.80) when the reference task is MNIST. This is attributed to both datasets containing gray-scale images of digits. On the other hand, when the reference task is unrelated to the target task i.c. the lask-relatedness value is high, transferability suffers, e.g., when the reference task is MNIST and the target task is FMNIST. Results in App. C.2.2 show similar results for the sentence classification task.
The gap between relttrelass and transferability is smaller when reference task performs well with a given encoder. Here we use MNIST and SVHN as two reference tasks and compute the task-relatedness and transferability with USPS and MNIST-M as target tasks, using CLIP (Vit B32) model. A linear classifier trained on top of the embeddings from the CL.IP model achieves 9998% accuracy on MNIST but only #619 % accuracy for SVHN, Due to this, transferability (USPS:2.02, MNIST-M:2.200 explained using task-relatedness with MNIST as the reference task (USPS:2.46, MNIST-M:2.48) is better than that computed using SVHN (USPS:2.66, MNIST-M:2.655 as the reference, even though MNIST-M is intuitively more similar to SVHN (as both contain colored images of digits). This is evident from the results of PE (Pre-trained Encoder) in Fig. b).
Improving the performance of an encoder on reference task improves transferability to other related (potentially unseen) lasks. show this we fully fine-tune the CLIP encoder on MNIST and SVHN tasks, increasing the accuracy of the classifiers for both MNIST and SVHN to 99% and 95%, respectively. Using the representations from these new encoders, we find that the transferability of both related target tasks improves along with task-relatedness (see FFE results in Fig. 4b). Here,
8

---

Table I: Task-relatedness achieves high (negative) Pearson correlation to the accuracy after end-to-end
fine-tuning for various tasks. For NCE [56], Leep [40] LogMe [6]]. SFDA [1]], OT-NCE, OTCE
[55], and H-score [6] positive correlation is better whereas for PACTran 19 and task-relatedness
(ours) negative correlation is better.
Target task
LogMe
Leep
NCE
PACTran
SFDA
H-Score
OT-NCE
OТСE
Ours
Pets
0.82
0.80
0.73
-0.82
0.37
0.77
0.88
0.86
-0.77
DTD
0.88
0.96
-0.19
-0.85
0.90
0.89
0.84
0.82
-0.97
Aircraft
-060
0.92
0.97
0.11
0.72
-0.80
0,56
0.60
-0.72
Average
0.37
0.90
0.50
-0.52
0.73
0.29
0.76
0.76
-0.82
we see that task-relatedness for MNIST-M and USPS is the best when the reference task is SVHN and MNIST, respectively, aligning with our intuition of semantic relatedness between these tasks. This also suggests that transferability on other related tasks can be improved by fully fine-tuning the encoder on these reference tasks. Thus, in scenarios where target tasks are private (such as proprietary Chest X-rays), an encoder trained to work well on related tasks (such publicly available Chest X-rays) is bound to achieve good transferability.
4.3 Task-relatedness for end-to-end transferability estimation
In this section, we show an efficient way of computing task-relatedness which enables its use for estimating transferability after end-to-end fine-tuning. While Alg. [ accurately estimates task- relatedness by minimizing the bound in Eq. 3 it could be inefficient due to the requirement of computing and minimizing the Wasserstein distance between distributions at every epoch. Thus, to make the computation efficient, we replace the Wasserstein distance computation in step 11 and 12 of Alg. [] with mean and covariance matching terms. Specifically, we define the distance between two distributions R" and T as
I(R",T) = I1RR"" - PT1 + ||ERe ET
(5)
/T), where /RR" "yl" fr ExEPR'IYT 2, ERR" IT Rnp"" pr E (2 PRR" )T (z /
and A is regularization coefficient. Using I(R", T) I in place of "T"RRTT)) makes the computation of task-relatedness by learning transformations A, B, and C' significantly more efficient.
Task-relatedness is an effective metric for the pre-trained model selection problem. The goal of this problem is to find a pre-trained model from model zoo that achieves the best accuracy target task after end-to-end fine-tuning of the model using labeled target data. Since and-to-end fine-tuning is costly (takes almost day to fully fine-tune a single model on a single target task as shown by (61). an effective transferability metric is significantly more efficient to compute and is correlated well with the accuracy after end-to-end finetuning. Using 5 different pre-trained models (supervised ResNeet500101/152 adversarially pre-trained [48] ResNet50 with E (0.1, 13) and ImageNet as the reference task, we show Table [ that task-relatedness achieves a high correlation with the accuracy after end-to-end fine-tuning on the target task. Our results also highlight the instability of various popular SЫТE metries, such as LogMe (61] and NCE (56) which can produce a high negative correlation, and those of PACTran [19] which achieve low correlation values on complex datasets. In comparison, task relateelasssness consistently achieves a good correlation for various target tasks. Computationally. it takes mere 3-4 minutes to learn the transformations to compute relkteelas,,ssss, providing a significant computation advantage over end-to-end fine-tuning. We also show that task-relatedness remains highly correlated with end-lo-end fine-tuning accuracy even with a limited amount of labeled data from the largel task as shown in Fig. 5 unlike other SbTE metrics.
given
Next, we show that task-relatedness can even be estimated without using labels from the target task. For scenarios, where labeled data from the tar- get task is unavailable, estimating transferability challenging. This is because both fine-tuning and most SbTE methods require labels to com- pute the transferability scores. Here we show that fask-relatedness can still be an effective measure to estimate transferability in this challenging setting. Since we use a transformative model and have ac- cess to a reference task/classifier, we can use the predictions from the reference task's 's classifier trans-
Table 2: Correlation of task-relatedness and end-to-end fine-tuning accuracy computed us- ing true and pseudo labels of the target task.
True
Pseudo
Target
labels
labels
Pets
-0.77
-0.76
DTD
-0.97
-0.91
Aircraft
-0.72
-0.16

---

LogME
Leep
NCE
OT-NCE
0.5
0.9
0.5
0.8
-0.5
0.7
-0.5
0.5
0.25
0.50
0.75
1.00
0.25
0.30
0.75
1.00
0.25
0.50
0.75
1.00
0.23
0.30
0.75
1.00
Percentage Hscoor
PAC Iran
OТCE
Ours
-0.75
0.8
0.5
-0.50
-0.25
0.7
0.0
-0.8
0.6
-0.5
0.5
0.7:
0.25
0:25
0,50
1.00
0.25
1.00
0.25
0.50
1,00
Percentage target data (%1
Percentage target data
Percentage target data
Percentage target data
Pets
DTD
Aircraft
Figure Task-relatedness (Ours) remains highly correlated with accuracy after end-to-end fine-tuning target task when using small percentage of target data unlike other SÉTE methods (LogME, Leep. NCE, PACTran, OT-NCE, OТCE, and H-Score) whose correlation is affected significantly. For LogMe, Leep. NCE, OT-NCE, OTCE, and H-score positive correlation is better whereas for PACTran and task-relatedness (ours) negative correlation is better.
formed via B (to obtain labels E 3/)) and estimate the pseudo-labels of the target data. Concretely,
pscudo-label for target sample Ty is obtained as yr ueseudo arg maXye>'> Bh re (11- (2r)) Results show that our task-relatedness estimated via pseudo-labeled target data still achieves a high correlation to transferability most datasets. For datasets such Pets and DTD, where transforming the reference task classifier produces high accuracy on the target task, the difference between the pseudo and true labels is small. Consequently, the difference in the correlations with pseudo and true labels is also small. Thus, when the reference and target tasks are related, transferability can be estimated accurately without requiring labels of the target task, showing that task-relatedness is an effective metric even for unsupervised transferability estimation.
Conclusion
We analyzed TL in terms of the relatedness between the target and reference task. Our analysis works by transforming the distribution a reference task to match that of the target. Using this we proved an upper bound transferability, defined mesk-relatednes, consisting of three interpretable terms, namely, the re-weighted reference task loss, label mismatch, and distribution mismatch. We proposed algorithm compute task-relatedness and demonstrated its effectiveness accurately predicting transferability (even without target labels) with SOTA models. Moreover, the high correlation of task-relatedness with accuracy after end-to-end fine-tuning and its efficient computability, makes it an effective metric for transferability estimation.
Limitations. We studied transferability using the cross-entropy loss and used Wasserstein distance- based distribution shift analysis due to their popularity. However, due to accuracy being the primary metric of interest in classification tasks and the difficulty of computing the Wasserstein distance with limited samples in a high dimensional representation space, extending the analysis to 0-1 0-1 loss and other divergence measures important directions which are not addressed here and are left for future works.
Acknowledgment
We thank the anonymous reviewers of this work for their insightful comments and suggestions. This work was supported by the NSF LPSCoR-Louusiana Materials Design Alliance (LAMMDA) program #0IA-1946231.
10

---

References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9430-6439, 2019.
[2] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and loannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804, 2019.
[3] David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. arXiv preprint arXiv:2002.00993, 2020.
[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223 223. PMLR, 2017.
[5] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theonee approach to transferability in task transfer learning. In 20/9 IEEE International Conference on Image Processing (ICIP), pages 2309-23 23132233,2222 2019.
[6] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng. Amir Zamir, and Leonidas Guibas. An information -thooretic approach to transferability in task transfer learning. In 2019 IEEE international conference on image processing (ICIP), pages 2309-2313, IEEE, 2019.
[7] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. theory of learning from different domains. Machine learning. 79(1):151-175, 2010.
[8] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of represen- tations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
[9] Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning theory and kernel machines, pages 567-580. Springer, 2003.
[10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924 2020.
[11] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 15597-1677. PMLR, 2020.
[12] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021
[13] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[14] Quan Cui, Bingchen Zhao, Zhao-Min Chen, Borui Zhao, Renjie Song, Boyan Zhou, Jiajun Liang, and Osamu Yoshie. Discriminability- transerrtttansffaabbiltty trade-off: An cettmmatiin-thhoreede perspective. In European Conference on Computer Vision, pages 20-37. Springer, 2022.
[15] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region- based fully convolutional networks. Advances in neural information processing systems, 29, 2016.
[16) Bharath Bhushan Damodaran, Benjamin Kellenherger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 447-463, 2018.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009.

---

[18] Jacob Devlin, Ming- deep bidirectional transformers for language understanding, 2019.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
[19] Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac- bayesian metrics for estimating the transferability of pretrained models to classification tasks. Springer, 2022.
In European Conference on Computer Vision, pages 252-2688
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[21] Kshitij Dwivedi, Jiahui Huang. Radoslaw Martin Cichy, and Gemma Roig. Duality diagram similarity: a generic framework for initialization selection in task transfer learning. In European Conference on Computer Vision, pages 497-513. Springer, 2020.
[22] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 12387-12396, 2019.
[23] Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aureie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, al. Pot: Python optimal transport. The Journal of Machine Learning Research, 22(1):3571-3578 2021.
[24] Jiechao Guan and Zhiwu Lu. Task relatedness-basct generalization bounds for meta learning.
In International Conference on Learning Representations, 2022.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[28] Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Under- standing few-shot learning: Measuring task relatedness and adaptation difficulty via attributes. Advances in Neural Information Processing Systems, 36, 2024.
[29] Long- Kai Huang, Ying Wei, Yu Rong, Qiang Yang, and Junzhou Huang. Frustratingly casy transferability estimation, 2022.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Certifying model accuracy under distribution shifts. arXiv preprint arXiv:2201.12440, 2022.
[32] Trung Le, Tuan Nguyen, Nhat Ho, Hung Bui, and Dinh Phung. Lamda: Label matching deep domain adaptation. In International Conference on Machine Learning, pages 6043-6054. PMLR, 2021.
[33] Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, and Soheil Kolouri. Wasserstein task embedding for measuring task similarities. arXiv preprint arXiv:2208. / /1726, 2022.
[34] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[35] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi, Fine- grained visual classification of aircraft. arXiv preprint 17X15 2013.
12

---

[36] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadch. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.34400 2009.
[37] Akshay Mehra, Bhavya Kailkhura, Pin- Yu Chen, and Jihun Hamm. Understanding the limits of unsupervised domain adaptation via data poisoning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[38] Akshay Mehra, Bhavya Kailkhura, Pin-Yo Yu Chen, and Jihun Hamm. Do domain generalization methods generalize well? In NeurIPS ML Safety Workshop, 2022.
[39] Akshay Mchra, Yunbei Zhang, and Jihun Hamm. Test-time assessment of model's performance on unseen domains via optimal transport. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 173-182, June 2024.
[40] Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: new measure to evaluate transferability of learned representations, 2020.
[41] Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. Exploring the role of task transferability in large-scale multi-task learning. arXiv preprint arXiv:2204. 11117, 2022.
[42] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1355 2010.
[43] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 1498-3505 IEEE, 2012.
[44] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 11(5-6):355-077, 2019.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
[47] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[48] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do ad- versarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3545, 2020.
[49] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
[50] Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint 1rXiv:2104.09425. 2021.
[51] Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models equal: predicting model transferability in self-challenging fisher space. In European Conference on Computer Vision, pages 286-302. Springer, 2022.
[52] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[53] Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some dis- tributional robustness with principled adversarial training. arXiv preprint 17X0v 2017.
13

---

[54] Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao. and Mingli Song. Depara: Deep attribution graph for deep knowledge transferability. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 39922-3930, 2020.
[55] Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations, 2021.
[56] Anh T. Tran, Cuong V. Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks, 2019.
[57] Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity, Advances in neural information processing systems, 33:7852-7862. 2020.
[58] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[59] Karl Weiss, Taghi M Khoshgoftaar, and Dingding Wang. survey of transfer learning. Journal of Big Data, 3(1):9, May 2016.
[60] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms. arXiv preprint 178.::707.0077777 2017.
[61] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning, 2021.
[62] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722.
[63] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
14

---

Appendix
We present the missing proofs of the theoretical results from Sec. along with justifications for the classifiers (hg', , as Corollaries Appendix (ffollowedd by related work on learning in the presence of distribution shift with the same feature and label space in Appendix B This is followed
by additional experimental results including NLP classification tasks with large pre-trained models in Appendix C
We conclude in Appendix D with details of the experiments and datasets used.
A
Proofs for Sec.
A.1 Notation
Table 3: Table of notations
Data related
R/T
Reference/Targe Target task.
AR/T
Images of reference/target tasks.
VnfT
Label set reference/target tasks.
KR/T
Number of classes in reference/target tasks.
Purr(a,0 v)
Data distribution of reference/target tasks.
(ZRT,, YR/T)
Samples (features and labels) of the reference/target tasks.
Model related
9
Encoder pre-trained pre-preeiiiiie pre-training dataset.
nn/T hR/T {
Representations extracted for reference/target tasks using g.
Classifier learned for reference/target tasks the representations of g-
Cross-entropy loss.
Lipschitz constant.
Task-transformation related
A
Parameters for feature transformation.
Parameters for label transformation.
C
Parameters for class- class-prior transformation. Distribution and classifier of R' after applying transformation C on R.
Pre (z, y) and h
Pro (2, y) and hn
Distribution and classifier of R" after applying transformation B on R'
Pr (2, y) and g
Distribution and classifier of R"" after applying transformation on R" Conditional entropy defined in Eq.
H1
d
Base distance between two samples as defined Eq. Wasserstein distance between two distributions defined in Eq.
Wa
Distance between two distributions based on their mean/variance defined in Eq. 5
A.2 Our task transformation model (Sec.
3.1
A.2.1 Class-Prior transformation (R R')
Lemma Let C := (v) KR TR(P)
be a vector of probability ratios and the classifier h (2) =
y=1
hR(2), then (x.)) (( (2), Epa(s.)) ((())(((R)))) v)], for any loss function E.
15

---

Proof.
( (=),
(((h))(=))
Σ (emm))l)))) yEYa
PR(U)
Σ
PR (enn - Σ
PR(V) Pet/)Berrrccil10000000000
PR(V)
yEVR
VEVR
E Palu)BpaamlCW) (since Pn(=lu) = Pre (=lu) by construction) vEYn
A.2.2 Label transformation (R² R")
Lemma 2. Let be a Kr x KR matrix with Bsj P(yR" = ilyre f) and hR (z) (z)x(z) and l. be the cross-entropy loss. Then, (ppn(s.v) (2), y)1 S
where H(RR))RRR is the conditional entropy (- E EoRnEDer PR (yR) Bug My log (ggBBagree
Bp (2) + H(YR=
Proof. Note that P(z) = PPR = (2)(z) by construction.
((hR) (2), = (x)
100 (2), Er(x)
Σ
Σ
Ply", (12) (since s e yR.
V" W
Er(:) w (1th)))))
= Br E (2) (B)","))))rr (since Byn.y' = P(y"
(2), (since P('"'v',' P(V"lv)))
= Ep(y) Σ
"'EVx"
Er(zs) Σ y"E
(B)
Since the loss is the cross-entropy loss, we have
(2), = -llog( Σ
By (x) < - (x) = - -lo((By - 1og(h((
jEV
16

---

Therefore, we have
E (xs))))))
- Ep(),))
Σ
(Bhh((2))"))Brrr))
8"EVn"
s E y"EVR"
Σ
By (log(By'.>)) log(h (x)))
P(
Σ
By log (yy Bp(an)) Σ B log(h (x)))
V'EYR"
V"E)aR"
Σ
By (By Br(x,r) log(h
- (z))
E
Bywl
V"EVR"
V'EVR"
1
Σ
y"EYnn
Σ
By log
(B + (p
V'EVi"
Σ
By log (B, vf ) + E (2), v/)l
=
V"EN
- E E P(y)) By" log B/ + (e
V'ER EY""
H('R VRr) + Ep (z.)) ((hr)
Corollary [eclow, shows the conditions under which the optimal softmax classifier for the domain R' remains optimal for the domain R" justifying our choice of classifier change from R' to R".
Corollary 1. Let be one-hot encoding of the labels, 1a'' D. B be KT x Kn permutation matrix and hg be the optimal softmax classifier for R' and YR" == ayRR arg maxyeY<> (Be(yR) )) then under the assumptions of Lemmal hR (z) == (bx((z) is the optimal softmax classifier for R..
Proof. Since yru" == =(yn)) = arg max (Be(yr)) we have
Epn,. (((h)e(()) )nr) = Ep(z.))) (BBhrr))((BB)))))
Σ
1/" "'EVR"
Σ Plots)))Ertooon)(((B(x))))))) (2)
V'EY'R'
Σ PllEEaaal((hx()) Ep (e(l) (R)
y'El'
The second last equality follows due to the symmetry of cross-entropy loss, i.e., l(h, v) log hy - log Bho(y) = {(Bh, (w))
Since mink q4 Ep [e (2), yR )] ming Epg [e( yz), and hg is optimal for R' we have h (2) Be (2) is the optimal softmax classifier for R"
A.2.3 Feature transformation (R" R"")
Lemma 3. Let A Z + Z be an invertible linear map features and the classifier hg (z)
hg (A (A-((z)) Then Ep (e(h (z), v)] = Ep (z.2) [6(h R² (z), v)1 for any loss
Proof. E. (pp (() = E P ((ll
v) Eppn(E.)) ((
(((h (2), (z),
17

---

Our Corollary blbllow shows that the optimal softmax classifier for domain R remains optimal for domain R" too.
Corollary 2. Let R be the optimal softmax classifier in domain Rf then under the ussumptions of Lemma AR (z) = hRo (z)) ) is the optimal softmax classifier in domain Rt"
Proof. When hR)e(2)
hg (A)) ming g (p (e [e(h (2), y)]
min Reg (z.3)) ) (e (0)m)(()), ()),)] by Lemma hence if hg is optimal for R" then so is hg for the domain R"
A.2.4 Three transformations combined (R -+++10 I(")
Let C: = Par(y) Kn
Theorem
be vector of probability ratios, B be Ky x Kn matrix with
PRlv) y=1
Bij = P'yR" = -llyR = j). A: Z - Z an invertible linear mup of features. Let the classifiers (2) == hR(2), (2) == Bh (2), (nme(2) lg (A- (z)). Assuming i is the cross-entropy loss, we have
E Pp ((( (=), v)1 s EPR(Z.)) ((()) + H(YRR VR.)
Re-weighted reference task loss
Label mismatch
Proof.
E Ppon((a)) (e (2), (a (ohher (Temma 3
S ( (elg) (2), v) + H (RRe) (Lemmnn 2
= Epa(Z (()))((nn))) v) + H (vr) (Lemma
Corollary 3. Let e be one-hot encoding of the labels, [Y = YRl. B AR. + Age be permutation matrix and yn" (yy)) arg maxyeY gm (Be(yn)) By then under the assumptions of Lemmas and 3 we have T P (2.g) [e(hn y) = T pa(())) (
((v))(hn(2).v))
Proof. Since yRu = =(yw)) = arg max,eax, (Belyr)) we have
Epg (()rre())) yn") = E (( (B
Σ P(y" [
v"EVR"
Σ
P(o(y')) Plot)))Erarnn)(((BB())).)))
v'EV'
Σ Pin)Erra B ((
d'EW'w'
The second last equality follows due to the symmetry of cross-entropy loss, i.c. e(h, v) log hy - log Bha(y) = L(Bh, (2)).
Therefore, have
E Epgnnte.)) (=),
= (p. [e (=), v) (Lemma 3
(py() [((h (2), v)] (from above)
EPa(z (()) (Lemma
A.3
Distribution mismatch between R" and T (Sec. 3.2
Lemma 4. Let U and Q be two distributions on Z x y with the same prior Poly i)
=
Poly = i) P(y = i). With the base distance d defined as Eq. 2 we have W Wa(Pu, Pq)
18

---

Proof. Let wy denote the optimal coupling for the conditional distributions (2)()) (o(2))) for y E Y and n- denote the the optimal coupling for the joint distributions (z, y). Po (2(2,)))
Then, under the definition of our base distance d, x ((2, y). (=',0') mass from the distribution U belonging to class can be moved to the classes y * of the distribution Q when the class priors of U and Q are the same. Moreover, since Σig (wy) 1 and
when y 7 yf i.e. no
E (1 Tij = P(y = k) for E y we have x every y.
= w P(y) for
Then, we can show that the total Wasserstein distance between the joint distributions can be expressed the sum of conditional Wasserstein distances, follows
Wa(Po v). Po(2.8))
Σ
/
x (2. (2', (1. (2) (2', (2) Mddde'
(Z..)), 0||= - = 00. 1y Jdzdz
V.v
Σ/
=112 co yyt dzdzz
V.V'
Σ / (((=,,)))(())))-pl) 2' 2'1lzdzdz' (since yy= Tuty' = 0)
V.v"
Σ P(y)ly=w / (1)(2,0)1l= 2'112dzea'
B.y
P(v)) / (G((,2")ll= - z'1ladzdz
E (P
Theorem 2
Let the distributions T and R" defined on the same domain Z x y and assumption holds, then E Bp (e(h(2), v)] pg)r() ((h(2))))) < Wa(Pie", Pr), with d as in Eq.
Distribution mismatch
Proof.
(pr(z.)) [[(h(2), ((h()))))(((())) w)]
EE [((h(()),)))) ((.)))((()))))) w)1
Epr(y) Ep (aly) (((h(2).)) y)] E (1) Ep (ela) (e( (c(h(2).
Brr(s) (Exr()))) [e(h(2), (((()))) w)] - Er (=1y) [((h(2)))))))))))))())) w)]l (since Pr(y) PR (v))
Epr(t) sup PoN'ET Lopssh---psschitt
Epr(*l) (f'(h' v)] P (#1a) 16' (e'(('(())) (h'(2), (=),v)]]
EPr(g)TT Wii (Pr (zly), PR (z [)) (Kantorovich Rubinstein duality)
T Wa(Pnve, Pr) (Lemma
A.4 Final bound (Sec. 3.3
Theorem 3 Let l. be the cross entropy loss then under the assumptions of Theorems Aand 2 we have,
E Ep)( [((hr()). & (C) (P))ee)lC(t)((hha(()...)) H( (in)) + Wa(PRun, Pr)
Re-weighted reference task loss
Label mismotch
Distribution mismatch
19

---

Proof. Let lo hr, to ha, and lo hg be Lipscchit (from Assumption 1
(pr(e.)(((hr))):))) I Epr(s.)) (s)) (Optimality difference)
s (p (1) v)] +7 Wa(Pne"; Pr) (Theorem 2
Bra(z.)) (()) v)] H(RR" (R
Wa(PRi Pr) (Theorem
In our experiments, we enforce the T-Lisschitz constraint for f hn and & hr and verify that the Lipschitz constant of & hnn remains close within tolerance.
A.5 Extension to non-linear classifiers and non-linear transformations
To extend our analysis to non-linear classffirr/transoormations,, we allow A 2 to be non- linear map and the classifiers h Jinea to be also non-linear (such as multi-layer perceptron). In addition to the Assumption (1), which requires o hg to be T -Liiscchitz, we also require that hg belongs the same class Hnon Jinear Jlnear ht and hg for any A. This holds for example when h linear and A is also linear or when is a multilayer perceptron and A is linear. With this additional assumption, all of without any change. Thus, Theorem [ Theorem and Theorem hoold for non-linear classifiers as well. With these extensions, our bounds can be used to explain transferability with non- non-linear classifiers.
proofs work for any linear/non-lineal transformation of the feature space,
B Additional related work
Distributional divergence-baser analyses of learning with distribution shifts (under same feature and label sets): Here we review some of the previous works that analyzed the problem of learning under distribution shifts in terms of distributional divergences such as the Wasserstein distance. These analyses apply when the feature and label spaces remain the same between the original and the shifted distribution.
showed that the performance on shifted distribution (target domain) can be estimated in terms of the performance of the source domain and the distance between the two domains' marginal distributions and labeling functions. Specifically, [8] showed that that
Early works 8
12 3.
-2 (h, .r) S (h, fs) + di (Pss Pr) + min{Er, [[s(s((s(=)))))))))))))>>>{))>{) - fr (=)]), Ep, [n
where dj denotes the total variation distance, f: Z [0, 1] denotes the labeling function, Z + {0. 1) denotes the hypothesis and Ep(h, == p(/(() - f(x)]] denote the risk of the hypothesis A follow up work (52], showed a similar result using type-1 Wasserstein distance for all K -Lisschitz continuous hypotheses i.e.,
(hh & Es(h, 2K - (rs, +
where is the combined error of the ideal hypothesis h² that minimizes the combined error Es(h, fs) + (h, fr). Another recent work (32) used a target transformation based approach and Wasserstein distance to quantify learning the presence of data and label shifts. Other works 31 50] also presented an analysis based Wasserstein distance to understand how the accuracy of smoothed classifies and robustness change the presence of distribution shifts. Compared to these works the bound proposed in Theorem considers cross-entropy loss (which is popular choice of the loss function in the classification setting) and uses a joint feature and labels Wasserstein distance rather than only marginal Wasserstein distance. These differences make the bound proposed Theorem 2 useful in the analysis transfer learning than those proposed in previous works when we have access to labeled target domain data.
2
Comparison with [56] The closest work to ours is that of [56), which showed that transferability to target task can be related to transferability to another task (source task in their work) and the label mismatch between the two tasks. However, the bound is proposed the tasks have the same features but different labels (i.e. same images labeled differently between the two tasks). In this setting, (56), showed that transferability is upper bounded by loss of the source classifier the source task and the conditional entropy (CE) of the label sets of the two tasks. We
restrictive setting when both
20

---

significantly extend this analysis to general tasks which is the most commonly used setting in practice (e.g.. our analysis allows us to study transfer learning from ImageNet with 1000 classes CIFAR- 100 with 100 unrelated classes, where both tasks have different images). In this setting, our main result in Theorem 3] shows that transferability involves additional terms such as the distribution mismatch term (in the form of Wasserstein distance), the prior mismatch term (in the form of re-weighted source loss) and the conditional entropy between the label sets. Moreover, the bound proposed by [56) is special case of our bound with C' being the vector of all ones (no prior change) and A being Identity (data distributions of two tasks are the same).
Comparison with other transferability estimation and model selection methods:
The problem
of transferability estimation has gained lot of attention recently, especially due to the availability of a larger number of pre-trained models. Earlier works used models end-to-end fine-tuned on target tasks to evaluate transferability via task-relatedness estimated using the actual target loss [62] and the Fisher information matrix
[0. However, the requirement of models trained on target tasks limits their practical utility. Other works [22 21 54 used the representation of the data from the tasks from these end-to-end fine-tuned models and developed similarity scores that achieved high correlation with transferability. However, these approaches are not practical since they require models end-to-end fine-tuned on the target task. Our work does not depend on models trained on target tasks as shown in our analysis (Theorem
3) making it both theoretically and practically sound.
Another line of work [5 40 29 61 55 51 [99 focuses on the problem of pre- pre-trained model selection where the goal is to find the best pre-trained classifier from a model zoo that will achieve the highest transferability to a particular downstream task. Thus, the main challenge of this problem is to be able to estimate transferability in way that is more efficient than fine-tuning the pre-trained models on the downstream tasks. To this end, recent works have proposed several scores that are correlated with the accuracy of the models after fine-tuning them on the target task, which we refer to as score-based transferability estimation (SbTE) methods, in our work.
However, unlike our work, the goal of SьТE works is not propose universal bound or identify terms that provably govern transferability. Moreover, while the scores proposed in SbTE works correlate well with transferability. they are only meaningful in a relative sense. Concretely, a score of 1 (c.g., for LogMe [46]) on CIFAR-100 task for a particular model does not indicate whether transferability is good or bad and requires comparison with scores of other pre-trained models on the same target task. On the other hand, our upper bound directly approximates transferability, e.g.. task for model implies that transfer learning will incur an
an upper bound of on the CIFAR- average cross-entropy loss of less than I implying a highly accurate transfer. Our results in Figs. 3 and 9 attest that our upper bound is indeed a good estimate of the transferability.
Another disadvantage of the scores proposed in these works is that they cannot be compared across target tasks, unlike our upper bound. As observed from Fig. 4 of LogMe [46], scores for CIFAR- 10 are lower than scores for CIFAR- 100 on the same pre-trained model, but, the transferability to
CIFAR-10 better than that CIFAR- 100. On the other hand, from our Fig. [ the upper bounds on CIFAR-10 are lower than those of CIFAR- 100 implying better transferability of classifiers pre-trained on ImageNet to CIFAR- 10. Thus our work is more suitable for estimating the absolute performance on various target tasks given particular pre-trained classifier.
Thus, while the goal of our work differs fundamentally from SbTE approaches the problem addressed in this work is of significant practical importance. In Sec. 4.3 we show that task-relatedness estimated via Alg. [ achieves competitive performance compared popular SbTE methods on this problem.
Comparison with task transfer learning approaches: This approach focuses on explaining the transfer performance based on the relationship between tasks. For instance, works such [28 57 (24] study the problem of few-shot lcarning (FSL) where a model is trained on data from related training tasks and is adapted to unseen task using only a few samples. Different from these works, we focus on the setting where a pre-trained encoder trained on some pre-training dataset is adapted with enough samples/gradient steps a downstream task. This downstream target task may or may not
have any relation to the pre-training task unlike 28 57 24
proposed a model-agnostic metric called Task Attribute Distance to gauge the success of transfer. Our work, on the other hand, defines task-relatedness based on the similarity of the representations of reference and target tasks in the representations of the pre-trained model (and is model dependent) rather than relying on the attribute information, which may not be available in TL
Concretely, 2
21

---

setting. [57] analyzes the sample complexity for learning model shared across tasks and adapting it to new target task and shows task diversity to be a crucial component for the success of transfer in the FSL setting. Our work on the other hand does not assume access to shared tasks or restrict the number of samples required for fine-tuning on the target task. Moreover, their notion of task diversity requires access to a set of training tasks that may not be available in the TL. setting, making our notion of task-relatedness more practical for TL. Lastly, [24) proposes a notion of task-relatedness for the FSL setting, allowing to utilize all the data from available training tasks to help learn model on a new task with a few gradient steps. This notion is model-agnostic and defined over the sample space (X' x Y) unlike measure which is defined in the representation space of the model whose transferability needs to be evaluated.
Thus while reltedelas is at the core TL, the notions proposed by 28 57 24] are suitable for task transfer setting where as our notion is more suitable for the inductive TI. setting.
C Additional experiments
C.1 Small-scale experiment to evaluate Alg. 1
1) Our algorithm produces transformations that achieve a better value for the bound compared to naive transformations. 2) Having the same number of classes in the reference task as in the target task leads to the best value of the bound. 3) It is only marginally better to have semantically related classes in the reference task.
We evaluate Alg. [or computing task-relatedness and predicting the transferability of a R8sNet-18 model to CIFAR-TO in two settings. In the first setting, we consider a reference task that comprises data from 10 classes chosen at random and 10 classes that are semantically related the labels of CIFARIO from ImageNet [17] (see App. D whereas in the setting we select reference task with 20 classes. We compare the cross-entropy loss on CIFARIO obtained after linear fine-tuning (transfcrabilitt) with our bound, by using various transformations including those learned via Alg. We test 3 different cases. The first is FixedAll: where all transformations are fixed (A is fixed to the Identity matrix, B is fixed to random permutation matrix in the setting with 10 classes, and to a matrix such that two reference classes match a single target class in the set- ting with 20 classes, D is fixed to the reference prior). The second LearnedA: where we use Alg. too learn only A while B. D are fixed as in FixedAll, and lastly LearnedAll: where all the transformations are learned via Alg. [ Our results in Fig. 6 show that in both settings using FixedAll, our bound is marginally better when the reference task has classes semantically related to the target task. In all cases, the bound becomes significantly better in the LearnedA setting due to learning the transformation A that transforms the reference distribution to better match the target and decrease the distribution mismatch term of the bound. Lastly, by learning all the transformations, in the LearneAll setting, we achieve the best value for the bound, regardless of whether there are randomly chosen or semantically related classes in the reference task. More- over, we see that the value of the bound estimated when the reference task has 10 source classes is better in all cases. This is due to smaller re- weighted reference loss since learning 20-way classifier is more challenging than learning 10-way classifier, especially with the Lipschitz constraint. In the setting with 20 classes, Alg. prefers to retain data from 10 of the 20 reference task classes, Fig. (right), reducing the re-weighted reference loss, makes B sparse, reducing the label mismatch term, and aligns the reference and target distributions, via A, reducing the distribution mismatch term.
Using 10 reference classes
2.402.522.51
2.602.502.49
Loss
Using reference classes
0.203.083.00
FixedAll LearnedA
FixedAll
LaurnedA
LearnedAll
Random Classes
Semantic Classes
Target 2.14
Reweighted Reference Less
Label Mismatch
Distribution Mismatch
Figure 6: Transformations optimized
using Eq. 4 produce a better task- relatedness than obtained by naively cho- sen transformations, primarily due to de- creased distribution mismatch. The pres- ence of the same number of classes the reference as those in the target achieves the smallest task-relatedness value. Se- mantically related classes in the refer- ence task help but only marginally.
22

---

Overall, our results show that 1) learning D prefers to retain the data from the same number of reference classes as those present in the target, 2) the B matrix eventually becomes sparse, making the label mismatch term zero, and 3) there is only small difference in task-relatedness between LearnedA and LearnedAll settings. Based on these insights, we use Alg. | in the LearnedA setting, with the reference task containing the same number of randomly sampled classes from ImageNet as the same number of classes in the target task (since it may not always possible to find semantically similar classes for all target tasks) and fix B to be random permutation matrix, for all other experiments in the paper. We select classes for the reference task from ImageNet due to its diversity and the fact that it has a large number of classes. However, any dataset can be used to define the reference task (e.g.. we used Digits datasets for the reference task in Sec.
Next, we evaluate the effect of knowing the exact matching between the labels of the reference and the target tasks in comparison to random permutation. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in setring where only A is learned and B set to an identity matrix and when B is set to random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9).
We find that the upper bound obtained when B is fixed to identity is only marginally better than the case when B is random permutation. Specifically, the difference between the bound when B is fixed to a random permutation and when B is an identity matrix is 0.10 for the MNIST-+USPS task and 0.17 for the USPS MMNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term. While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related such as for FMNIST and MNIST. Thus, fixing B to random permutation matrix yields a reliable estimate of transferability in most cases.
C.1.1 Visualization of the transformed data via t-SNE for various settings in Sec.
Here we use the setting considered in App. C.1 where we consider 20 randomly selected classes from ImageNet as the reference task and consider the transfer to CIFAR-IO. We plot the results of using different transformations using t-SNE to show how various transformations affect the upper bound in Theorem 3] Our results in Fig. [efft) show that when no transformations are learned (FixedAll), the 20 random reference task classes do not overlap with the 10 target classes leading to an increased Wasserstein distance which in turn leads to larger upper bound. By learning the transformation A (LeamedA), Fig. center) shows significantly better alignment between the classes of the reference and target which leads to a decreased Wasserstein distance and hence a tighter upper bound. Moreover, by learning all the transformations (LearnedAll), Fig- rright) shows that not only do the distributions align well but also the prior of the reference is changed to only keep 10 reference classes to match the prior of the target distribution providing a further improvement in the upper bound. This clearly shows the effectiveness of our proposed optimization algorithm in learning various transformations to minimize the upper bound.
C.1.2 Effectiveness of minimizing the upper bound in Theorem 3 via solving Eq. 4
In Fig. g] we show how the upper bound changes as the optimization progresses to transform the reference task (ImageNet) into four target tasks with the ResNet-18 model. Similar experiments in Sec. 4. of the paper we optimize over the transformation A while B and 7) are fixed to a random permutation matrix and the reference prior. After about 600 epochs the optimization problem converges to a local minima.
C.2 Additional results for the impact of the reference task on task-relatedness (Sec.
C.2.1 Additional results for image classification
Here we provide details of the experiment presented in Sec. 4.2 on task-relatedness and transferability. Similar to the results presented in Fig. of the main paper, the results in Fig. show that when the reference and the target tasks are related then task-relatedness is good as in the case when the reference task is MNIST and target is USPS vice versa achieving a smaller gap to transferability. When the target tasks are unrelated to the reference task data then both the transfcrability and task-relatedness are low. E.g., when the reference task is MNIST and the
about the effect of the reference task
23

---

Togget
Taget
Taye
Tear elermed Ratorenca
Tonakereed
101
gg
10,
Figure 7: (Best viewed in color.) t-SNE visualizations of the effect of various transformations the bound in Theorem when data from 20 randomly selected classes from ImageNet are used to transfer CIFAR-10. R-00. When all transformations are fixed (FixedAll, left) the distance between the distribution R" (transformed reference) and 7 is high explaining the large upper bound. Learning just the transformation A using the algorithm proposed in Sec. (3.4 significantly reduces the distance between R" and T leading to a tighter upper bound (center). Learning all the transformations further improves the matching (right). Especially, learning B and D change the class priors of the reference so that the same number of classes from the reference are used for matching as those present in the target. This evident from the right plot where only 10 unique reference task clusters are visible compared to 20 in the center plot, with fixed D. Moreover, the zoomed-in portion shows that for the center figure two classes from the reference task (green and 3) match with class (blue) of the target whereas single class from the reference task (green 18) matches class I (blue) of the target in the right figure.
Imagenet CIFAR-10 (ResNet-18) Imagenet CIFAR-100 (ResNiet:18) Imagenet
Imaganet Aircraft (ResNNtt-18)
xx]
Epochs
Figure 8: Reduction of the proposed upper bound is shown transformations are learned by solving the optimization problem in Eq. 3. After 600 epochs, the upper bound stabilizes showing the convergence of the optimization problem. Each subplot shows the effect of learning the transformation parameters for the transfer learning task with ImageNet as the reference task and ResNet-18 (trained in supervised way) for different target tasks. The solid line is the average after 5 random restarts and the shaded portion shows their standard deviation.
target is FMNIST or vice-versa. Task-relatedness is also correlated to transferability, i.e., model trained on MNIST achieves better transferability to USPS than FMNIST.
C.2.2 Results for NLP sentence classification task
In this section, we use sentence classification NLP task demonstrate further the effect of the reference task on task-relatedness, For this experiment, we first fine-tune the entire DostilRoBERTa
34 model distilled on OpenAl's WebText dataset, using a subsample of 10,000 points from the DBPedia dataset. We then use these fine-tuned models to evaluate the transferability to AG news, SST-5, and Yelp datasets. The results in Fig. [h]ww that transferability on AG News the smallest among the three datasets. This coincides with the task-relatedness value obtained after learning the
24

---

AN50
RN101
RM152
CLIP RN50
ADV(0.1) RM18
ADV(0.1) RM5O
ADV1) RMIS
ADV(3) RN50
CLIP Vit-832
Loca
SIMCLR
MOCO
Onea crant Alcran
Peb
craai cran 100 Aircaat
DYD
Crenise
pat
oPanLE cranto Ainzon
comta cpalios
Pet Dro craniee crasteo Aircort
cPARI cpetited
Pude oro on creae1 150 Aicroft
(a) Image classification tasks
DistilBERT
DistiRRoBERTa
AG News
437-3
AG Hews
55
(b) Sentence classification tasks
Full results for comparison transferability vs. task-relatedness for large pre-trained models on image and sentence classification tasks. Task-relatedness consistently achieves small gap to transfcrability. We denote cross coos--entropy loss on the y-axis. Plot-title denotes the pre-trained model and the x-axis denotes the target tasks.
Figure 9:
Target
Reweighted
DBPedia
Loss
2
USPS
FMNIST
USAS
MNIST
MNIST
FMNIST
AG News
VELPS
5573
Figure 10: Decomposition of task-relatedness into its three components illustrates that our task- relatedness (specifically the distribution mismatch term) explains the difference transferability. Similar tasks such USPS and MNIST have the highest transferability and also have the highest task-relatedness.
transformations which explains why transfer from DBPedia to AG News more successful compared to other target tasks. This observation is reasonable, especially considering that both DBPedia and AG News have structured information. Moreover, since DBPedia related to Wikipedia, the terms and entities appearing in AG News more related to those appearing in DBPedia in comparison to terms/entitics appearing in SST-5 and Yelp which consist of movie reviews and reviews collected from Yelp.
For our experiments, in this section, we follow a similar setting of fixing be random permutation matrix, C' to the prior of the reference task, and only learn the transformation A. We sample 10,000 points from DBPedia belonging to the same number classes as those present in the target task (e.g., for AG News we sample data from 4 randomly selected classes of DBPedia) and use this data as the reference task data train h with gradient norm penalty (r 0.02). All experiments are run for random seeds and average results are reported in Fig. 12
25

---

Bonken Inhe Bey
Accrma
Asturno
CIFAR10
CIFAR100
Pets
DTD
Aircraft
Figure 11: Task-relatedness is highly (negatively) correlated (Pearson correlation coefficient in the subplot title) with the accuracy of the models after end-to-end fine-tuning. Each subplot shows transfer learning with various target tasks for a specific model architecture and training method.
1.70
1.55
1.50
Task-relatedness
1.65
Reforance Task
Transferability
1.45
Refurence DBPadia
DBPedia
1.80
1.40
1.43
1.72
1.18
1.56
1.55
1.55
1.35
1.30
1.50
1.25
1.45
1120
Figure 12: Task-relatedness (Left) and transferability (Right) highty correlated across various reference-target pairs. A target task related to the reference task (DBPedia) such as AG News achieves better transferability (with DistillRoBERaa model) and task-relatedness compared to less-related tasks such as SST-5 and Yelp5.
C.3 Lipschitz constrained linear fine-tuning
C.3.1 Implementing softmax classification with Lipschitz loss
To use the bound Theorem. 3 it is required that the loss be Lipiscchitz continuous w.r.t. 2 in the input domain Z. To enforce this, while learning the weights of the softmax classifier (linear fine-tuning) for the reference task or the target, we add the gradient norm penalty used previous 4 and solve the following optimization problem
works 52
min 1
[(h(zz), + p m0 V2((h(z)) 732 Σ N
(0~ 10*)
where 2) - log E, wne is the cross-entropy loss.
=
C.3.2 Trade-off between empirical and predicted transferability
Constraining the Lipschitz coefficient of the classifier increases both the target and the reference task cross-entropy loss since the hypothesis set is being restricted. The smaller the is, the larger the loss becomes. On the other hand, the smaller 7 makes the distribution mismatch term in Theorem 3 also smaller. Since the bound the sum of the reference task loss and the distribution mismatch (and label mismatch), there a trade-off determined by the value of r. We illustrate the effect of the values of on the empirical and predicted transferability. As mentioned previously, we train both the classifier for the reference task h and the target br with additional penalty on the gradient norm make them -ipsshhhi In Fig. [3] we present results varying the value of r for the transfer to the Pets dataset with ImageNet the reference task. For this experiment, we selected 37 random classes from ImageNet and only learned the transform A by keeping B fixed to a random permutation and C' fixed the uniform prior over reference task classes. We observe that the performance of linear fine-tuning degrades we decrease the value of r but explainability through the bound improves since the distribution mismatch term (dependent on 7) decreases in the bound. However, making too small is not preferable since it leads to an increase in the first term of the bound (re-weighted reference task loss) increasing the bound overall. Moreover, also leads to a degradation in accuracy after linear fine-tuning, For our experiments, we use 7 0.02 since it doesn't 't decrease the accuracy of fine-tuning significantly and leads to small gap between empirical and predicted transferability.
26

---

Target Loss
Reweighted Reference
RN18
RN50
10
10
Loss
0.01
0.02
0.05
00
0.02
05
0
0.
Figure 13: Trade-off between the cross cross-entropy loss (y-axis) after linear fine-tuning and the upper bound Theorem 3 ja function of (x-axis), for ResNet18 and ResNet50 models pre-trained on ImageNet and linearly fine-tuned on the Pets dataset. Increasing the value of leads to decrease in the entropy loss after fine-tuning but increases in the proposed bound mainly due to the r Wa term
D Details of the experiments
All codes are written in Python using Tensorflow/Pytorch and were run on an Intel(R) Xcon(R) Platinum 8358 CPU with 200 GB of RAM and an Nvidia A10 GPU. Implementation and hyperpa- rameters are described below. Our codes can be found the supplementary material. We report an average of three independent runs for experiments in Sec. 4.2 and 4.3
D.1 Dataset details
In our work, we used the standard image classification benchmark datasets along with standard natural language processing datasets]
Aircraft [35): consists of 10,000 aircraft images belonging to 100 classes.
CIFAR- 10/100 [30]: These datasets contain 60,000 images belonging to 10/100 categories. Addi- tionally, created two subsets of CIFAR100 with the first 25 (small CIFAR-100-S) and 50 (medium CIFAR-100-M) classes.
DTD 13:
consists of 5,640 textural images belonging to 47 categories.
Fashion MNIST 600 consists of 70,000 grayscale images belonging to 10 categories.
Pets 43
consists of 7049 images Cats and Dogs spread across 37 categories.
ImageNet Net
17
consists of 1.1 million images belonging to 1000 categories.
Yelp 63 consists of 650,000 training and 50,000 test examples belonging to 5 classes.
Stanford Sentiment Treebank (SST-5) consists of 8,544 training and 2,210 samples belonging to 5 classes.
AG News 63
consists of 120,000 training and 7,600 test examples belonging 4 classes
DBPedia
x3
consists of 560,000 training and 70,000 test examples belonging to classes
D.2 Semantically similar classes for CIFAR-10 from ImageNet
For experiments with CIFAR-10 10 in Sec. C.I selected the following semantically similar classes from ImageNet, (airliner, minivan, cock, tabby cat, ox, chihuahua, bull- frog, sorrel, submarine, fire engine).
D.3 Additional experimental details
Details of the optimization problem in Eq. and Alg. [ In Step 5 of the algorithm, we use the network simplex flow algorithm from POT 23 to compute the optimal coupling. Since computing
the Wasserstein distance over the entire dataset can be slow, we follow [16 and compute the coupling
All NLP datasets and models obtained from http:::://ugggggfaae.ccoo
27

---

over batches. Note that the base distance defined in Eq. 2 is non non-differentiable. Thus, we use a differentiable approximation d((2, (2', == features =²) + |le(v) =(v')) lll (with 10*) where (z, y) and (z', v2) are samples from the domains R" and T and e(-) denotes the one-hot embedding of the labels in Step 5/6 The first three terms in Step 6 of our algorithm added to penalize
correspond the terms in the objective of Eq. 4 while the two additional terms
the constraints of class prior matching Pr(v) = BD and invertibility of the matrix A, respectively as required by Theorem 3
We use the softmax operation to ensure B and D are valid probability
matrix and vector.
In our experiments, in Sec. 4.1 we used pre-trained models available from Pytorch for ResNet 18550, along with publicly available pre-trained models provided in the official repositories of each training method. For each experiment, we subsample data from the ImageNet dataset belonging to the same number of classes as those present in the target dataset and use this data to train the linear layer on top of the representations extracted from the pre-trained model along with gradient norm penalty (Reference task classifier). To speed up the experiments, we use only 10,000 points from the subsample of ImageNet for training the linear classifier and computing the transfer. For evaluation, we use similar subsample of the validation dataset of ImageNet containing all the samples belonging the subsampled classes. Fine-tuning on this dataset takes about 0.05 seconds per epoch for the task of transfer from ImageNet to Pets with the ResNet-18 model (we run the fine-tuning for total of 5000 epochs).
Along with training the linear classifiers with a gradient norm penalty (with J = 0.02), we standardize the features extracted from the pre-trained models to remove their mean (along each axis) and make them have a unit standard deviation (along each axis). While standardizing the features do not have a significant impact the loss of the classifiers, including it makes it easier to match the distributions of the reference task and target data after transformations. Since our optimization problem transforms the reference task distribution to match the distribution of the target by solving the optimization problem in Eq. by working on mini-batches, the size of the batch must be greater than the dimension of the representation space of the pre-trained encoder. E.g., for ResNet18 models which have representation dimension of 512, we use a batch size of 1000 and for ResNet50 models which have a representation dimension of 2048, we use batch size of 2500. Having a smaller batch size than the dimension could lead to noisy gradient since for that batch the transformation can achieve a perfect matching, which may not generalize to data from other batches or unseen test data.
While computing the transformations, we apply the same augmentation (re-sizing and center crop- ping)/normalization to the training data as those applied the test data. Along with this, we extract the features of the training and test data from the pre-trained model once and use these to train the linear layer. We note that this is done to save the computation time and better results could be obtained by allowing for extracting features after data augmentation for every batch.
Finally, for our experiments in Sec. 4.2, the encoders are trained end-to-end the reference task. This is in contrast to our other experiments where the encoders are mre-trained and data from the reference task is only used for linear fine-tuning. Using these models, rask-relatedness is evaluated by fine-tuning a linear layer using the data from the target task as well as the transformations computed by solving Eq. 4 We used 0.2 here. We run the experiments with 3 random seeds and report the average results.
For our experiments in Sec. 4.3 we used the official code from 61 to compute the scores for NCE, Leep, and LogMe along with the official code of [51] for SFDA. For PACTran [19,, we also use their official code with the PACTran- PACTran-Gaussian method with N/K- 100, 3 = 10N, o² D/100 where N denotes the number of samples and K denotes the number of classes. This setting is similar to the PACTran-Grr Gaaann-Gassyya füx setting used in their work with the difference that we use N/K 100 to use a sufficiently large number of samples, especially considering that all our other results for SЫтЕ methods are computed on the full training set. For OTCE, we follow the official code and compute the recommended OTT-based NCE score and OTCE score (A1 = -0.0001 and X = -1) using 4000 randomly selected training samples from the two tasks. For the source task, we subsample the same number of classes as the target task and use. For the H-score, we use the official code to compute the H-score [6).
For estimating transferability by solving Eq. 5 we set >= for all the experiments in Sec.
28

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract summarizes the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Sec. and elaborated here.
Here we present some of the potential limitations of our work and discuss the avenues for future work. The analysis presented in the paper studies transferability using the popular cross-ents enssopy loss similar to that analyzed by [56]. While this is important and practically useful, extending the analysis to other losses, primarily to the 0-1 loss, is interesting future direction since in most classification tasks, accuracy is the metric of primary interest.
Another limitation of our work is the dependence of our bound on the Wasserstein distance. While it is a popular choice for analyzing performance in various distribution shift scenarios [52 16 50]. it is difficult to estimate in practice due to its sensitivity to the number of samples
and the dimension of the representation space. An analysis based on a different and easy-to- compute divergence measure might be more amenable for making transferability estimation via task-relatedness easier and faster.
We emphasize that even though we present some of the limitations our current work above, these are by means weaknesses of our work, which non-trivially extends the current under- standing of the transfer learning setting through a rigorous analysis and achieves impressive performance on practical applications. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to toxeeettt system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
29

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limita- tions that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in devel- oping norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In App.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: In App.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
30

---

(c) If the contribution is a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our codes can be found at http:////ggitubb..cccaahasa////e
TaskTraansferAnalyysis
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guiddes/CodeSuubiissiooo iooooiccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc/publice. http://pwbblccgaaaaeee//eeeeaaan
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and basclines. only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In App.D)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yss]
Justification:
An average of three independent runs reported.
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, the hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric emoor hars that would yield results that are out of range (e.g. negative error rates). If eтror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In App.D.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
There are no ethics concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA
Guidelines:
32

---

The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode. com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
33

---

For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "jAy6noFt6e", "reviewer": "Reviewer_3SCA", "summary": "The paper presents a way of assessing the transferability of representations from a pre-trained model to a target task by assessing the impact of the pre-trained model's representations on a reference task. By \"transforming\" the reference task into the target task, the paper produces a bound on the training loss of the target task", "questions": "1. Work focuses  on classification based tasks -- could you discuss how this applies to generative tasks\n2. Can you discuss how faithful the estimates / method is when K_r < K_t  ?\n3. The analysis is for training error -- how is transferability estimation affected if there is overfitting to the reference task ?", "limitations": "N/A -- paper presents some limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents both theoretical and empirical analysis of a problem\n2. Some of the provided experiments are reasonably comprehensive\n3. The idea of tackling the transferability problem by introducing a reference task seems novel AFAIK", "weaknesses": "My main issues with this paper are as follows. \n1.  **Utility of the method** :  I was initially excited by the following sentence in the introduction of the paper -- *\"Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis\"*. However,  it seems that Algorithm 1 requires access to the target labels $\\mathcal{Y}_{T}$. This calls into question this stated advantage over the SbTE methods. Specifically, SbTE methods like TransRate and LEEP do not require  training on a reference task which not only comes with extra overhead but whose selection requires a-priori meta-intuition about which tasks are similar to the target task. Also, none of the empirical evaluations presented actually show that the proposed method is superior to SbTE approaches for choosing models form a zoo. Specifically, the paper presents hard to interpret correlation measures in Table 1 as proof of transferability -- when a superior test would be compare deltas in final training performance of the models the models that each method select from a zoo.\n2. **Correctness / Impact of claims**:  \n  [a]. From line 238 - 241: *However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.*. This seems to undercut the utility of the analysis presented. Why does this work ? Why even discuss these terms if they are effectively fixed ?  \n  [b]. It feels like the insight of *Highly related reference–target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks.* is just punting the problem of transferability estimation to task similarity estimation (and now calling on practitioner intuition to decide which tasks are highly related). Thus it does not seem that the authors are actually effectively solving the problem of transferability estimation -- they are just transforming it into a different problem.  \n  [c].  The authors claim  \"tight bounds / small gaps between task related and transferability\" but this is based on (imo) just eyeballing the graphs in  Figure 3 and 9. We are not provided with a rigorous way to interpret the gap. Eg -- a very simple way to artificially present a small gap is to just plot the the losses at random init of the classification head for the the reference and target tasks that have the same number of class labels. \n3. **Strength of underlying technical assumptions**:    \n  [a].  the K_r > K_t seems like a strong assumption that limits the methods applicability. What if the most similar task to the target task has K_r < K_t.   \n  [b]. Also as mentioned above in 238 - 241 it seems the label and class-prior matching component of the algorithm effectively has no influence on the final algo results.\n\n\nScore updated after discussion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We respond to your concerns below:\n\n> Utility: A) I was ... methods. B) Specifically, ... target task.\n\nA: While Alg. 1 requires labels of the target task, we discuss in Lines 329-347 how to use Alg. 1 for the case when target labels are unavailable. The main idea is to use the transformation model to obtain the pseudo-labels (Line 337) of the target data in the label space of the **target** task. (Note that this is different from the pseudo-labels of the target data in the label space of the **source** task as used by LEEP [34]). Using these pseudo-labels of the target data as a proxy for $Y_T$, we can now use Alg. 1. Since no other SbTE method (to the best of our knowledge) can work without target labels, we stated this in the introduction of the paper. We will update the input of Alg. 1 appropriately to clarify this. \n\nB: We emphasize that the reference task does not need to be similar to the target task to use our methodology for estimating transferability. Any benchmark/publicly available dataset can serve as the reference dataset and its use is not a bottleneck for our analysis/approach. Instead, it makes it possible to study transferability provably, enables us to estimate transferability without target labels, and provides insights into improving the transferability of a model to downstream tasks. \n\n> None of ..\n\nWhile the best way to evaluate transferability is to fine-tune a model end-to-end on the target task, it is computationally prohibitive. E.g., You et al., 2021 [54] show that end-to-end fine-tuning of a ResNet-50 model on the Aircraft dataset, with hyperparameter search, requires about a day’s worth of computation (Sec 5.5 and Table 5 of You et al., 2021). This is the motivation for most  SbTE approaches that efficiently estimate transferability without end-to-end fine-tuning. \n\nPrevious works [5, 34, 25, 54, 49] reported the Pearson correlation coefficient between the accuracy of models after end-to-end fine-tuning and the proposed scores. A high correlation of an SbTE score indicates that comparing the scores of various models and choosing the one with the highest score is most likely the best-performing model on a given target task. Thus, we report correlation coefficients. (Since task-relatedness is based on loss, a higher negative correlation is desirable for us.)\n\n\n> [a]. From line 238 - 241..\n\nWe ablate the use of different transformations and measure the effect of learning each of them in Appendix C.1. Our experiments show that minimizing the conditional entropy by learning transformation B prefers a sparse mapping between the classes of the reference and the target tasks. As a result, one class from the target is mapped uniquely to one class from the reference task, and the priors of extra reference classes are reduced to zero (This is illustrated in Fig. 7 of Appendix C.1.1). Based on this, we use a reference task with the same number of classes as the target task and fix the matrix B to be a random permutation of the identity matrix. \n\nThus, while each term in the bound has its impact, fixing the matrix B and choosing a reference task with the same number of classes as the target, we simplified the optimization problem in Eq. 1 and made it efficient for practical usage. \n\n> [b]. It feels..\n\nThe main purpose of the experiments in Sec. 4.2 is to demonstrate the validity of our analysis and the fact when the target task is a transformation of the reference task, a model’s transferability to the target task can be provably explained based on that of the reference task. For e.g., for an encoder trained on MNIST, ground truth transferability to USPS is better than that to FMNIST (Fig. 4(a) right). This is also what is suggested by our task-relatedness analysis/metric using MNIST as the reference task (Fig. 4(a) left). Thus, as mentioned in Fig 1, given a pre-trained encoder and a reference task, we can provably explain the encoder’s transferability to those target tasks that are transformations of the reference task. \n\nFor the practical purposes of estimating transferability on the other hand, any reference task can be used (i.e., any publicly available or benchmark data) to compute task-relatedness. As shown in Sec 4.3, the use of a reference task to estimate transferability leads to a more stable estimate of transferability regardless of the number of target samples used as well as enables estimating transferability without access to target labels. Thus, while being theoretically sound, task-relatedness provides practical advantages over previous transferability estimation methods. \n\n> [c] The authors..\n\nA small gap in transferability implies that the difference between the left-hand side of the bound in Theorem 3 and the right-hand side computed by learning the transformations via Alg. 1 is small. While we agree that losses at random initialization for reference and target loss might show a small gap, it will not be very useful. Thus, we measure the left-hand side of the bound after training on the target task and then compare task-relatedness with this value. The target loss presented as the blue bars in Figs 3 and 9 show this value.  \n\n\n> The $K_r > K_t$ ... when $K_r < K_t$ ?\n\nPlease see our joint response.\n\n>  classification based ... generative tasks\n\nPlease see our joint response.\n\n> The analysis..\n\nThis seems to be a misunderstanding since our analysis is for expected loss. The classifiers $h_R$ and $h_T$ are learned on the training data and we report transferability and task-relatedness values on the test data from the two tasks similar to any ML pipeline. \n\nWhen there is overfitting on the reference task, we can expect the reweighted reference task loss (computed on test data) to be higher, which may lead to a larger gap between the actual and predicted transferability. We emphasize that this is not unique to our method as any analysis based on a source/reference task (e.g., [6,7]) will behave similarly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bATQRVBvyn", "reviewer": "Reviewer_E8p5", "summary": "This paper analyzes transfer learning from the perspective of task relatedness, a model for transforming from a reference task (pretraining task) to the target task is proposed, which consist of prior transform, label transform, and feature transform. \nThe task relatedness is then measured as the distribution mismatch of the transformed distribution and the target distribution.\nExperiments have shown that the proposed task relatedness tightly upperbounds the transferrability to a range of architectures and datasets.", "questions": "The appendix has answered most of my questions.\nMy only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence?\n\n\nMinor:\n\nA possible related work: Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective, ECCV 2022.", "limitations": "The paper has discussed its limitations", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The motivation and the derivation of the proposed task relatedness make senses to me.\n2. The empirical results have shown the advantage of the proposed task relatedness.", "weaknesses": "1. The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad that you found our work intuitive. We respond to your concerns below:\n\n\n> The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.\n\nThe datasets used in our work are the ones used by prior works. This was done to make the comparison with previously proposed methods for transferability estimation easier. Moreover, most model backbones used in our work require data to have dimensions 224x224, so we resized images from all datasets including CIFAR to match those dimensions. This shows that our method handles data with a larger resolution. \n\n> My only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence?\n\nSimilar to other problems in deep learning this is a non-convex minimization problem where SGD converges to a local minima. Fig 8, in Appendix C.1.2  shows how Alg. 1 minimizes the proposed upper bound by learning different transformations of the reference task (ImageNet) into four target tasks (CIFAR-10, CIFAR-100, Pets, and Aircraft) with the ResNet-18 model. After approximately 600 epochs (approximately 2 minutes of wall clock time on our hardware)  the Alg. 1 converges to local minima.\n\nTo demonstrate that Alg. 1 converges to a reasonable solution, we consider transfer learning using a 20-class subset of Imagenet as the reference task and CIFAR-10 as the target task. Results in Fig. 6 (bottom left) and Fig. 7 (in the Appendix) provide clear evidence of Alg. 1 finding the transformations that minimize the upper bound. \n\nSpecifically, without learning any transformations the upper bound is large (~ 3.22 in Fig. 6) and the reference and target classes do not match (leftmost plot in Fig. 7). However, learning all the transformations reduces the upper bound (3.06 in Fig. 6), suppresses the prior of 10 source classes to match the 10 classes in CIFAR-10 and reduces the Wasserstein distance between the transformed source and target classes (rightmost plot in Fig. 7) illustrating the working and convergence of the optimization problem to a good solution. \n\n\n> Compare with Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective\n\nWe thank the reviewers for pointing us to this work. We briefly compare our approach to it here and will include the discussion in the main paper. \n\nThe work demonstrated a tradeoff between a model's discriminability and transferability properties as training progresses and proposed a learning framework to alleviate this tradeoff. Our work on the other hand focuses on analyzing the transferability in terms of relatedness of the representations of the reference and target tasks after training is complete.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25Shjh0uxF", "reviewer": "Reviewer_dEJE", "summary": "This paper proposes an analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. It aims to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. Their analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. The experiments results demonstrate the effectiveness of the proposed method.", "questions": "Please see Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(i) This work analyzes transferability for classification tasks and provides the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting.\n\n(ii) This work proposes an optimization problem to compute task-relatedness using a small amount of target labels and shows that it can even predict performance after end-to-end fine-tuning without requiring target labels.\n\n(iii) This work conduct multiple experiments to demonstrate the effectiveness of the proposed method.", "weaknesses": "(i) This work mentioned that it is the first to analyze transferability for classification tasks under cross-domain cross-task settings. However, there are multiple works [1-5] that have focused on the transferability and generalization of learned representations. Meanwhile, transferability is mainly evaluated by utilizing the performance differences between different domains and tasks with distribution shifts. This is also the case in previous works, but this is not the case for the introduction of motivation in this paper.\\\n[1] Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes.\\\n[2] Otce: A transferability metric for cross-domain cross-task representations.\\\n[3] On the theory of transfer learning: The importance of task diversity.\\\n[4] An information-theoretic approach to transferability in task transfer learning.\\\n[5] Task relatedness-based generalization bounds for meta learning.\n\n(ii) (Minor) Although this work focuses more on theoretical exploration, empirical verification on real-world datasets is also necessary, such as ablation studies on the three proposed terms, trade-off experiments (performance vs. efficiency vs. memory footprint) due to the introduction of additional computations, and comparisons with baselines on transferability. \n\n(iii) (Minor) The formatting and layout of some formulas and tables need to be adjusted to make the reading clearer and more intuitive while some equations with writing errors, such as Eq. 3, L527-528, the proof of Corollary 1. At the same time, due to the large number of formulas and physical quantities, it may be better to construct a table to explain the symbols and definitions.\n\n(iv) (Minor) The track chosen by the paper is not that matched to its content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We respond to your concerns below:\n\n> Compare with [1,2,3,4,5]\n\nWe thank the reviewers for pointing us to additional related works. We briefly compare our approach to them here and will include the discussion in the main paper.\n\n[1,3,5]: study the problem of few-shot learning (FSL) where a model is trained on data from related training tasks and is adapted to an unseen task using only a few samples. Different from these works, we focus on the transfer learning (TL) setting where a pre-trained encoder trained on some pre-training dataset is adapted with enough samples/gradient steps to a downstream task. This downstream target task may or may not have any relation to the pre-training task unlike [1,3,5].\n\nConcretely, [1] proposed a model-agnostic metric called Task Attribute Distance to gauge the success of learning in the FSL setting. Our work, on the other hand, defines task-relatedness based on the similarity of the representations of reference and target tasks in the representations of the pre-trained model (and is model dependent) rather than relying on the attribute information, which may not be available in the TL setting. \n\n[2] analyzes the sample complexity for learning a model shared across tasks and adapting it to a new target task and shows task diversity to be a crucial component for the success of FSL. Our work on the other hand does not assume access to shared tasks or restrict the number of samples required for fine-tuning on the target task. Moreover, their notion of task diversity requires access to a set of training tasks that may not be available in the TL setting, making our notion of task-relatedness more practical for TL.\n\n[3] proposes a notion of task-relatedness for the FSL setting, allowing to utilize all the data from available training tasks to help learn a model on a new task with a few gradient steps. This notion is model-agnostic and defined over the sample space ($X \\times Y$) unlike our measure which is defined in the representation space of the model whose transferability needs to be evaluated.\n\nThus while task-relatedness is at the core of both TL and FSL, the works [1,3,5] proposed notions relevant to the FSL setting whereas our work proposed a notion relevant to the TL setting. \n\nPlease see the joint response empirical comparison against [2,4].\n\n> Although this work ... baselines on transferability.\n\n**Ablation studies:** In Appendix C.1, we show the effect of using different transformations on the value of task-relatedness computed by solving Eq. 3. We present the results in three settings, the first where none of the transformations are learned, the second where only the feature transformation (parameterized by A) is learned, and the third where all the transformations are learned. The experimental results show that task-relatedness achieves the smallest gap to transferability when all transformations are learned, closely followed by learning only the feature transformation, and significantly better than not learning any transformation. These three settings capture the effect of the three different transformations proposed in our transformation model.\n\n**Trade-off experiments:** In Appendix C.1.2, we evaluate the number of epochs required by Alg 1 to minimize the proposed upper bound (compute task-relatedness) for four target tasks using the ResNet-18 model and Imagenet as the reference task. Our results in Fig 8 show that after approximately 600 epochs (approximately 2 minutes of wall clock time on our hardware) Alg. 1 learns the transformations that lead to a good estimate of transferability. \n\nFor the problem of end-to-end transferability estimation (Sec 4.3) for different models, we find that the computation of task-relatedness is significantly faster than the computation of actual end-to-end fine-tuning. Specifically, the computation of task-relatedness requires roughly only 3-4 minutes for computation, compared to end-to-end fine-tuning which can require computation of several hours. E.g., You et al., 2021 [54] show that end-to-end fine-tuning of a ResNet-50 model on the Aircraft dataset, with hyperparameter search, requires about a day’s worth of computation to achieve the best accuracy of 86.6% (see Sec 5.5 and Table 5 of You et al., 2021). In comparison, task-relatedness can be estimated very efficiently. Compared to other SbTE approaches such as SFDA, our approach increases the computational time by ~30 seconds but gets consistently better results as shown in Table 1 and Fig 5 of the paper and the pdf in the joint response. \n\n**Comparisons with baselines on transferability:** In Fig 3 and 4(b), we compare the estimate of transferability computed via Alg. 1 to the ground truth value of transferability (denoted by the blue bar labeled target loss). Our results show that task-relatedness incurs a small error compared to actual transferability. \n\nIn Table 1, we present the result of comparing the correlation of task-relatedness and accuracy after end-to-end fine-tuning and compare it to the correlation of five popular score-based transferability estimation methods. Along with these we also compare the effect of the number of target labels available on the correlation of each of these methods with the accuracy after end-to-end fine-tuning.\n\nThus, we believe that we have already provided a detailed empirical evaluation of our methodology and compared our approach with popular transferability methods. However, if the reviewer would like to see any specific experiments to be added, we would be happy to provide them as well. \n\n> The formatting ... definitions.\n\nWe thank the reviewer for this excellent suggestion. We will create a table to clarify all the symbols and their meaning. We will expand and adjust the tables and equations to be more readable in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1eoaQNU5U", "reviewer": "Reviewer_sD5Z", "summary": "This paper considers transfer learning in a general cross-domain, cross-task setting. It introduces the concept of \"task-relatedness\" into measuring whether transfer learning can be successful. Specifically, task-relatedness consists of (1) a reference loss term for measuring the difference in class prior distribution between the target and source tasks; (2) a label mismatch term for measuring the conditional entropy between the label distributions; and (3) a distribution mismatch term  for measuring the difference in feature distributions. To make computing task-relatedness practical, the paper proposes several approximation schemes. It shows the metric is highly correlated with model transferrability in practice and can be used for selecting pretrained models.", "questions": "1. Class-prior transformation assumes the reference task has more classes than the target task. Is this too strong an assumption? Does the proposed method applies to the settings where the target task has more labels?\n\n2. In Figure 3, the rankings of transferability for different target tasks are the same across all models. This is perhaps because the same reference dataset is used, and the experiments do not show the effect of varying model backbones. It would be better if the paper can evaluate multiple reference datasets on multiple models. (Figure 4 does look at different reference datasets but the model is fixed here.)\n\n3. Does the proposed metric applyy to non-classification tasks like regression or auto-regressive prediction?\n \n4. What's the max cross-domain distance the theory holds? I’d imagine that when the domain gap is large, the theory breaks. Have you studied that?", "limitations": "See weaknesses.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practical significance: The topic studied is important to the field of machine learning, as fine-tuning large-scale pretrained models gradually becomes the to-go option for model development. The idea of decomposing transferability into prior, feature, and label distribution differences is clean and interesting. \n2. Complete pipeline: the paper not only proposes task-relatedness as a transferability metric but also shows how it can be transformed into a training objective for effective transfer learning. This makes the paper more comprehensive and the proposed method more valuable.\n3. Presentation is clear and code for reproducing the experiments is provided", "weaknesses": "1. Limited novelty: the idea of using distribution distances to measure transferability is not new. As the related work section has pointed out, using the Wasserstein distance to estimate domain difference has been studied by OTDD [1]. Works like OTCE [2] also uses the proposed metric to do pretrained model selection. One argument the paper makes is that the proposed task-relatedness works without requiring target labels. While this does add some practical significance, the proposed method is a simple pseudo-labeling scheme without theoretical justification.\n2. Limited evaluation: the baselines considered in the experiments are not comprehensive, e.g., OTCE is not evaluated, and other recent OT-based works like [3] are not discussed. There's also no comparison for different pseudo-labeling schemes for the missing-label setting. Besides, there's no ablation study showing the contribution of each of the three terms proposed for computing task-relatedness.\n\n[1] Geometric Dataset Distances via Optimal Transport. David Alvarez-Melis. Nicolò Fusi.\n\n[2] OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations. Yang Tan, Yang Li, Shao-Lun Huang.\n\n[3] Wasserstein Task Embedding for Measuring Task Similarities. Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, Soheil Kolouri.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are delighted that you found our analysis clear and method practically valuable. We respond to your concerns below:\n\n> 1. Limited novelty: (A) the idea ... is not new. (B) the proposed method ... justification.\n\nA: While we agree with the reviewer that distribution distance is effective at gauging transferability, prior works such as OTDD and OTCE offer limited theoretical justification to explain a model’s transferability. As mentioned by the reviewer in their paper’s summary, our analysis shows that there are two additional terms (namely, reweighted reference loss and conditional entropy of label sets) along with distribution distance to provably explain transferability. Moreover, unlike OTDD and OTCE which rely on the distance between the distributions of the reference and the target tasks, in the input and representation space, respectively, our analysis shows that the distance between the **transformed** distribution of the reference task in the representation space (i.e. P_{R’’’} obtained after using the transformation A) and the distribution of the target task is what explains transferability provably (Theorem 3). \n\nB: We agree with the reviewer that task-relatedness can estimate transferability without requiring target labels easily and we emphasize that this ability is unique to our approach. This is because, to the best of our knowledge, no other transferability estimation method can produce pseudo-labels of the target data in the label space of the **target** task (Methods such as LEEP [34] rely on pseudo-labels of the target data in the label space of the **source** task). Using these labels for the target task, we can compute task-relatedness as per Def. 2 and still estimate the transferability of the models (Tabel 2). This makes task-relatedness the only method that can gauge the transferability of models without access to target labels. \n\nThus, compared to prior works, our paper advances both our theoretical and practical understanding of transfer learning in a meaningful way.\n \n\n> 2. Limited evaluation: (A) OTCE is ... not discussed. (B) no comparison ... pseudo-labeling schemes (C) no ablation study ... task-relatedness.\n\nA: Please see our joint response for comparison with OTCE.  \n\nWe thank the reviewer for pointing us to [a] and we briefly compare our approach to them here. We will include this discussion in the main paper.\n\n[a] proposes a model-agnostic approach that relies on optimal transport to compute the distance between the tasks similar to OTDD [3]. However, our work specifically focuses on understanding the transferability of different models both theoretically and practically and hence proposes a model-dependent metric for it. \n\nB: As mentioned above, no previous score-based transferability estimation methods can work without access to labels of the target data. Methods such as LEEP [34], use pseudo-labels of the target data from the source classifier but still require labels of the target data to compute their transferability score. Thus, to the best of our knowledge, task-relatedness is the only approach that can work when no target labels are available. Due to this, we show its effectiveness by comparing the correlation of task-relatedness with and without target labels to actual transferability in Table 2. Our results show that even without target labels, task-relatedness achieves a good correlation with transferability when reference and target tasks are related. \n\nC: In Appendix C.1, we study the effect of different transformations on task-relatedness. We present three settings, the first where no transformations are learned, the second where feature transformation is learned, and the third where all the transformations are learned. Our results show that task-relatedness achieves the smallest gap to transferability when all transformations are learned, closely followed by learning only the feature transformation, and significantly better than not learning any transformation. These three settings capture the effect of the three different transformations proposed in our transformation model. We also break down task-relatedness into its three corresponding terms in Figs. 3 or 4b to highlight the contribution of each term.\n\n> 1. Class-prior ... assumption? \n\nPlease see our joint response.\n\n> 2. In Figure 3, ... fixed here.)\n\nThe full results of Fig 3 presented in Fig 9 (in the Appendix) already include models trained with different model backbones such as ResNet-18, ResNet-50, ResNet-101, and ResNet-152 trained with a variety of training algorithms. Along with these results, Fig 4(a) shows the effect of training the same model backbone on different reference datasets and evaluates transferability to a variety of target tasks. Fig 4(b), evaluates transferability to various target tasks using the original pre-trained CLIP encoder and CLIP encoder fine-tuned end-to-end on different reference tasks. Thus, our experiments already evaluate the effect of different model backbones and reference datasets on transferability and task-relatedness. \n\n> 3. Application ... auto-regressive prediction?\n\nPlease see our joint response.\n\n> 4. What's the ... holds?\n\nSimilar to any other upper bounds that analyze the problem of distribution shift (e.g., [6]), a large upper bound implies that theory might not be able to explain the results in practice. This is true for task-relatedness as well since a large upper bound does not necessarily guarantee poor transferability. Using various datasets/tasks (in Fig 3 and 9) we show that task-relatedness estimates transferability closely. Moreover, under the conditions mentioned in Lines 221-225, our bound is indeed tight.\n\n**New References** \n\n[a]: Wasserstein Task Embedding for Measuring Task Similarities", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable feedback and insightful questions. We are encouraged that all the reviewers found the main contribution of the paper of rigorously analyzing transferability to target tasks in a cross-domain cross-task transfer learning setting using task-relatedness novel (3SCA), intuitive (E8p5), practically valuable (sD5Z); and our empirical evaluation comprehensive (3SCA, sD5Z).

We are glad that reviewers E8p5 and dEjE appreciated the significance and contributions of our work and recommended acceptance. While 3SCA and sD5Z recommended reject and borderline reject, respectively, the weaknesses pointed out by them are primarily clarification questions about the paper’s setting and utility, which we have addressed comprehensively in this rebuttal. Please see our comments below for answers to each reviewer's specific questions. Along with these, we have included additional results comparing our method with OTCE and Hscore [a] for the end-to-end transferability estimation problem considered in Sec 4.3.

Based on these answers and clarifications we have addressed the concerns of each of the reviewers. Hence, we hope reviewers will consider our answers, increase their ratings, and recommend acceptance. If additional questions arise during the discussion phase, we are just a post away and would be happy to address them.

> **[dEJE, sD5Z]: Additional results for OTCE and Hscore**

OTCE [49] and [a] propose score-based transferability metrics for the problem of end-to-end transferability estimation similar to those discussed in Sec. 4.3. Specifically, as discussed in Sec. 2, OTCE proposes a linear combination of negative conditional entropy between the label sets of the source and target tasks computed using OT-based coupling matrix and the Wasserstein distance between the representations of the two tasks. [a] on the other hand, solves the HGR maximum correlation problem to analyze transfer learning in the setting when the source and target domain have the same features, similar to [50]. They propose to use normalized Hscore as a way to estimate transferability. We have expanded our empirical section to include these baselines in Sec 4.3. We emphasize that both these methods are specifically proposed for score-based transferability estimation and do not provide a rigorous analysis of transferability, unlike our approach.

The comparisons to these were omitted due to the requirement of auxiliary tasks required to obtain hyperparameters crucial for OTCE’s performance as mentioned by the authors and some considered baselines such as LEEP and LogMe had already shown improvements over Hscore. However, we have expanded our empirical results to include these methods. 

For OTCE, we follow the official code and compute the recommended OT-based NCE score and OTCE score ($\lambda_1=-0.0001$ and $\lambda_2=-1$) using 4000 randomly selected training samples from the two tasks. For the source task, we subsample data from the same number of classes as the target task. For the H-score, we use the official code to compute the metric. The results for these two new metrics are presented in the attached pdf corresponding to Table 1 of the paper which evaluates the Pearson correlation of the metrics to the end-to-end fine-tuning accuracy of five models and Fig 5 of the paper which evaluates the sensitivity of the metrics to different numbers of target samples. 

Consistent with the results in Table 1 and Fig 5 of the paper, task-relatedness achieves a high correlation to end-to-end fine-tuning accuracy better than most previous SbTE methods. This correlation also remains stable regardless of the number of target samples, unlike most other SbTE methods whose correlation degrades severely in scenarios with fewer samples from the target task.

> **[3SCA, sD5Z]: Does the proposed method apply to the settings where the target task has more classes than the reference task?**

Yes, since this assumption does not affect the analysis. We assumed it since we focus on explaining transferability with the performance/classifier of the reference task. Thus, scenarios where the reference task has fewer classes than the target would intuitively lead to a poor understanding of transferability. While task-relatedness can be computed in this setting, the presence of fewer classes in the reference task can increase the conditional entropy and the distribution mismatch terms. This is because the mass of the extra target classes will be split across multiple reference task classes leading to a higher distribution mismatch. The conditional entropy will be higher because a sparse mapping (such as one-to-one) between reference and target task classes cannot be obtained. 

Practically, since a practitioner can choose any reference task to evaluate task-relatedness, it's preferable to use one with more classes to get a better estimate of transferability.

> **[3SCA, sD5Z]: Does the proposed metric apply to non-classification tasks?**

The popularity of transfer learning in the classification setting is the primary reason we analyze this setting. The transformation model used in our work includes components such as labels or prior changes, which are most suitable for classification tasks. While the distribution mismatch component could be useful beyond classification tasks, it is unclear as to what form of distribution divergence is most suitable. Developing an appropriate transformation model for auto-regressive tasks and the performance metrics associated with them requires further research. Thus, we believe that it is an excellent suggestion by the reviewers to extend the analysis to other non-classification tasks, however, it is non-trivial and currently out of the scope of this paper.

**New References** 

[a]: An information-theoretic approach to transferability in task transfer learning.

Meta-review: The paper addresses task-relatedness in cross-domain cross-task transfer learning. The main contribution is an upper bound on transferability of the representations of pre-trained models to downstream tasks measured in terms of their relatedness to a given reference task. The proposed task-relatedness works without requiring target labels, and the notion of a reference task has been well received.
 
Initial reviews raised several issues, including clarifications of the contributions, limited empirical evidence, and related work comparisons. The rebuttal successfully addressed most of these concerns, leading to two reviewers raising their scores. After careful deliberation, the decision was made to accept this paper - congratulations to the authors! It is crucial, however, that all the promised improvements and clarifications including the related work, sensitivity analysis and presentation clarity, are addressed in the final version of the paper.
Decision: Accept (poster)