Title: Understanding the Transferability of Representations via Task-Relatedness
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transfer Learning Analysis

Distribution Shift
Abstract: The growing popularity of transfer learning due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in a cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks.Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels of the target task and its high correlation with the model's accuracy after end-to-end fine-tuning on the target task makes it a useful metric for transferability estimation. Our empirical results of using task-relatedness on the problem of selecting the best pre-trained model from a model zoo for a target task highlight its utility for practical problems.

Understanding the Transferability of Representations via Task-Relatedness

Abstract

The growing popularity of transfer learning, due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in a cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels of the target task and its high correlation with the model's accuracy after end-to-end fine-tuning on the target task makes it a useful metric for transferability estimation. Our empirical results of using task-relatedness on the problem of selecting the best pre-trained model from a model zoo for a target task highlight its utility for practical problems.

1 Introduction

Transfer learning (TL) [42, 59] is a powerful tool for developing high-performance machine learning models, especially in current times when large models [45, 11, 12, 18] pre-trained on huge amounts of data are being fine-tuned for various downstream tasks. While large pre-trained models achieve impressive performance on downstream tasks even in the zero-shot inference setting [45], their performance can often be improved by fine-tuning them on data from target tasks. However, our understanding of when representations from these models lead to classifiers that achieve high performance (i.e., high transferability) to downstream tasks is still lacking.

Analytical works based on domain adaptation [8, 7, 52, 36, 32, 37, 38] can only explain cross-domain tasks (i.e., when only features/label priors change across tasks) but in the TL setting, label sets can also change (i.e., cross-task setting). Recently, [56] showed that the relatedness between the label sets of the two tasks measured using conditional entropy can explain the difference in their transferability. However, [56] focused only on the cross-task setting, and analysis for transferability in a general cross-domain cross-task setting is not addressed. Apart from these analytical works, another line of work focuses on proposing transferability metrics that correlate well with performance on downstream tasks after end-to-end fine-tuning. We refer to these works as score-based transferability estimation (SbTE) metrics [61, 55, 29, 40, 19, 51]. These works focus on developing scores for selecting a pre-trained model from a model zoo, that achieves the best transferability on a target task. While these works address a practical problem, they do not focus on providing an analysis of transferability.

Thus, we first rigorously analyze the transferability of the representations in producing high-performing classifiers and propose a novel approach that studies transferability in terms of its relatedness to a reference task (see Fig. 1). This is in line with previous analytical works [7, 2, 56, 55] which studied the model's performance on target tasks in terms of the source task in different settings such as domain adaptation/generalization and recently TL. However, there's a crucial difference: we study transferability in terms of a reference task instead of the source task since it is impractical to assume the knowledge of the source task used to train large models such as CLIP [45] or GPT, commonly used for TL.

Our approach works by transforming the distribution (and classifier) of a reference task, by transforming its class-prior distribution, label set, and feature space to obtain a new distribution that is similar to that of the target task (Fig. 2). Based on these transformations, we show that transferability can be provably explained (and is tightly upper bounded) using three

Aircrafts

Textures

Digits

Pets

Chest X-rays

e.g., ImageNet

e.g., Pets

Task-relatedness

Fixed

Pre-trained

Encoder

e.g. CLIP

Transferability

High

?

Trainable

Flowers

Suite of Downstream Tasks

interpretable terms. A weighted reference loss term appearing due to the class prior distribution difference between the tasks, a label mismatch term appearing as conditional entropy between the label distributions of the tasks, and a distribution mismatch term appearing as the Wasserstein distance between the transformed reference and target distributions (Theorem 3). We define task-relatedness as the sum of these three terms (a smaller value implies higher relatedness). We propose an optimization problem (Eq. 4) and an algorithm (Alg. 1) to learn the transformations to compute it. Using state-of-the-art (SOTA) pre-trained models, with different architectures, trained with various training methods on computer vision (CV) and natural language processing (NLP) tasks, we show that taskrelatedness achieves a small gap to transferability (Sec. 4.1). Our analysis also leads to new insights into learning in the TL setting such as to improve the transferability of an encoder on a downstream task, one can improve the encoder's transferability on related reference tasks (Sec. 4.2). This is particularly useful when practitioners intend to develop encoders that achieve high transferability to proprietary (and potentially inaccessible) datasets.

We also demonstrate the utility of task-relatedness in estimating the accuracy of the model after end-to-end fine-tuning. While the TL setting assumes access to target labels, the high computational cost of end-to-end fine-tuning of a pre-trained model on a target task calls for developing metrics that are efficiently computable and highly correlated with end-to-end fine-tuning accuracy. To this end, we propose to use task-relatedness computed in the penultimate layer of the pre-trained model as our transferability estimation metric. To further improve the computational efficiency of task-relatedness we only measure the difference between the class-wise means and covariances of the distributions in lieu of the Wasserstein distance as required in Theorem 3. This enables the computation of task-relatedness with only the statistics of the reference/target tasks. Our empirical results (Sec. 4.3) attest that task-relatedness achieves a high correlation with the model's accuracy after end-to-end fine-tuning on the target task making it an effective metric for selecting a pre-trained model from a model zoo that achieves the best accuracy on the target task. Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis. Our main contributions are:

We rigorously analyze transferability for classification tasks. Our analysis, to the best of our knowledge, leads to the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting.

We propose an optimization problem to efficiently compute task-relatedness, using a small amount of target labels and show that it can even predict performance after end-to-end finetuning without requiring target labels.

Using SOTA models and CV/NLP tasks, we show that task-relatedness accurately predicts transferability and show that transferability to unseen tasks can be improved by improving transferability to known (related) tasks.

2 Related Work

Transfer learning (TL): TL [42, 59, 18, 49, 46, 15, 14] has been studied widely and consists of various settings including transductive transfer, inductive transfer, and task transfer learning. The transductive setting also referred to as domain adaptation [8, 7] focuses on reducing the shift between two domains. The task transfer setting focuses on identifying the relationship between tasks, regardless of the model, to explain the transfer performance (see Appendix B for more details). Lastly, the inductive transfer setting focuses on using an inductive bias such as fine-tuning a pre-trained model (trained via adversarial training [48], self-supervised learning [11, 10, 12] or by combining language and image information [45]) to improve the performance on a target task. Our work focuses on the inductive transfer learning setting and proposes an upper bound on transferability of the representations of pre-trained models to downstream tasks.

Analytical works for learning under distribution shifts: Prior works [8, 7, 52, 36, 32, 37, 38] analytically explained learning under distribution shifts using distributional divergence between the marginal distributions and a label mismatch term. However, these results are applicable under assumptions such as covariate or label shift which need not be satisfied in TL where both the data distribution and the label spaces can be different (see App. B for detailed comparison). Recently, [56] proposed an upper bound on transferability in a restrictive setting of the same features for both tasks, however, our analysis does not require such an assumption. Other works [9, 47, 41] analyzed the representation for the multi-task learning setting. These works showed that when tasks are weakly related, a single representation space (model) may not perform well for all tasks. However, the TL setting differs from both of these and our work aims to analyze transferability in this setting.

Score-based transferability estimation (SbTE): These works [5, 40, 29, 61, 55, 39] use data from the target task and produce a score correlated with transferability. Such a score is useful for selecting the model from a model zoo that leads to the best transferability to a target task. [56] proposed the Negative Conditional Entropy (NCE) score that predicts transferability using the negative conditional entropy between labels of the tasks but requires the two tasks to have the same input instances. [6] estimates transferability by solving the HGR maximum correlation problem and using normalized Hscore, in the same setting as [56]. [40] proposed the LEEP score and computed NCE using soft (pseudo) labels for the target task from a pre-trained model. OT-CE [55] combined Wasserstein distance [3] and NCE whereas [5, 61] estimate likelihood and the marginalized likelihood of labeled target examples to estimate transferability. [33] proposes a model-agnostic approach that also relies on optimal transport to compute the distance between the tasks similar to OTDD [3]. In contrast, we focus on analyzing transferability in terms of task-relatedness theoretically along with demonstrating its effectiveness as a transferability estimation metric for the pre-trained model selection problem.

3 Analysis of TL using task-relatedness

Problem setting and notations: Let P R ( x, y ) and P T ( x, y ) denote the distributions of the reference and the target tasks, defined on X R ×Y R and X T ×Y T respectively. We assume that the feature spaces are common ( X R = X T = X ) such as RGB images, but the reference label set Y R = { 1 , 2 , · · · , K R } and the target label set Y T = { 1 , 2 , · · · , K T } can be entirely different. We assume the number of reference task classes ( K R ) are greater than or equal to the number of target classes ( K T ). In the TL setting, an encoder (feature extractor) g : X → Z is pre-trained on a dataset with or without labels depending on the training method (e.g., supervised vs. self-supervised). We denote the resultant push-forward distributions of R and T on the encoder output space as P R ( z, y ) and P T ( z, y ) . With a fixed encoder g , a classifier (linear or non-linear), h ( z ) : Z → ∆ , that outputs a probability vector is learned for the reference ( h R ) and the target ( h T ) separately, where ∆ R/T is a K R /K T simplex for R/T . The classifier h R = arg min h ∈H E ( z,y ) ∈ P R [ ℓ ( h ( z ; g ) , y )] and h T = arg min h ∈H E ( z,y ) ∈ P T [ ℓ ( h ( z ; g ) , y )] where H is the set of classifiers and ℓ ( h ( z ) , y ) = -log( h ( z ) y ) is the cross-entropy loss. Table 3 in App. A summarizes the notations used in our work. Next, we define transferability as commonly used in the literature.

Bee

Lion

Cat

Dog

Cat

Dog

Prior

Transform

C

Label

Transform

B

Feature

Transform

A

𝒉

𝑹

(𝒛)

𝒉

𝑹

!

𝒛

𝑷

𝑹

(𝒛, 𝒚)

Cat

Dog

Distribution

mismatch

Reference

𝑹

(e.g., Imagenet)

Target

𝑻

(e.g., CIFAR-10)

𝑷

𝑹

!

(𝒛, 𝒚)

𝑷

𝑹

!!

(𝒛, 𝒚)

𝑷

𝑹

!!!

(𝒛, 𝒚)

𝑷

𝑻

(𝒛, 𝒚)

𝒉

𝑹

!!

𝒛

𝒉

𝑹

!!!

𝒛

𝒉

𝑻

𝒛

𝑹

!

𝑹

!!

𝑹

!!!

Lion

Definition 1. (Transferability). Transferability of the representations from an encoder g on a target task T for classifiers in H is defined as E ( z,y ) ∈ P T [ ℓ ( h T ( z ; g ) , y )] .

In the next section, we show the analysis with H as the class of linear classifiers for ease of explanation and discuss its extension to non-linear classifiers in App. A.5. Proofs for Sec. 3 are in App. A.

3.1 Our task transformation model

The reference and the target tasks share the same encoder but do not share label sets or data distributions. Therefore, to relate the two tasks, we propose a chain of three simple transformations: 1) prior transformation (from R to R ′ ), 2) label transformation (from R ′ to R ′′ ), and 3) feature transformation (from R ′′ to R ′′′ ). The R ′ , R ′′ , R ′′′ are intermediate domain names after each of the transformations are applied. The corresponding classifier in each domain is denoted by h R ′ , h R ′′ , and h R ′′′ as illustrated in Fig. 2. The distribution after the transformations ( P R ′′′ ) has the same feature Z R ′′′ = Z T = Z and label sets Y R ′′′ = Y T as the target task T , and consequently, the loss of the transformed classifier h R ′′′ and the target classifier h T can be related.

Class-prior transformation ( R → R ′ ) : Since the reference task has more classes than the target task ( K R ≥ K T ), many of the reference task classes are likely irrelevant for transfer to the target classes, e.g., while transferring from ImageNet to CIFAR10, only a small portion of ImageNet classes are relevant to CIFAR10 classes. The prior transformation accounts for the relative importance of the reference classes. This is illustrated in Fig. 2, where changing the class prior of R reduces the prior of the Bee class and increases the priors of Wolf and Lion classes (shown by the changed size of classes Wolf and Lion in R ′ ). While transforming the prior of R , we keep the conditional distribution and the classifier the same i.e., P R ′ ( z | y ) = P R ( z | y ) and h R ′ ( z ) = h R ( z ) . Lemma 1 in App. A.2.1 shows that the expected loss of the classifier h R on R ′ is a re-weighted version of the loss of h R on R .

Label transformation ( R ′ → R ′′ ) : Next, we use a label transformation to match the label sets of the new domain R ′′ and that of the target domain. To this end, we specify the conditional distribution B ij := P ( y R ′′ = i | y R ′ = j ) ( B ij ∈ [0 , 1] , ∀ i, j, ∑ i B ij = 1 , ∀ j ). The label y R ′′ of an example from the domain R ′′ is obtained via BP ( y R ′ ) . This generative process doesn't require the feature, i.e., P R ′′ ( y R ′′ | y R ′ , z ) = P R ′′ ( y R ′′ | y R ′ ) . B with sparse entries (i.e., only one entry of a column is 1) models a deterministic map from Y R to Y T ; B with dense entries models a weaker association. This process is illustrated in Fig. 2 which shows the map from { Bee, Wolf, Lion } ⊂ Y R ′ to { Dog, Cat } ⊂ Y T after using B . Under this model, a reasonable choice of classifier for R ′′ is h R ′′ ( z ) = Bh R ′ ( z ) . Lemma 2 in App. A.2.2 shows that the expected loss of h R ′′ depends on the loss of h R ′ and the conditional entropy between the label sets of the tasks R ′ and R ′′ and Corollary 1 shows the conditions for optimality of h R ′′ .

Feature transformation ( R ′′ → R ′′′ ): The final step involves changing the feature space of the distribution R ′′ . We apply an invertible linear transformation A to the distribution in R ′′ to obtain the new distribution R ′′′ . After the transformation, the classifier associated with the new domain R ′′′ is h R ′′′ ( z ) = h R ′′ ( A -1 ( z )) . This is illustrated in Fig. 2 after feature transform using A . Lemma 3 in

App. A.2.3 shows that a linear transform of the space and classifier does not incur any additional loss and Corollary 2 shows that the optimality of h R ′′ implies optimality of h R ′′′ . Using these, we get Theorem 1 by defining conditional entropy as follows

Theorem 1. Let C := [ P R ′ ( y ) P R ( y ) ] K R y =1 be a vector of probability ratios , B be a K T × K R matrix with B ij = P ( y R ′′ = i | y R ′ = j ) , A : Z → Z be an invertible linear map of features. Let the classifiers h R ′ ( z ) := h R ( z ) , h R ′′ ( z ) := Bh R ′ ( z ) , h R ′′′ ( z ) := h R ′′ ( A -1 ( z )) . Assuming ℓ is the cross-entropy loss, we have

Theorem 1 provides an upper bound on the loss of the final transformed classifier/distribution in terms of the loss of the reference classifier/distribution. The re-weighted reference loss shows that the performance of the transformed classifier on the new domain is linked to the label-wise re-weighted loss of the reference classifier on R . This implies that one can use only the relevant reference classes to contribute to the bound. The label mismatch term shows that the performance of the distribution R ′′′ and R depends on the conditional entropy H ( Y R ′′ |Y R ′ ; B ) between the label distributions of the domain R ′′ and R ′ . A high value of H implies that the labels of the reference task are unrelated leading to lower transferability, whereas a low H implies higher transferability. Corollary 3 in App. A.2.4 shows when the bound in Theorem 1 becomes equality.

3.2 Distribution mismatch between P R ′′′ and P T

After the three transformations, the transformed reference P R ′′′ ( z, y ) can be compared with the target P T ( z, y ) . However, these are only simple transformations and P R ′′′ cannot be made identical to P T in general. This mismatch can be measured by the Wasserstein or Optimal Transport distance [44, 58]. Since our goal is to match two joint distributions defined on Z × Y we use

̸

with z, z ′ ∈ Z and y, y ′ ∈ Y as our base distance [53] to define the (type-1) Wasserstein distance

Using Eq. 2, the Wasserstein distance between the joint distributions is the weighted sum of the Wasserstein distance between conditional distributions ( P ( z | y ) ) (Lemma 4 in App. A). Theorem 2 below explains the gap between the losses due to the distribution mismatch.

Assumption 1. 1) The composition of the loss function and the classifier ℓ ◦ h is a τ -Lipschitz function w.r.t to ∥ · ∥ 2 norm, i.e., | ℓ ( h ( z ) , y ) -ℓ ( h ( z ′ ) , y ) | ≤ τ ∥ z -z ′ ∥ 2 for all y ∈ Y , z, z ′ ∈ Z where h ∈ H . 2) P T ( y ) = P R ′′′ ( y ) .

The assumption 2), can be satisfied since we have full control on the prior P R ′′′ ( y ) via B and C .

Theorem 2. Let the distributions T and R ′′′ be defined on the same domain Z ×Y and assumption 1 holds, then

with d as in Eq. 2.

Theorem 2 shows that when ℓ ◦ h is τ -Lipschitz then the performance gap between the R ′′′ and T is bounded by the type-1 Wasserstein distance between the two distributions. The Lipschitz coefficient of the composition can be bounded by τ , by penalizing the gradient norm w.r.t z at training time. Thus, for linear fine-tuning, we train the classifiers h R and h T with an additional gradient norm penalty max { 0 , ∥∇ z ℓ ( h ( z ) , y ) ∥ 2 -τ } to make them conform to the Lipschitz assumption (see App. C.3). Note that constraining the Lipschitz constant restricts the hypothesis class. The trade-off between the Lipschitz constant and the performance of h is empirically evaluated in App. C.3.1.

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

Loss

RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

ADV(1) RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

SIMCLR RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

SWAV RN50

AG_News

YELP-5

SST-5

0

1

2

DistilRoBERTa

Target Loss

Reweighted Reference Loss

3.3 Bounding transferability using task-relatedness

Here, we combine the results obtained in Theorem 1 and Theorem 2. The final bound proposed in Theorem 3 is one of our main contributions which explains transferability as a sum of three interpretable and measurable gaps.

Theorem 3. Let ℓ be the cross-entropy loss, E P T ( z,y ) [ ℓ ( h T ( z ) , y )] ≤ E P R ( z,y ) [ C ( y ) ℓ ( h R ( z ) , y )] + H ( Y R ′′ |Y R ′ ) + τ W d ( P R ′′′ , P T ) .

The theorem shows that transferability can be decomposed into the loss incurred while transforming the class prior distribution, label space, and feature space of the reference distribution (first two terms) and the residual distance between the distribution obtained after transformations and the actual target distribution (last term). Based on the terms in the upper bound we define task-relatedness as follows.

Definition 2. (Task-relatedness). The relatedness between a target and a reference task is defined as E P R ( z,y ) [ C ( y ) ℓ ( h R ( z ) , y )] + H ( Y R ′′ |Y R ′ ) + τ W d ( P R ′′′ , P T ) .

A smaller value of the task-relatedness measure implies higher relatedness of the reference and the target tasks. In particular, when the target task is a transformation of the reference task then there exist transformations A,B, and C such that the distribution R ′′′ perfectly matches the distribution of the target task (i.e., W d ( P R ′′′ , P T ) = 0 ). Moreover, when labels are deterministically related (Corollary 3) our bound becomes an equality.

Lastly, while we presented an analysis for linear fine-tuning here (for simplicity of presentation), our bounds hold for non-linear classifiers and non-linear feature transformations as well (see App. A.5).

3.4 Estimating task-relatedness

The optimization problem for learning the transformations A,B , and C to compute task-relatedness in Theorem 3 is presented below. We use two new variables: inverse of the transformation A , denoted by ¯ A := A -1 and a transformed reference prior distribution denoted by D ( y ) := C ( y ) P R ( y ) .

Alg. 1 shows how we solve Eq. 4 (see App. D for additional details of the algorithm). Fig. 8 in App. C.1.2 shows how the upper bound is reduced as the optimization proceeds. Computationally, a single epoch of Alg. 1 takes a mere 0.17 seconds on our hardware for transfer from ImageNet to Pets for the ResNet-18 model (we ran Alg. 1 for 2000 epochs). In App. C.1, we show the effectiveness of learning the transformations using Alg. 1 on small-scale transfer tasks. Our results show that when

Algorithm 1 Minimization of the bound in Theorem 3

Input : Reference task samples and labels ( Z R , Y R ), Target task samples ( Z T ), Target task labels ( Y T ) (optional).

Output : Estimate of task-relatedness using the learned transformations A, ¯ A,B,D .

Init:

A := ¯ A := I , D := P R ( y ) , random B ∈ R K T × K R

1: Randomly sample n R points ( z i R , y i R ) ∼ ( Z R , Y R ) as per the class prior D .

3: Randomly sample n T points ( z j T , y j T ) ∼ ( Z T , Y T ) .

2: if Y T is available then

4: else

6: # Compute pseudo-labels for the target samples z T .

5: Randomly sample n T points ( z j T ) ∼ ( Z T ) .

7: y j T = arg max y ∈Y T Bh R ( A -1 z T ) for j = 1 , · · · , n T .

8: end if

9: Compute ( z i R ′′′ , y i R ′′′ ) = ( Az i R , arg max y Be ( y i R )) , for i = 1 , · · · , n R .

10: Assign Y R ′ := Y R and Y R ′′ := Y T .

11: Compute the optimal coupling π ∗ between the distributions R ′′′ and T by minimizing W d ( P R ′′′ , P T ) , i.e.,

12: Using π ∗ , solve for A, ¯ A,B,D using mini-batch SGD

13: Repeat 1 - 12 until convergence.

the reference task has classes semantically related to the target task, Alg. 1 learns transformations that achieve the smallest gap to transferability. However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A , in our experiments.

4 Empirical Analysis

Here, we empirically demonstrate the effectiveness of task-relatedness in explaining transferability in various settings. We present additional results in App. C and dataset/experimental details in App. D. Our codes can be found at https://github.com/akshaymehra24/TaskTransferAnalysis .

4.1 Task-relatedness achieves a small gap to actual transferability

Task-relatedness tightly upper bounds transferability across various architectures, pretraining methods, and datasets. We demonstrate this by using various pre-trained models with architectures such as Vision Transformers (ViT) [20], ResNet-18/50/101/152 [27], DistilRoBERTa [34] trained with various pretraining methods including supervised training, adversarial training [48], SimCLR [11], MoCo [26], SwAV [10], and MAE [25]. We also consider a wide range of target datasets including, CIFAR10/100, Aircraft, Pets, DTD, AG-News, Yelp-5, and SST-5 whose details are in App. D.

MNIST FMNIST

USPS

Target Tasks

FMNIST MNIST

USPS

Reference Tasks

3.43

2.80

3.76

3.53

3.12

3.65

Task-relatedness

0

1

2

3

MNIST FMNIST

USPS

Target Tasks

FMNIST MNIST

USPS

Reference Tasks

1.72

1.23

1.77

1.64

1.14

1.70

Transferability

0.0

0.5

1.0

1.5

(a)

PE

FFE

PE

FFE

0

1

2

3

Loss

2.02

2.46

USPS

1.88

2.03

2.20

2.48

MNIST-M

2.12

2.27

MNIST

PE

FFE

PE

FFE

0

1

2

3

2.02

2.66

USPS

1.94

2.15

2.20

2.65

MNIST-M

2.04

2.25

SVHN

Target Loss

Reweighted Reference Loss

(b)

For this experiment, we fix the reference task to be ImageNet [17] for image classification and to DBPedia for sentence classification tasks and use Alg. 1 to estimate task-relatedness. The results in Fig. 3 and 9 (in the Appendix) show that our bound achieves a small gap to actual transferability. As the task-relatedness between the reference and the target tasks improves, transferability also improves showing that task-relatedness and transferability are strongly correlated. Task-relatedness is also strongly correlated with the accuracy of the end-to-end fine-tuned classifiers on the target task. In Fig. 11 (in the Appendix), we show high Pearson correlation coefficients ( ≥ -0 . 57 ) for task-relatedness and accuracy after fully fine-tuning various pre-trained encoders using data from various target tasks.

4.2 Effect of the reference task on task-relatedness

Highly related reference-target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks. To understand how a reference task affects task-relatedness and eventually transferability, we consider two experiments using convolutional and CLIP-trained models with various character recognition tasks such as MNIST, Fashion-MNIST (FMNIST), SVHN, MNIST-M, and USPS. Of these datasets, SVHN and MNIST-M contain colored images while the rest contain gray-scale images. In the first experiment, we train convolutional models on MNIST, FMNIST, and USPS and measure pairwise transferability. Here we use the reference task to be the same task as that used for training the models. The results in Fig. 4(a) show that transferability to those target tasks is higher for which task-relatedness metric's value is smaller. Specifically, USPS achieves the best transferability ( 1 . 23 ) and the smallest task-relatedness ( 2 . 80 ) when the reference task is MNIST. This is attributed to both datasets containing gray-scale images of digits. On the other hand, when the reference task is unrelated to the target task i.e., the task-relatedness value is high, transferability suffers, e.g., when the reference task is MNIST and the target task is FMNIST. Results in App. C.2.2 show similar results for the sentence classification task.

The gap between task-relatedness and transferability is smaller when a reference task performs well with a given encoder. Here we use MNIST and SVHN as two reference tasks and compute the task-relatedness and transferability with USPS and MNIST-M as target tasks, using CLIP (Vit B32) model. A linear classifier trained on top of the embeddings from the CLIP model achieves ≈ 98% accuracy on MNIST but only ≈ 61% accuracy for SVHN. Due to this, transferability (USPS:2.02, MNIST-M:2.20) explained using task-relatedness with MNIST as the reference task (USPS: 2.46 , MNIST-M: 2.48 ) is better than that computed using SVHN (USPS:2.66, MNIST-M:2.65) as the reference, even though MNIST-M is intuitively more similar to SVHN (as both contain colored images of digits). This is evident from the results of PE (Pre-trained Encoder) in Fig. 4(b).

Improving the performance of an encoder on a reference task improves transferability to other related (potentially unseen) tasks. To show this we fully fine-tune the CLIP encoder on MNIST and SVHN tasks, increasing the accuracy of the classifiers for both MNIST and SVHN to 99% and 95%, respectively. Using the representations from these new encoders, we find that the transferability of both related target tasks improves along with task-relatedness (see FFE results in Fig. 4(b)). Here,

we see that task-relatedness for MNIST-M and USPS is the best when the reference task is SVHN and MNIST, respectively, aligning with our intuition of semantic relatedness between these tasks. This also suggests that transferability on other related tasks can be improved by fully fine-tuning the encoder on these reference tasks. Thus, in scenarios where target tasks are private (such as proprietary Chest X-rays), an encoder trained to work well on related tasks (such as publicly available Chest X-rays) is bound to achieve good transferability.

4.3 Task-relatedness for end-to-end transferability estimation

In this section, we show an efficient way of computing task-relatedness which enables its use for estimating transferability after end-to-end fine-tuning. While Alg. 1, accurately estimates taskrelatedness by minimizing the bound in Eq. 3, it could be inefficient due to the requirement of computing and minimizing the Wasserstein distance between distributions at every epoch. Thus, to make the computation efficient, we replace the Wasserstein distance computation in step 11 and 12 of Alg. 1, with mean and covariance matching terms. Specifically, we define the distance between two distributions R ′′′ and T as

where µ R ′′′ /T := 1 n R ′′′ /T ∑ z ∈ P R ′′′ /T z , Σ R ′′′ /T := 1 n R ′′′ /T ∑ z ∈ P R ′′′ /T ( z -µ R ′′′ /T ) T ( z -µ R ′′′ /T ) , and λ is a regularization coefficient. Using Γ( R ′′′ , T ) in place of W d ( R ′′′ , T ) , makes the computation of task-relatedness by learning transformations A,B, and C significantly more efficient.

Task-relatedness is an effective metric for the pre-trained model selection problem. The goal of this problem is to find a pre-trained model from a model zoo that achieves the best accuracy on a given target task after end-to-end fine-tuning of the model using labeled target data. Since end-to-end fine-tuning is costly (takes almost a day to fully fine-tune a single model on a single target task as shown by [61]), an effective transferability metric is significantly more efficient to compute and is correlated well with the accuracy after end-to-end finetuning. Using 5 different pre-trained models (supervised ResNet-50/101/152, adversarially pre-trained [48] ResNet50 with ϵ ∈ { 0 . 1 , 1 } ) and ImageNet as the reference task, we show in Table 1, that task-relatedness achieves a high correlation with the accuracy after end-to-end fine-tuning on the target task. Our results also highlight the instability of various popular SbTE metrics, such as LogMe [61] and NCE [56] which can produce a high negative correlation, and those of PACTran [19] which achieve low correlation values on complex datasets. In comparison, task-relatedness consistently achieves a good correlation for various target tasks. Computationally, it takes a mere 3-4 minutes to learn the transformations to compute task-relatedness, providing a significant computation advantage over end-to-end fine-tuning. We also show that task-relatedness remains highly correlated with end-to-end fine-tuning accuracy even with a limited amount of labeled data from the target task as shown in Fig. 5 unlike other SbTE metrics.

Next, we show that task-relatedness can even be estimated without using labels from the target task.

For scenarios, where labeled data from the target task is unavailable, estimating transferability is challenging. This is because both fine-tuning and most SbTE methods require labels to compute the transferability scores. Here we show that task-relatedness can still be an effective measure to estimate transferability in this challenging setting. Since we use a transformative model and have access to a reference task/classifier, we can use the predictions from the reference task's classifier trans-

Table 2: Correlation of task-relatedness and end-to-end fine-tuning accuracy computed using true and pseudo labels of the target task.

formed via B (to obtain labels ∈ Y T ) and estimate the pseudo -labels of the target data. Concretely, pseudo-label for a target sample x T is obtained as y pseudo T = arg max y ∈Y T Bh R ( A -1 ( z T )) . Results in Table 2 show that our task-relatedness estimated via pseudo-labeled target data still achieves a high correlation to transferability on most datasets. For datasets such as Pets and DTD, where transforming the reference task classifier produces high accuracy on the target task, the difference between the pseudo and true labels is small. Consequently, the difference in the correlations with pseudo and true labels is also small. Thus, when the reference and target tasks are related, transferability can be estimated accurately without requiring labels of the target task, showing that task-relatedness is an effective metric even for unsupervised transferability estimation.

5 Conclusion

Weanalyzed TL in terms of the relatedness between the target and a reference task. Our analysis works by transforming the distribution of a reference task to match that of the target. Using this we proved an upper bound on transferability, defined as task-relatedness, consisting of three interpretable terms, namely, the re-weighted reference task loss, label mismatch, and distribution mismatch. We proposed an algorithm to compute task-relatedness and demonstrated its effectiveness at accurately predicting transferability (even without target labels) with SOTA models. Moreover, the high correlation of task-relatedness with accuracy after end-to-end fine-tuning and its efficient computability, makes it an effective metric for transferability estimation.

Limitations. We studied transferability using the cross-entropy loss and used Wasserstein distancebased distribution shift analysis due to their popularity. However, due to accuracy being the primary metric of interest in classification tasks and the difficulty of computing the Wasserstein distance with limited samples in a high dimensional representation space, extending the analysis to 0-1 loss and other divergence measures are important directions which are not addressed here and are left for future works.

6 Acknowledgment

We thank the anonymous reviewers of this work for their insightful comments and suggestions. This work was supported by the NSF EPSCoR-Louisiana Materials Design Alliance (LAMDA) program #OIA-1946231.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for metalearning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6430-6439, 2019.

Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804 , 2019.

David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. arXiv preprint arXiv:2002.02923 , 2020.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoretic approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP) , pages 2309-2313, 2019.

Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoretic approach to transferability in task transfer learning. In 2019 IEEE international conference on image processing (ICIP) , pages 2309-2313. IEEE, 2019.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79(1):151-175, 2010.

Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems , 19:137, 2007.

Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning theory and kernel machines , pages 567-580. Springer, 2003.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Quan Cui, Bingchen Zhao, Zhao-Min Chen, Borui Zhao, Renjie Song, Boyan Zhou, Jiajun Liang, and Osamu Yoshie. Discriminability-transferability trade-off: An information-theoretic perspective. In European Conference on Computer Vision , pages 20-37. Springer, 2022.

Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional networks. Advances in neural information processing systems , 29, 2016.

Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 447-463, 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.

Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pacbayesian metrics for estimating the transferability of pretrained models to classification tasks. In European Conference on Computer Vision , pages 252-268. Springer, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.

Kshitij Dwivedi, Jiahui Huang, Radoslaw Martin Cichy, and Gemma Roig. Duality diagram similarity: a generic framework for initialization selection in task transfer learning. In European Conference on Computer Vision , pages 497-513. Springer, 2020.

Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12387-12396, 2019.

Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aureie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. The Journal of Machine Learning Research , 22(1):3571-3578, 2021.

Jiechao Guan and Zhiwu Lu. Task relatedness-based generalization bounds for meta learning. In International Conference on Learning Representations , 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

Minyang Hu, Hong Chang, Zong Guo, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes. Advances in Neural Information Processing Systems , 36, 2024.

Long-Kai Huang, Ying Wei, Yu Rong, Qiang Yang, and Junzhou Huang. Frustratingly easy transferability estimation, 2022.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. . , 2009.

Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Certifying model accuracy under distribution shifts. arXiv preprint arXiv:2201.12440 , 2022.

Trung Le, Tuan Nguyen, Nhat Ho, Hung Bui, and Dinh Phung. Lamda: Label matching deep domain adaptation. In International Conference on Machine Learning , pages 6043-6054. PMLR, 2021.

Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, and Soheil Kolouri. Wasserstein task embedding for measuring task similarities. arXiv preprint arXiv:2208.11726 , 2022.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430 , 2009.

Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, and Jihun Hamm. Understanding the limits of unsupervised domain adaptation via data poisoning. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, and Jihun Hamm. Do domain generalization methods generalize well? In NeurIPS ML Safety Workshop , 2022.

Akshay Mehra, Yunbei Zhang, and Jihun Hamm. Test-time assessment of a model's performance on unseen domains via optimal transport. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 173-182, June 2024.

Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations, 2020.

Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. Exploring the role of task transferability in large-scale multi-task learning. arXiv preprint arXiv:2204.11117 , 2022.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359, 2010.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 , 2017.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems , 33:3533-3545, 2020.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.

Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104.09425 , 2021.

Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: predicting model transferability in a self-challenging fisher space. In European Conference on Computer Vision , pages 286-302. Springer, 2022.

Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 , 2017.

Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song. Depara: Deep attribution graph for deep knowledge transferability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3922-3930, 2020.

Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations, 2021.

Anh T. Tran, Cuong V. Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks, 2019.

Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems , 33:7852-7862, 2020.

Cédric Villani. Optimal transport: old and new , volume 338. Springer, 2009.

Karl Weiss, Taghi M Khoshgoftaar, and Dingding Wang. A survey of transfer learning. Journal of Big Data , 3(1):9, May 2016.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning, 2021.

Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3712-3722.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems , 28, 2015.

Appendix

We present the missing proofs of the theoretical results from Sec. 3 along with justifications for the classifiers ( h R ′ , h R ′′ , h R ′′′ ) as Corollaries in Appendix A followed by related work on learning in the presence of distribution shift with the same feature and label space in Appendix B. This is followed by additional experimental results including NLP classification tasks with large pre-trained models in Appendix C. We conclude in Appendix D with details of the experiments and datasets used.

A Proofs for Sec. 3

A.1 Notation

A.2 Our task transformation model (Sec. 3.1)

A.2.1 Class-Prior transformation ( R → R ′ )

Proof.

A.2.2 Label transformation ( R ′ → R ′′ )

Lemma 2. Let B be a K T × K R matrix with B ij = P ( y R ′′ = i | y R ′ = j ) and h R ′′ ( z ) := Bh R ′ ( z ) and ℓ be the cross-entropy loss. Then, E P R ′′ ( z,y ) [ ℓ ( h R ′′ ( z ) , y )] ≤ E P R ′ ( z,y ) [ ℓ ( h R ′ ( z ) , y )] + H ( Y R ′′ |Y R ′ ) , where H ( Y R ′′ |Y R ′ ) is the conditional entropy ( -∑ y R ′ ∈Y R ′ ∑ y R ′′ ∈Y R ′′ P R ′ ( y R ′ ) B y R ′′ ,y R ′ log( B y R ′′ ,y R ′ )) .

Proof. Note that P ( z ) := P R ′ ( z ) = P R ′′ ( z ) by construction.

Since the loss ℓ is the cross-entropy loss, we have

Therefore, we have

Corollary 1 below, shows the conditions under which the optimal softmax classifier for the domain R ′ remains optimal for the domain R ′′ , justifying our choice of classifier change from R ′ to R ′′ .

Corollary 1. Let e be one-hot encoding of the labels, |Y R ′′ | = |Y R ′ | , B be a K T × K R permutation matrix and h R ′ be the optimal softmax classifier for R ′ and y R ′′ := σ ( y R ′ ) := arg max y ∈Y R ′′ ( Be ( y R ′ )) y then under the assumptions of Lemma 2, h R ′′ ( z ) := Bh R ′ ( z ) is the optimal softmax classifier for R ′′ .

Proof. Since y R ′′ := σ ( y R ′ ) := arg max y ∈Y T ( Be ( y R ′ )) y we have

The second last equality follows due to the symmetry of cross-entropy loss, i.e., ℓ ( h, y ) = -log h y = -log Bh σ ( y ) = ℓ ( Bh,σ ( y )) .

Since min h R ′′ E P R ′′ [ ℓ ( h R ′′ ( z ) , y R ′′ )] = min h R ′ E P R ′ [ ℓ ( h R ′ ( z ) , y R ′ )] and h R ′ is optimal for R ′ we have h R ′′ ( z ) := Bh R ′ ( z ) is the optimal softmax classifier for R ′′ .

A.2.3 Feature transformation ( R ′′ → R ′′′ )

Lemma 3. Let A : Z → Z be an invertible linear map of features and the classifier h R ′′′ ( z ) := h R ′′ ( A -1 ( z )) . Then E P R ′′′ ( z,y ) [ ℓ ( h R ′′′ ( z ) , y )] = E P R ′′ ( z,y ) [ ℓ ( h R ′′ ( z ) , y )] for any loss ℓ .

Our Corollary 2 below shows that the optimal softmax classifier for domain R ′′ remains optimal for domain R ′′′ too.

Corollary 2. Let h R ′′ be the optimal softmax classifier in domain R ′′ then under the assumptions of Lemma 3, h R ′′′ ( z ) = h R ′′ ( A -1 ( z )) is the optimal softmax classifier in domain R ′′′ .

Proof. When h R ′′′ ( z ) = h R ′′ ( A -1 ( z )) , min h R ′′ E P R ′′ ( z,y ) [ ℓ ( h R ′′ ( z ) , y )] = min h R ′′′ E P R ′′′ ( z,y ) [ ℓ ( h R ′′′ ( z ) , y )] by Lemma 3, hence if h R ′′ is optimal for R ′′ then so is h R ′′′ for the domain R ′′′ .

A.2.4 Three transformations combined ( R → R ′′′ )

Theorem 1. Let C := [ P R ′ ( y ) P R ( y ) ] K R y =1 be a vector of probability ratios , B be a K T × K R matrix with B ij = P ( y R ′′ = i | y R ′ = j ) , A : Z → Z be an invertible linear map of features. Let the classifiers h R ′ ( z ) := h R ( z ) , h R ′′ ( z ) := Bh R ′ ( z ) , h R ′′′ ( z ) := h R ′′ ( A -1 ( z )) . Assuming ℓ is the cross-entropy loss, we have

Proof.

Corollary 3. Let e be one-hot encoding of the labels, | Y R ′′′ | = | Y R | , B : ∆ R ′ → ∆ R ′′ be a permutation matrix and y R ′′ := σ ( y R ′ ) := arg max y ∈Y R ′′ ( Be ( y R ′ )) y then under the assumptions of Lemmas 1, 2, and 3 we have E P R ′′′ ( z,y ) [ ℓ ( h R ′′′ ( z ) , y )] = E P R ( z,y ) [ C ( y ) ℓ ( h R ( z ) , y )] .

Proof. Since y R ′′ := σ ( y R ′ ) := arg max y ∈Y T ( Be ( y R ′ )) y we have

The second last equality follows due to the symmetry of cross-entropy loss, i.e., ℓ ( h, y ) = -log h y = -log Bh σ ( y ) = ℓ ( Bh,σ ( y )) .

Therefore, we have

A.3 Distribution mismatch between R ′′′ and T (Sec. 3.2)

Lemma 4. Let U and Q be two distributions on Z × Y with the same prior P U ( y = i ) = P Q ( y = i ) = P ( y = i ) . With the base distance d defined as in Eq. 2, we have W d ( P U , P Q ) = ∑ y P ( y ) W ∥·∥ 2 ( P U ( z | y ) , P Q ( z | y )) .

Proof.

A.4 Final bound (Sec. 3.3)

Theorem 3. Let ℓ be the cross entropy loss then under the assumptions of Theorems 1 and 2 we have,

̸

̸

Proof. Let ω ∗ y denote the optimal coupling for the conditional distributions ( P U ( z | y ) , P Q ( z | y )) for y ∈ Y and π ∗ denote the the optimal coupling for the joint distributions ( P U ( z, y ) , P Q ( z, y )) . Then, under the definition of our base distance d , π ∗ (( z, y ) , ( z ′ , y ′ )) = 0 when y = y ′ i.e. no mass from the distribution U belonging to class y can be moved to the classes y ′ = y of the distribution Q when the class priors of U and Q are the same. Moreover, since ∑ ij ( ω ∗ y ) ij = 1 and ∑ { i,j : y i = y ′ j = k } π ∗ ij = P ( y = k ) for k ∈ Y we have π ∗ (( z, y ) , ( z ′ , y ′ )) = ω ∗ y ( z, z ′ ) P ( y )1 y = y ′ for every y, y ′ ∈ Y .

̸

Then, we can show that the total Wasserstein distance between the joint distributions can be expressed as the sum of conditional Wasserstein distances, as follows

̸

Theorem 2. Let the distributions T and R ′′′ be defined on the same domain Z × Y and assumption 1 holds, then E P T ( z,y ) [ ℓ ( h ( z ) , y )] -E P R ′′′ ( z,y ) [ ℓ ( h ( z ) , y )] ≤ τ W d ( P R ′′′ , P T ) , with d as in Eq. 2.

︸

︷︷

︸

Distribution mismatch

̸

Proof. Let ℓ ◦ h T , ℓ ◦ h R , and ℓ ◦ h R ′′′ be τ -Lipschitz (from Assumption 1).

In our experiments, we enforce the τ -Lipschitz constraint for ℓ ◦ h R and ℓ ◦ h T and verify that the Lipschitz constant of ℓ ◦ h R ′′′ remains close to τ within tolerance.

A.5 Extension to non-linear classifiers and non-linear transformations

To extend our analysis to non-linear classifiers/transformations, we allow A : Z → Z to be a nonlinear map and the classifiers h ∈ H non _ linear to be also non-linear (such as multi-layer perceptron). In addition to the Assumption 1 (1), which requires ℓ ◦ h R ′′′ to be τ -Lipschitz, we also require that h R ′′′ belongs to the same class H non _ linear as h T and h R for any A . This holds for example when h is linear and A is also linear or when h is a multilayer perceptron and A is linear. With this additional assumption, all of our proofs work for any linear/non-linear transformation of the feature space, without any change. Thus, Theorem 1, Theorem 2 and Theorem 3 hold for non-linear classifiers as well. With these extensions, our bounds can be used to explain transferability even with non-linear classifiers.

B Additional related work

Distributional divergence-based analyses of learning with distribution shifts (under same feature and label sets): Here we review some of the previous works that analyzed the problem of learning under distribution shifts in terms of distributional divergences such as the Wasserstein distance. These analyses apply when the feature and label spaces remain the same between the original and the shifted distribution.

Early works [8, 52, 36] showed that the performance on a shifted distribution (target domain) can be estimated in terms of the performance of the source domain and the distance between the two domains' marginal distributions and labeling functions. Specifically, [8] showed that that

where d 1 denotes the total variation distance, f : Z → [0 , 1] denotes the labeling function, h : Z → { 0 , 1 } denotes the hypothesis and E P ( h, f ) := E z ∼ P [ | h ( z ) -f ( z ) | ] denote the risk of the hypothesis h . A follow up work [52], showed a similar result using type-1 Wasserstein distance for all K -Lipschitz continuous hypotheses i.e.,

where λ is the combined error of the ideal hypothesis h ∗ that minimizes the combined error E S ( h, f S ) + E T ( h, f T ) . Another recent work [32] used a target transformation-based approach and Wasserstein distance to quantify learning in the presence of data and label shifts. Other works [31, 50] also presented an analysis based on Wasserstein distance to understand how the accuracy of smoothed classifies and robustness change in the presence of distribution shifts. Compared to these works the bound proposed in Theorem 2 considers cross-entropy loss (which is a popular choice of the loss function in the classification setting) and uses a joint feature and labels Wasserstein distance rather than only marginal Wasserstein distance. These differences make the bound proposed in Theorem 2 useful in the analysis of transfer learning than those proposed in previous works when we have access to labeled target domain data.

Comparison with [56]: The closest work to ours is that of [56], which showed that transferability to a target task can be related to transferability to another task (source task in their work) and the label mismatch between the two tasks. However, the bound is proposed in a restrictive setting when both the tasks have the same features but different labels (i.e. same images labeled differently between the two tasks). In this setting, [56], showed that transferability is upper bounded by loss of the source classifier on the source task and the conditional entropy (CE) of the label sets of the two tasks. We

significantly extend this analysis to general tasks which is the most commonly used setting in practice (e.g., our analysis allows us to study transfer learning from ImageNet with 1000 classes to CIFAR-100 with 100 unrelated classes, where both tasks have different images). In this setting, our main result in Theorem 3, shows that transferability involves additional terms such as the distribution mismatch term (in the form of Wasserstein distance), the prior mismatch term (in the form of re-weighted source loss) and the conditional entropy between the label sets. Moreover, the bound proposed by [56] is a special case of our bound with C being the vector of all ones (no prior change) and A being Identity (data distributions of two tasks are the same).

Comparison with other transferability estimation and model selection methods: The problem of transferability estimation has gained a lot of attention recently, especially due to the availability of a larger number of pre-trained models. Earlier works used models end-to-end fine-tuned on target tasks to evaluate transferability via task-relatedness estimated using the actual target loss [62] and the Fisher information matrix [1]. However, the requirement of models trained on target tasks limits their practical utility. Other works [22, 21, 54] used the representation of the data from the tasks from these end-to-end fine-tuned models and developed similarity scores that achieved high correlation with transferability. However, these approaches are not practical since they require models end-to-end fine-tuned on the target task. Our work does not depend on models trained on target tasks as shown in our analysis (Theorem 3) making it both theoretically and practically sound.

Another line of work [5, 40, 29, 61, 55, 51, 19] focuses on the problem of pre-trained model selection where the goal is to find the best pre-trained classifier from a model zoo that will achieve the highest transferability to a particular downstream task. Thus, the main challenge of this problem is to be able to estimate transferability in a way that is more efficient than fine-tuning the pre-trained models on the downstream tasks. To this end, recent works have proposed several scores that are correlated with the accuracy of the models after fine-tuning them on the target task, which we refer to as score-based transferability estimation (SbTE) methods, in our work.

However, unlike our work, the goal of SbTE works is not to propose a universal bound or identify terms that provably govern transferability. Moreover, while the scores proposed in SbTE works correlate well with transferability, they are only meaningful in a relative sense. Concretely, a score of 1 (e.g., for LogMe [46]) on a CIFAR-100 task for a particular model does not indicate whether transferability is good or bad and requires comparison with scores of other pre-trained models on the same target task. On the other hand, our upper bound directly approximates transferability, e.g., an upper bound of 1 on the CIFAR-100 task for a model implies that transfer learning will incur an average cross-entropy loss of less than 1 implying a highly accurate transfer. Our results in Figs. 3, and 9 attest that our upper bound is indeed a good estimate of the transferability.

Another disadvantage of the scores proposed in these works is that they cannot be compared across target tasks, unlike our upper bound. As observed from Fig. 4 of LogMe [46], scores for CIFAR-10 are lower than scores for CIFAR-100 on the same pre-trained model, but, the transferability to CIFAR-10 is better than that to CIFAR-100. On the other hand, from our Fig. 9, the upper bounds on CIFAR-10 are lower than those of CIFAR-100 implying better transferability of classifiers pre-trained on ImageNet to CIFAR-10. Thus, our work is more suitable for estimating the absolute performance on various target tasks given a particular pre-trained classifier.

Thus, while the goal of our work differs fundamentally from SbTE approaches the problem addressed in this work is of significant practical importance. In Sec. 4.3, we show that task-relatedness estimated via Alg. 1 achieves competitive performance compared to popular SbTE methods on this problem.

Comparison with task transfer learning approaches: This approach focuses on explaining the transfer performance based on the relationship between tasks. For instance, works such as [28, 57, 24] study the problem of few-shot learning (FSL) where a model is trained on data from related training tasks and is adapted to an unseen task using only a few samples. Different from these works, we focus on the setting where a pre-trained encoder trained on some pre-training dataset is adapted with enough samples/gradient steps to a downstream task. This downstream target task may or may not have any relation to the pre-training task unlike [28, 57, 24].

Concretely, [28] proposed a model-agnostic metric called Task Attribute Distance to gauge the success of transfer. Our work, on the other hand, defines task-relatedness based on the similarity of the representations of reference and target tasks in the representations of the pre-trained model (and is model dependent) rather than relying on the attribute information, which may not be available in TL

setting. [57] analyzes the sample complexity for learning a model shared across tasks and adapting it to a new target task and shows task diversity to be a crucial component for the success of transfer in the FSL setting. Our work on the other hand does not assume access to shared tasks or restrict the number of samples required for fine-tuning on the target task. Moreover, their notion of task diversity requires access to a set of training tasks that may not be available in the TL setting, making our notion of task-relatedness more practical for TL. Lastly, [24] proposes a notion of task-relatedness for the FSL setting, allowing to utilize all the data from available training tasks to help learn a model on a new task with a few gradient steps. This notion is model-agnostic and defined over the sample space ( X × Y ) unlike our measure which is defined in the representation space of the model whose transferability needs to be evaluated.

Thus while task-relatedness is at the core TL, the notions proposed by [28, 57, 24] are suitable for task transfer setting where as our notion is more suitable for the inductive TL setting.

C Additional experiments

C.1 Small-scale experiment to evaluate Alg. 1

1) Our algorithm produces transformations that achieve a better value for the bound compared to naive transformations. 2) Having the same number of classes in the reference task as in the target task leads to the best value of the bound. 3) It is only marginally better to have semantically related classes in the reference task.

We evaluate Alg. 1 for computing task-relatedness and predicting the transferability of a ResNet-18 model to CIFAR-10 in two settings. In the first setting, we consider a reference task that comprises data from 10 classes chosen at random and 10 classes that are semantically related to the labels of CIFAR10 from ImageNet [17] (see App. D) whereas in the setting we select reference task with 20 classes. We compare the cross-entropy loss on CIFAR10 obtained after linear fine-tuning (transferability) with our bound, by using various transformations including those learned via Alg. 1.

We test 3 different cases. The first is FixedAll: where all transformations are fixed ( A is fixed to the Identity matrix, B is fixed to a random permutation matrix in the setting with 10 classes, and to a matrix such that two reference classes match a single target class in the setting with 20 classes, D is fixed to the reference prior). The second is LearnedA: where we use Alg. 1 to learn only A while B,D are fixed as in FixedAll, and lastly LearnedAll: where all the transformations are learned via Alg. 1. Our results in Fig. 6, show that in both settings using FixedAll, our bound is marginally better when the reference task has classes semantically related to the target task. In all cases, the bound becomes significantly better in the LearnedA setting due to learning the transformation A that transforms the reference distribution to better match the target and decrease the distribution mismatch term of the bound. Lastly, by learning all the transformations, in the LearnedAll setting, we achieve the best value for the bound, regardless of whether there are randomly chosen or semantically related classes in the reference task. Moreover, we see that the value of the bound estimated when the reference task has 10 source classes is better in all cases. This is due to smaller re-weighted reference loss since learning a 20-way classifier is more challenging than learning a 10-way classifier, especially with the Lipschitz constraint. In the setting with 20 classes, Alg. 1, prefers

Using 10 reference classes

Using 20 reference classes

to retain data from 10 of the 20 reference task classes, Fig. 7 (right), reducing the re-weighted reference loss, makes B sparse, reducing the label mismatch term, and aligns the reference and target distributions, via A , reducing the distribution mismatch term.

Overall, our results show that 1) learning D prefers to retain the data from the same number of reference classes as those present in the target, 2) the B matrix eventually becomes sparse, making the label mismatch term zero, and 3) there is only a small difference in task-relatedness between LearnedA and LearnedAll settings. Based on these insights, we use Alg. 1 in the LearnedA setting, with the reference task containing the same number of randomly sampled classes from ImageNet as the same number of classes in the target task (since it may not always possible to find semantically similar classes for all target tasks) and fix B to be a random permutation matrix, for all other experiments in the paper. We select classes for the reference task from ImageNet due to its diversity and the fact that it has a large number of classes. However, any dataset can be used to define the reference task (e.g., we used Digits datasets for the reference task in Sec. 4.2).

Next, we evaluate the effect of knowing the exact matching between the labels of the reference and the target tasks in comparison to a random permutation. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in a setting where only A is learned and B is set to an identity matrix and when B is set to a random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9).

We find that the upper bound obtained when B is fixed to identity is only marginally better than the case when B is a random permutation. Specifically, the difference between the bound when B is fixed to a random permutation and when B is an identity matrix is 0.10 for the MNIST → USPS task and 0.17 for the USPS → MNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term. While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related such as for FMNIST and MNIST. Thus, fixing B to a random permutation matrix yields a reliable estimate of transferability in most cases.

C.1.1 Visualization of the transformed data via t-SNE for various settings in Sec. C.1

Here we use the setting considered in App. C.1 where we consider 20 randomly selected classes from ImageNet as the reference task and consider the transfer to CIFAR-10. We plot the results of using different transformations using t-SNE to show how various transformations affect the upper bound in Theorem 3. Our results in Fig. 7(left) show that when no transformations are learned (FixedAll), the 20 random reference task classes do not overlap with the 10 target classes leading to an increased Wasserstein distance which in turn leads to a larger upper bound. By learning the transformation A (LearnedA), Fig. 7(center) shows a significantly better alignment between the classes of the reference and target which leads to a decreased Wasserstein distance and hence a tighter upper bound. Moreover, by learning all the transformations (LearnedAll), Fig. 7(right) shows that not only do the distributions align well but also the prior of the reference is changed to only keep 10 reference classes to match the prior of the target distribution providing a further improvement in the upper bound. This clearly shows the effectiveness of our proposed optimization algorithm in learning various transformations to minimize the upper bound.

C.1.2 Effectiveness of minimizing the upper bound in Theorem 3 via solving Eq. 4

In Fig. 8, we show how the upper bound changes as the optimization progresses to transform the reference task (ImageNet) into four target tasks with the ResNet-18 model. Similar to experiments in Sec. 4.1 of the paper we optimize over the transformation A while B and D are fixed to a random permutation matrix and the reference prior. After about 600 epochs the optimization problem converges to a local minima.

C.2 Additional results for the impact of the reference task on task-relatedness (Sec. 4.2)

C.2.1 Additional results for image classification

Here we provide details of the experiment presented in Sec. 4.2 about the effect of the reference task on task-relatedness and transferability. Similar to the results presented in Fig. 4 of the main paper, the results in Fig. 10 show that when the reference and the target tasks are related then task-relatedness is good as in the case when the reference task is MNIST and target is USPS or vice versa achieving a smaller gap to transferability. When the target tasks are unrelated to the reference task data then both the transferability and task-relatedness are low. E.g., when the reference task is MNIST and the

Target

Target

Target

target is FMNIST or vice-versa. Task-relatedness is also correlated to transferability, i.e., a model trained on MNIST achieves better transferability to USPS than FMNIST.

C.2.2 Results for NLP sentence classification task

In this section, we use sentence classification NLP task to demonstrate further the effect of the reference task on task-relatedness. For this experiment, we first fine-tune the entire DistilRoBERTa [34] model distilled on OpenAI's WebText dataset, using a subsample of 10,000 points from the DBPedia dataset. We then use these fine-tuned models to evaluate the transferability to AG news, SST-5, and Yelp datasets. The results in Fig. 12 show that transferability on AG News is the smallest among the three datasets. This coincides with the task-relatedness value obtained after learning the

0

2

4

Loss

RN18

0

2

4

RN50

0

2

4

0

2

4

RN152

0

2

4

CLIP RN50

0

2

4

Loss

ADV(0.1) RN18

0

2

4

ADV(0.1) RN50

0

2

4

ADV(1) RN18

0

2

4

ADV(1) RN50

0

2

4

CLIP Vit-B32

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

Loss

SIMCLR RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

MOCO RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

SWAV RN50

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

MAE ViT-B16

CIFAR10

CIFAR100-S

Pets

DTD

CIFAR100-M

CIFAR100

Aircraft

0

2

4

CLIP ViT-B16

Target Loss

Reweighted Reference Loss

(a) Image classification tasks

DistilBERT

2

DistilRoBERTa

1

YELP-5

SST-5

AG_News

YELP-5

SST-5

0

(b) Sentence classification tasks

USPS

FMNIST

0

2

4

Loss

MNIST

USPS

MNIST

0

2

4

FMNIST

MNIST

FMNIST

0

2

4

USPS

AG_News

YELP-5

SST-5

0

1

2

DBPedia

Target Loss

Reweighted Reference Loss

transformations which explains why transfer from DBPedia to AG News is more successful compared to other target tasks. This observation is reasonable, especially considering that both DBPedia and AG News have structured information. Moreover, since DBPedia is related to Wikipedia, the terms and entities appearing in AG News are more related to those appearing in DBPedia in comparison to terms/entities appearing in SST-5 and Yelp which consist of movie reviews and reviews collected from Yelp.

For our experiments, in this section, we follow a similar setting of fixing B to be a random permutation matrix, C to the prior of the reference task, and only learn the transformation A . We sample 10,000 points from DBPedia belonging to the same number of classes as those present in the target task (e.g., for AG News we sample data from 4 randomly selected classes of DBPedia) and use this data as the reference task data to train h R with gradient norm penalty ( τ = 0.02). All experiments are run for 3 random seeds and average results are reported in Fig. 12.

2

1

0

AG_News

75

80

85

90

95

3.0

3.5

4.0

4.5

5.0

ResNet-50 (rho=-0.63)

Accuracy (%)

Task-relatedness

80

85

90

95

3.0

3.5

4.0

4.5

5.0

Accuracy (%)

Task-relatedness

75

80

85

90

95

3.5

4.0

4.5

5.0

ResNet-152 (rho=-0.59)

Accuracy (%)

Task-relatedness

75

80

85

90

95

3.0

3.5

4.0

4.5

5.0

Adv ResNet-50 (eps=0.1, rho=-0.62)

Accuracy (%)

Task-relatedness

75

80

85

90

95

3.0

3.5

4.0

4.5

5.0

Adv ResNet-50 (eps=1, rho=-0.57)

Accuracy (%)

Task-relatedness

AG News

SST 5

Yelp 5

Target Tasks

DBPedia

Reference Task

1.43

1.72

1.65

Task-relatedness

1.45

1.50

1.55

1.60

1.65

1.70

AG News

SST 5

Yelp 5

Target Tasks

DBPedia

Reference Task

1.18

1.56

1.55

Transferability

1.20

1.25

1.30

1.35

1.40

1.45

1.50

1.55

C.3 Lipschitz constrained linear fine-tuning

C.3.1 Implementing softmax classification with τ -Lipschitz loss

To use the bound Theorem. 3, it is required that the loss be τ -Lipschitz continuous w.r.t. z in the input domain Z . To enforce this, while learning the weights of the softmax classifier (linear fine-tuning) for the reference task or the target, we add the gradient norm penalty as used in previous works [52, 4] and solve the following optimization problem

where ℓ ( h ( z ) , y ) = -w T y z +log ∑ j e w T j z is the cross-entropy loss.

C.3.2 Trade-off between empirical and predicted transferability

Constraining the Lipschitz coefficient of the classifier increases both the target and the reference task cross-entropy loss since the hypothesis set is being restricted. The smaller the τ is, the larger the loss becomes. On the other hand, the smaller τ makes the distribution mismatch term in Theorem 3 also smaller. Since the bound is the sum of the reference task loss and the distribution mismatch (and label mismatch), there is a trade-off determined by the value of τ . We illustrate the effect of the values of τ on the empirical and predicted transferability. As mentioned previously, we train both the classifier for the reference task h R and the target h T with an additional penalty on the gradient norm to make them τ -Lipschitz. In Fig. 13, we present results of varying the value of τ for the transfer to the Pets dataset with ImageNet as the reference task. For this experiment, we selected 37 random classes from ImageNet and only learned the transform A by keeping B fixed to a random permutation and C fixed to the uniform prior over reference task classes. We observe that the performance of linear fine-tuning degrades as we decrease the value of τ but explainability through the bound improves since the distribution mismatch term (dependent on τ ) decreases in the bound. However, making τ too small is not preferable since it leads to an increase in the first term of the bound (re-weighted reference task loss) increasing the bound overall. Moreover, it also leads to a degradation in the accuracy after linear fine-tuning. For our experiments, we use τ = 0 . 02 since it doesn't decrease the accuracy of fine-tuning significantly and leads to a small gap between empirical and predicted transferability.

0.01

0.02

0.05

0.1

0.2

0

5

10

Loss

RN18

0.01

0.02

0.05

0.1

0.2

0

5

10

RN50

Target Loss

Reweighted Reference Loss

D Details of the experiments

All codes are written in Python using Tensorflow/Pytorch and were run on an Intel(R) Xeon(R) Platinum 8358 CPU with 200 GB of RAM and an Nvidia A10 GPU. Implementation and hyperparameters are described below. Our codes can be found in the supplementary material. We report an average of three independent runs for experiments in Sec. 4.2 and 4.3.

D.1 Dataset details

In our work, we used the standard image classification benchmark datasets along with standard natural language processing datasets 1 .

Aircraft [35]: consists of 10,000 aircraft images belonging to 100 classes.

CIFAR-10/100 [30]: These datasets contain 60,000 images belonging to 10/100 categories. Additionally, we created two subsets of CIFAR100 with the first 25 (small CIFAR-100-S) and 50 (medium CIFAR-100-M) classes.

DTD[13]: consists of 5,640 textural images belonging to 47 categories.

Fashion MNIST [60]: consists of 70,000 grayscale images belonging to 10 categories.

Pets [43]: consists of 7049 images of Cats and Dogs spread across 37 categories.

ImageNet [17]: consists of 1.1 million images belonging to 1000 categories.

Yelp [63]: consists of 650,000 training and 50,000 test examples belonging to 5 classes.

Stanford Sentiment Treebank (SST-5) [63]: consists of 8,544 training and 2,210 test samples belonging to 5 classes.

AG News [63]: consists of 120,000 training and 7,600 test examples belonging to 4 classes

DBPedia [63]: consists of 560,000 training and 70,000 test examples belonging to 14 classes

D.2 Semantically similar classes for CIFAR-10 from ImageNet

For our experiments with CIFAR-10 in Sec. C.1, we selected the following semantically similar classes from ImageNet, { airliner, minivan, cock, tabby cat, ox, chihuahua, bullfrog, sorrel, submarine, fire engine }.

D.3 Additional experimental details

Details of the optimization problem in Eq. 4 and Alg. 1. In Step 5 of the algorithm, we use the network simplex flow algorithm from POT [23] to compute the optimal coupling. Since computing the Wasserstein distance over the entire dataset can be slow, we follow [16] and compute the coupling

over batches. Note that the base distance defined in Eq. 2 is non-differentiable. Thus, we use a differentiable approximation ˜ d (( z, y ) , ( z ′ , y ′ )) := d features ( z, z ′ ) + ν · ∥ e ( y ) -e ( y ′ ) ∥ 2 (with ν = 10 8 ) where ( z, y ) and ( z ′ , y ′ ) are samples from the domains R ′′′ and T and e ( · ) denotes the one-hot embedding of the labels in Step 5/6 . The first three terms in Step 6 of our algorithm correspond to the terms in the objective of Eq. 4 while the two additional terms are added to penalize the constraints of class prior matching P T ( y ) = BD and invertibility of the matrix A , respectively as required by Theorem 3. We use the softmax operation to ensure B and D are a valid probability matrix and vector.

In our experiments, in Sec. 4.1, we used pre-trained models available from Pytorch for ResNet18/50, along with publicly available pre-trained models provided in the official repositories of each training method. For each experiment, we subsample data from the ImageNet dataset belonging to the same number of classes as those present in the target dataset and use this data to train the linear layer on top of the representations extracted from the pre-trained model along with a gradient norm penalty (Reference task classifier). To speed up the experiments, we use only 10,000 points from the subsample of ImageNet for training the linear classifier and computing the transfer. For evaluation, we use a similar subsample of the validation dataset of ImageNet containing all the samples belonging to the subsampled classes. Fine-tuning on this dataset takes about 0.05 seconds per epoch for the task of transfer from ImageNet to Pets with the ResNet-18 model (we run the fine-tuning for a total of 5000 epochs).

Along with training the linear classifiers with a gradient norm penalty (with τ = 0 . 02 ), we standardize the features extracted from the pre-trained models to remove their mean (along each axis) and make them have a unit standard deviation (along each axis). While standardizing the features do not have a significant impact on the loss of the classifiers, including it makes it easier to match the distributions of the reference task and target data after transformations. Since our optimization problem transforms the reference task distribution to match the distribution of the target by solving the optimization problem in Eq. 4 by working on mini-batches, the size of the batch must be greater than the dimension of the representation space of the pre-trained encoder. E.g., for ResNet18 models which have a representation dimension of 512, we use a batch size of 1000 and for ResNet50 models which have a representation dimension of 2048, we use a batch size of 2500. Having a smaller batch size than the dimension could lead to a noisy gradient since for that batch the transformation can achieve a perfect matching, which may not generalize to data from other batches or unseen test data.

While computing the transformations, we apply the same augmentation (re-sizing and center cropping)/normalization to the training data as those applied to the test data. Along with this, we extract the features of the training and test data from the pre-trained model once and use these to train the linear layer. We note that this is done to save the computation time and better results could be obtained by allowing for extracting features after data augmentation for every batch.

Finally, for our experiments in Sec. 4.2, the encoders are trained end-to-end on the reference task. This is in contrast to our other experiments where the encoders are pre-trained and data from the reference task is only used for linear fine-tuning. Using these models, task-relatedness is evaluated by fine-tuning a linear layer using the data from the target task as well as the transformations computed by solving Eq. 4. We used τ = 0 . 2 here. We run the experiments with 3 random seeds and report the average results.

For our experiments in Sec. 4.3, we used the official code from [61] to compute the scores for NCE, Leep, and LogMe along with the official code of [51] for SFDA. For PACTran [19], we also use their official code with the PACTran-Gaussian method with N/K = 100 , β = 10 N,σ 2 = D/ 100 where N denotes the number of samples and K denotes the number of classes. This setting is similar to the PACTran-Gauss fix setting used in their work with the difference that we use N/K = 100 to use a sufficiently large number of samples, especially considering that all our other results for SbTE methods are computed on the full training set. For OTCE, we follow the official code and compute the recommended OT-based NCE score and OTCE score ( λ 1 = -0 . 0001 and λ 2 = -1 ) using 4000 randomly selected training samples from the two tasks. For the source task, we subsample the same number of classes as the target task and use. For the H-score, we use the official code to compute the H-score [6].

For estimating transferability by solving Eq. 5, we set λ =0.01 for all the experiments in Sec. 4.3.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes the paper's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in Sec. 5 and elaborated here.

Here we present some of the potential limitations of our work and discuss the avenues for future work. The analysis presented in the paper studies transferability using the popular cross-entropy loss similar to that analyzed by [56]. While this is important and practically useful, extending the analysis to other losses, primarily to the 0-1 loss, is an interesting future direction since in most classification tasks, accuracy is the metric of primary interest.

Another limitation of our work is the dependence of our bound on the Wasserstein distance. While it is a popular choice for analyzing performance in various distribution shift scenarios [52, 16, 50], it is difficult to estimate in practice due to its sensitivity to the number of samples and the dimension of the representation space. An analysis based on a different and easy-tocompute divergence measure might be more amenable for making transferability estimation via task-relatedness easier and faster.

We emphasize that even though we present some of the limitations of our current work above, these are by no means weaknesses of our work, which non-trivially extends the current understanding of the transfer learning setting through a rigorous analysis and achieves impressive performance on practical applications.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In App. A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In App. D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our codes can be found at https://github.com/akshaymehra24/ TaskTransferAnalysis .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In App. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: An average of three independent runs are reported.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In App. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There are no ethics concerns.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers transfer learning in a general cross-domain, cross-task setting. It introduces the concept of \"task-relatedness\" into measuring whether transfer learning can be successful. Specifically, task-relatedness consists of (1) a reference loss term for measuring the difference in class prior distribution between the target and source tasks; (2) a label mismatch term for measuring the conditional entropy between the label distributions; and (3) a distribution mismatch term  for measuring the difference in feature distributions. To make computing task-relatedness practical, the paper proposes several approximation schemes. It shows the metric is highly correlated with model transferrability in practice and can be used for selecting pretrained models.\n\nSTRENGTHS:\n1. Practical significance: The topic studied is important to the field of machine learning, as fine-tuning large-scale pretrained models gradually becomes the to-go option for model development. The idea of decomposing transferability into prior, feature, and label distribution differences is clean and interesting. \n2. Complete pipeline: the paper not only proposes task-relatedness as a transferability metric but also shows how it can be transformed into a training objective for effective transfer learning. This makes the paper more comprehensive and the proposed method more valuable.\n3. Presentation is clear and code for reproducing the experiments is provided\n\nWEAKNESSES:\n1. Limited novelty: the idea of using distribution distances to measure transferability is not new. As the related work section has pointed out, using the Wasserstein distance to estimate domain difference has been studied by OTDD [1]. Works like OTCE [2] also uses the proposed metric to do pretrained model selection. One argument the paper makes is that the proposed task-relatedness works without requiring target labels. While this does add some practical significance, the proposed method is a simple pseudo-labeling scheme without theoretical justification.\n2. Limited evaluation: the baselines considered in the experiments are not comprehensive, e.g., OTCE is not evaluated, and other recent OT-based works like [3] are not discussed. There's also no comparison for different pseudo-labeling schemes for the missing-label setting. Besides, there's no ablation study showing the contribution of each of the three terms proposed for computing task-relatedness.\n\n[1] Geometric Dataset Distances via Optimal Transport. David Alvarez-Melis. Nicolò Fusi.\n\n[2] OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations. Yang Tan, Yang Li, Shao-Lun Huang.\n\n[3] Wasserstein Task Embedding for Measuring Task Similarities. Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, Soheil Kolouri.\n\nQUESTIONS:\n1. Class-prior transformation assumes the reference task has more classes than the target task. Is this too strong an assumption? Does the proposed method applies to the settings where the target task has more labels?\n\n2. In Figure 3, the rankings of transferability for different target tasks are the same across all models. This is perhaps because the same reference dataset is used, and the experiments do not show the effect of varying model backbones. It would be better if the paper can evaluate multiple reference datasets on multiple models. (Figure 4 does look at different reference datasets but the model is fixed here.)\n\n3. Does the proposed metric applyy to non-classification tasks like regression or auto-regressive prediction?\n \n4. What's the max cross-domain distance the theory holds? I’d imagine that when the domain gap is large, the theory breaks. Have you studied that?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. It aims to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. Their analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. The experiments results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n(i) This work analyzes transferability for classification tasks and provides the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting.\n\n(ii) This work proposes an optimization problem to compute task-relatedness using a small amount of target labels and shows that it can even predict performance after end-to-end fine-tuning without requiring target labels.\n\n(iii) This work conduct multiple experiments to demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n(i) This work mentioned that it is the first to analyze transferability for classification tasks under cross-domain cross-task settings. However, there are multiple works [1-5] that have focused on the transferability and generalization of learned representations. Meanwhile, transferability is mainly evaluated by utilizing the performance differences between different domains and tasks with distribution shifts. This is also the case in previous works, but this is not the case for the introduction of motivation in this paper.\\\n[1] Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes.\\\n[2] Otce: A transferability metric for cross-domain cross-task representations.\\\n[3] On the theory of transfer learning: The importance of task diversity.\\\n[4] An information-theoretic approach to transferability in task transfer learning.\\\n[5] Task relatedness-based generalization bounds for meta learning.\n\n(ii) (Minor) Although this work focuses more on theoretical exploration, empirical verification on real-world datasets is also necessary, such as ablation studies on the three proposed terms, trade-off experiments (performance vs. efficiency vs. memory footprint) due to the introduction of additional computations, and comparisons with baselines on transferability. \n\n(iii) (Minor) The formatting and layout of some formulas and tables need to be adjusted to make the reading clearer and more intuitive while some equations with writing errors, such as Eq. 3, L527-528, the proof of Corollary 1. At the same time, due to the large number of formulas and physical quantities, it may be better to construct a table to explain the symbols and definitions.\n\n(iv) (Minor) The track chosen by the paper is not that matched to its content.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper analyzes transfer learning from the perspective of task relatedness, a model for transforming from a reference task (pretraining task) to the target task is proposed, which consist of prior transform, label transform, and feature transform. 
The task relatedness is then measured as the distribution mismatch of the transformed distribution and the target distribution.
Experiments have shown that the proposed task relatedness tightly upperbounds the transferrability to a range of architectures and datasets.

STRENGTHS:
1. The motivation and the derivation of the proposed task relatedness make senses to me.
2. The empirical results have shown the advantage of the proposed task relatedness.

WEAKNESSES:
1. The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.

QUESTIONS:
The appendix has answered most of my questions.
My only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence?


Minor:

A possible related work: Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective, ECCV 2022.

LIMITATIONS:
The paper has discussed its limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a way of assessing the transferability of representations from a pre-trained model to a target task by assessing the impact of the pre-trained model's representations on a reference task. By \"transforming\" the reference task into the target task, the paper produces a bound on the training loss of the target task\n\nSTRENGTHS:\n1. The paper presents both theoretical and empirical analysis of a problem\n2. Some of the provided experiments are reasonably comprehensive\n3. The idea of tackling the transferability problem by introducing a reference task seems novel AFAIK\n\nWEAKNESSES:\nMy main issues with this paper are as follows. \n1.  **Utility of the method** :  I was initially excited by the following sentence in the introduction of the paper -- *\"Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis\"*. However,  it seems that Algorithm 1 requires access to the target labels $\\mathcal{Y}_{T}$. This calls into question this stated advantage over the SbTE methods. Specifically, SbTE methods like TransRate and LEEP do not require  training on a reference task which not only comes with extra overhead but whose selection requires a-priori meta-intuition about which tasks are similar to the target task. Also, none of the empirical evaluations presented actually show that the proposed method is superior to SbTE approaches for choosing models form a zoo. Specifically, the paper presents hard to interpret correlation measures in Table 1 as proof of transferability -- when a superior test would be compare deltas in final training performance of the models the models that each method select from a zoo.\n2. **Correctness / Impact of claims**:  \n  [a]. From line 238 - 241: *However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.*. This seems to undercut the utility of the analysis presented. Why does this work ? Why even discuss these terms if they are effectively fixed ?  \n  [b]. It feels like the insight of *Highly related reference–target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks.* is just punting the problem of transferability estimation to task similarity estimation (and now calling on practitioner intuition to decide which tasks are highly related). Thus it does not seem that the authors are actually effectively solving the problem of transferability estimation -- they are just transforming it into a different problem.  \n  [c].  The authors claim  \"tight bounds / small gaps between task related and transferability\" but this is based on (imo) just eyeballing the graphs in  Figure 3 and 9. We are not provided with a rigorous way to interpret the gap. Eg -- a very simple way to artificially present a small gap is to just plot the the losses at random init of the classification head for the the reference and target tasks that have the same number of class labels. \n3. **Strength of underlying technical assumptions**:    \n  [a].  the K_r > K_t seems like a strong assumption that limits the methods applicability. What if the most similar task to the target task has K_r < K_t.   \n  [b]. Also as mentioned above in 238 - 241 it seems the label and class-prior matching component of the algorithm effectively has no influence on the final algo results.\n\n\nScore updated after discussion\n\nQUESTIONS:\n1. Work focuses  on classification based tasks -- could you discuss how this applies to generative tasks\n2. Can you discuss how faithful the estimates / method is when K_r < K_t  ?\n3. The analysis is for training error -- how is transferability estimation affected if there is overfitting to the reference task ?\n\nLIMITATIONS:\nN/A -- paper presents some limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}