Understanding the Transferability of Representations via Task-Relatedness

Authors: akshay  mehra
yunbei  zhang
jihun  hamm
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:13:05

Abstract: The growing popularity of transfer learning due to the availability of models pre-trained on vast amounts of data, makes it imperative to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. However, the exact conditions under which transfer learning succeeds in a cross-domain cross-task setting are still poorly understood. To bridge this gap, we propose a novel analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. Our analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks.Our experiments using state-of-the-art pre-trained models show the effectiveness of task-relatedness in explaining transferability on various vision and language tasks. The efficient computability of task-relatedness even without labels of the target task and its high correlation with the model's accuracy after end-to-end fine-tuning on the target task makes it a useful metric for transferability estimation. Our empirical results of using task-relatedness on the problem of selecting the best pre-trained model from a model zoo for a target task highlight its utility for practical problems.

Main paper: 
 introduction: 
 Transfer learning (TL) [42, 59] is a powerful tool for developing high-performance machine learning models, especially in current times when large models [45, 11, 12, 18] pre-trained on huge amounts of data are being fine-tuned for various downstream tasks. While large pre-trained models achieve impressive performance on downstream tasks even in the zero-shot inference setting [45] , their performance can often be improved by fine-tuning them on data from target tasks. However, our understanding of when representations from these models lead to classifiers that achieve high performance (i.e., high transferability) to downstream tasks is still lacking.Analytical works based on domain adaptation [8, 7, 52, 36, 32, 37, 38] can only explain cross-domain tasks (i.e., when only features/label priors change across tasks) but in the TL setting, label sets can also change (i.e., cross-task setting). Recently, [56] showed that the relatedness between the label sets of the two tasks measured using conditional entropy can explain the difference in their transferability. However, [56] focused only on the cross-task setting, and analysis for transferability in a general cross-domain cross-task setting is not addressed. Apart from these analytical works, another line of work focuses on proposing transferability metrics that correlate well with performance on downstream tasks after end-to-end fine-tuning. We refer to these works as score-based transferability estimation (SbTE) metrics [61, 55, 29, 40, 19, 51] . These works focus on developing scores for selecting a pre-trained model from a model zoo, that achieves the best transferability on a target task. While these works address a practical problem, they do not focus on providing an analysis of transferability. Figure 1 : Given a pre-trained encoder (e.g., CLIP [45] ), how does the performance after fine-tuning it on a reference task (e.g., Ima-geNet) relate to the performance after finetuning it on other tasks? Through a rigorous bound on transferability (Theorem 3) in terms of the relatedness between a reference and a target task, we show that tasks related to the reference task achieve provably better performance after fine-tuning.Thus, we first rigorously analyze the transferability of the representations in producing high-performing classifiers and propose a novel approach that studies transferability in terms of its relatedness to a reference task (see Fig. 1 ). This is in line with previous analytical works [7, 2, 56, 55] which studied the model's performance on target tasks in terms of the source task in different settings such as domain adaptation/generalization and recently TL. However, there's a crucial difference: we study transferability in terms of a reference task instead of the source task since it is impractical to assume the knowledge of the source task used to train large models such as CLIP [45] or GPT, commonly used for TL.Our approach works by transforming the distribution (and classifier) of a reference task, by transforming its class-prior distribution, label set, and feature space to obtain a new distribution that is similar to that of the target task (Fig. 2 ). Based on these transformations, we show that transferability can be provably explained (and is tightly upper bounded) using three interpretable terms. A weighted reference loss term appearing due to the class prior distribution difference between the tasks, a label mismatch term appearing as conditional entropy between the label distributions of the tasks, and a distribution mismatch term appearing as the Wasserstein distance between the transformed reference and target distributions (Theorem 3). We define task-relatedness as the sum of these three terms (a smaller value implies higher relatedness). We propose an optimization problem (Eq. 4) and an algorithm (Alg. 1) to learn the transformations to compute it. Using state-of-the-art (SOTA) pre-trained models, with different architectures, trained with various training methods on computer vision (CV) and natural language processing (NLP) tasks, we show that taskrelatedness achieves a small gap to transferability (Sec. 4.1). Our analysis also leads to new insights into learning in the TL setting such as to improve the transferability of an encoder on a downstream task, one can improve the encoder's transferability on related reference tasks (Sec. 4.2). This is particularly useful when practitioners intend to develop encoders that achieve high transferability to proprietary (and potentially inaccessible) datasets.We also demonstrate the utility of task-relatedness in estimating the accuracy of the model after end-to-end fine-tuning. While the TL setting assumes access to target labels, the high computational cost of end-to-end fine-tuning of a pre-trained model on a target task calls for developing metrics that are efficiently computable and highly correlated with end-to-end fine-tuning accuracy. To this end, we propose to use task-relatedness computed in the penultimate layer of the pre-trained model as our transferability estimation metric. To further improve the computational efficiency of task-relatedness we only measure the difference between the class-wise means and covariances of the distributions in lieu of the Wasserstein distance as required in Theorem 3. This enables the computation of task-relatedness with only the statistics of the reference/target tasks. Our empirical results (Sec. 4.3) attest that task-relatedness achieves a high correlation with the model's accuracy after end-to-end fine-tuning on the target task making it an effective metric for selecting a pre-trained model from a model zoo that achieves the best accuracy on the target task. Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis. Our main contributions are:• We rigorously analyze transferability for classification tasks. Our analysis, to the best of our knowledge, leads to the first upper bound on transferability in terms of task-relatedness in a cross-domain cross-task setting.• We propose an optimization problem to efficiently compute task-relatedness, using a small amount of target labels and show that it can even predict performance after end-to-end finetuning without requiring target labels.• Using SOTA models and CV/NLP tasks, we show that task-relatedness accurately predicts transferability and show that transferability to unseen tasks can be improved by improving transferability to known (related) tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract summarizes the paper's contributions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Transfer learning (TL): TL [42, 59, 18, 49, 46, 15, 14] has been studied widely and consists of various settings including transductive transfer, inductive transfer, and task transfer learning.The transductive setting also referred to as domain adaptation [8, 7] focuses on reducing the shift between two domains. The task transfer setting focuses on identifying the relationship between tasks, regardless of the model, to explain the transfer performance (see Appendix B for more details). Lastly, the inductive transfer setting focuses on using an inductive bias such as fine-tuning a pre-trained model (trained via adversarial training [48] , self-supervised learning [11, 10, 12] or by combining language and image information [45] ) to improve the performance on a target task. Our work focuses on the inductive transfer learning setting and proposes an upper bound on transferability of the representations of pre-trained models to downstream tasks.Analytical works for learning under distribution shifts: Prior works [8, 7, 52, 36, 32, 37, 38] analytically explained learning under distribution shifts using distributional divergence between the marginal distributions and a label mismatch term. However, these results are applicable under assumptions such as covariate or label shift which need not be satisfied in TL where both the data distribution and the label spaces can be different (see App. B for detailed comparison). Recently, [56] proposed an upper bound on transferability in a restrictive setting of the same features for both tasks, however, our analysis does not require such an assumption. Other works [9, 47, 41] analyzed the representation for the multi-task learning setting. These works showed that when tasks are weakly related, a single representation space (model) may not perform well for all tasks. However, the TL setting differs from both of these and our work aims to analyze transferability in this setting.Score-based transferability estimation (SbTE): These works [5, 40, 29, 61, 55, 39] use data from the target task and produce a score correlated with transferability. Such a score is useful for selecting the model from a model zoo that leads to the best transferability to a target task. [56] proposed the Negative Conditional Entropy (NCE) score that predicts transferability using the negative conditional entropy between labels of the tasks but requires the two tasks to have the same input instances. [6] estimates transferability by solving the HGR maximum correlation problem and using normalized Hscore, in the same setting as [56] . [40] proposed the LEEP score and computed NCE using soft (pseudo) labels for the target task from a pre-trained model. OT-CE [55] combined Wasserstein distance [3] and NCE whereas [5, 61] estimate likelihood and the marginalized likelihood of labeled target examples to estimate transferability. [33] proposes a model-agnostic approach that also relies on optimal transport to compute the distance between the tasks similar to OTDD [3] . In contrast, we focus on analyzing transferability in terms of task-relatedness theoretically along with demonstrating its effectiveness as a transferability estimation metric for the pre-trained model selection problem.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Discussed in Sec. 5 and elaborated here.Here we present some of the potential limitations of our work and discuss the avenues for future work. The analysis presented in the paper studies transferability using the popular cross-entropy loss similar to that analyzed by [56] . While this is important and practically useful, extending the analysis to other losses, primarily to the 0-1 loss, is an interesting future direction since in most classification tasks, accuracy is the metric of primary interest. Another limitation of our work is the dependence of our bound on the Wasserstein distance.While it is a popular choice for analyzing performance in various distribution shift scenarios [52, 16, 50] , it is difficult to estimate in practice due to its sensitivity to the number of samples and the dimension of the representation space. An analysis based on a different and easy-tocompute divergence measure might be more amenable for making transferability estimation via task-relatedness easier and faster. We emphasize that even though we present some of the limitations of our current work above, these are by no means weaknesses of our work, which non-trivially extends the current understanding of the transfer learning setting through a rigorous analysis and achieves impressive performance on practical applications. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
analysis of tl using task-relatedness: 
 Problem setting and notations: Let P R (x, y) and P T (x, y) denote the distributions of the reference and the target tasks, defined on X R ×Y R and X T ×Y T respectively. We assume that the feature spaces are common (X R = X T = X ) such as RGB images, but the reference label setY R = {1, 2, • • • , K R } and the target label set Y T = {1, 2, • • • , K T } can be entirely different.We assume the number of reference task classes (K R ) are greater than or equal to the number of target classes (K T ). In the TL setting, an encoder (feature extractor) g : X → Z is pre-trained on a dataset with or without labels depending on the training method (e.g., supervised vs. self-supervised). We denote the resultant push-forward distributions of R and T on the encoder output space as P R (z, y) and P T (z, y). With a fixed encoder g, a classifier (linear or non-linear), h(z) : Z → ∆, that outputs a probability vector is learned for the reference (h R ) and the target (h T ) separately, where ∆ R/T is a K R /K T simplex for R/T . The classifier h R = arg min h∈H E (z,y)∈P R [ℓ(h(z; g), y)] and h T = arg min h∈H E (z,y)∈P T [ℓ(h(z; g), y)] where H is the set of classifiers and ℓ(h(z), y) = -log(h(z) y ) is the cross-entropy loss. 𝑷 𝑹 ! (𝒛, 𝒚) 𝑷 𝑹 !! (𝒛, 𝒚) 𝑷 𝑹 !!! (𝒛, 𝒚) 𝑷 𝑻 (𝒛, 𝒚) 𝒉 𝑹 !! 𝒛 𝒉 𝑹 !!! 𝒛 𝒉 𝑻 𝒛 𝑹 ! 𝑹 !! 𝑹 !!!Figure 2 : : Overview of our task transformation model: A series of transformations are applied to the reference distribution P R (z, y) and classifier h R to produce the transformed distribution P R ′′′ and classifier h R ′′′ to explain transferability to the downstream target task. Class-prior transformation (R → R ′ ) changes the class prior of the reference distribution (e.g., an irrelevant Bee class in R now has smaller prior) followed by label set transformation (R ′ → R ′′ ) (e.g., to match {Lion, Wolf} with {Cat, Dog}), followed by feature space transformation (R ′′ → R ′′′ ) to match the feature distribution of the target task P T (z, y).Definition 1. (Transferability). Transferability of the representations from an encoder g on a target task T for classifiers in H is defined asE (z,y)∈P T [ℓ(h T (z; g), y)].In the next section, we show the analysis with H as the class of linear classifiers for ease of explanation and discuss its extension to non-linear classifiers in App. A.5. Proofs for Sec. 3 are in App. A.The reference and the target tasks share the same encoder but do not share label sets or data distributions. Therefore, to relate the two tasks, we propose a chain of three simple transformations: 1) prior transformation (from R to R ′ ), 2) label transformation (from R ′ to R ′′ ), and 3) feature transformation (from R ′′ to R ′′′ ). The R ′ , R ′′ , R ′′′ are intermediate domain names after each of the transformations are applied. The corresponding classifier in each domain is denoted by h R ′ , h R ′′ , and h R ′′′ as illustrated in Fig. 2 . The distribution after the transformations (P R ′′′ ) has the same feature Z R ′′′ = Z T = Z and label sets Y R ′′′ = Y T as the target task T , and consequently, the loss of the transformed classifier h R ′′′ and the target classifier h T can be related.Class-prior transformation (R → R ′ ): Since the reference task has more classes than the target task (K R ≥ K T ), many of the reference task classes are likely irrelevant for transfer to the target classes, e.g., while transferring from ImageNet to CIFAR10, only a small portion of ImageNet classes are relevant to CIFAR10 classes. The prior transformation accounts for the relative importance of the reference classes. This is illustrated in Fig. 2 , where changing the class prior of R reduces the prior of the Bee class and increases the priors of Wolf and Lion classes (shown by the changed size of classes Wolf and Lion in R ′ ). While transforming the prior of R, we keep the conditional distribution and the classifier the same i.e., P R ′ (z|y) = P R (z|y) and h R ′ (z) = h R (z). Lemma 1 in App. A.2.1 shows that the expected loss of the classifier h R on R ′ is a re-weighted version of the loss of h R on R.Label transformation (R ′ → R ′′ ): Next, we use a label transformation to match the label sets of the new domain R ′′ and that of the target domain. To this end, we specify the conditional distributionB ij := P (y R ′′ = i|y R ′ = j) (B ij ∈ [0, 1], ∀ i, j, i B ij = 1, ∀j).The label y R ′′ of an example from the domain R ′′ is obtained via BP (y R ′ ). This generative process doesn't require the feature, i.e., P R ′′ (y R ′′ |y R ′ , z) = P R ′′ (y R ′′ |y R ′ ). B with sparse entries (i.e., only one entry of a column is 1) models a deterministic map from Y R to Y T ; B with dense entries models a weaker association. This process is illustrated in Fig. 2 which shows the map from {Bee, Wolf, Lion} ⊂ Y R ′ to {Dog, Cat} ⊂ Y T after using B. Under this model, a reasonable choice of classifier for R ′′ is h R ′′ (z) = Bh R ′ (z). Lemma 2 in App. A.2.2 shows that the expected loss of h R ′′ depends on the loss of h R ′ and the conditional entropy between the label sets of the tasks R ′ and R ′′ and Corollary 1 shows the conditions for optimality of h R ′′ .Feature transformation (R ′′ → R ′′′ ): The final step involves changing the feature space of the distribution R ′′ . We apply an invertible linear transformation A to the distribution in R ′′ to obtain the new distribution R ′′′ . After the transformation, the classifier associated with the new domainR ′′′ is h R ′′′ (z) = h R ′′ (A -1 (z)). This is illustrated in Fig. 2 after feature transform using A. Lemma 3 in App. A.2.3 shows that a linear transform of the space and classifier does not incur any additional loss and Corollary 2 shows that the optimality of h R ′′ implies optimality of h R ′′′ . Using these, we get Theorem 1 by defining conditional entropy as followsH(Y R ′′ |Y R ′ ) = - y R ′ ∈Y R ′ y R ′′ ∈Y R ′′ P R ′ (y R ′ )B y R ′′ ,y R ′ log(B y R ′′ ,y R ′ ).(1)Theorem 1. Let C := P R ′ (y) P R (y) K R y=1be a vector of probability ratios , B be a K T × K R matrix withB ij = P (y R ′′ = i|y R ′ = j), A : Z → Z be an invertible linear map of features. Let the classifiers h R ′ (z) := h R (z), h R ′′ (z) := Bh R ′ (z), h R ′′′ (z) := h R ′′ (A -1 (z)).Assuming ℓ is the cross-entropy loss, we haveE P R ′′′ (z,y) [ℓ(h R ′′′ (z), y)] ≤ E P R (z,y) [C(y)ℓ(h R (z), y)]Re-weighted reference loss+ H(Y R ′′ |Y R ′ ) Label mismatch.Theorem 1 provides an upper bound on the loss of the final transformed classifier/distribution in terms of the loss of the reference classifier/distribution. The re-weighted reference loss shows that the performance of the transformed classifier on the new domain is linked to the label-wise re-weighted loss of the reference classifier on R. This implies that one can use only the relevant reference classes to contribute to the bound. The label mismatch term shows that the performance of the distribution R ′′′ and R depends on the conditional entropy H(Y R ′′ |Y R ′ ; B) between the label distributions of the domain R ′′ and R ′ . A high value of H implies that the labels of the reference task are unrelated leading to lower transferability, whereas a low H implies higher transferability. Corollary 3 in App. A.2.4 shows when the bound in Theorem 1 becomes equality.3.2 Distribution mismatch between P R ′′′ and P T After the three transformations, the transformed reference P R ′′′ (z, y) can be compared with the target P T (z, y). However, these are only simple transformations and P R ′′′ cannot be made identical to P T in general. This mismatch can be measured by the Wasserstein or Optimal Transport distance [44, 58] . Since our goal is to match two joint distributions defined on Z × Y we useEQUATIONwith z, z ′ ∈ Z and y, y ′ ∈ Y as our base distance [53] to define the (type-1) Wasserstein distanceEQUATIONUsing Eq. 2, the Wasserstein distance between the joint distributions is the weighted sum of the Wasserstein distance between conditional distributions (P (z|y)) (Lemma 4 in App. A). Theorem 2 below explains the gap between the losses due to the distribution mismatch. Assumption 1. 1) The composition of the loss function and the classifier ℓ• h is a τ -Lipschitz function w.r.t to ∥ • ∥ 2 norm, i.e., |ℓ(h(z), y) -ℓ(h(z ′ ), y)| ≤ τ ∥z -z ′ ∥ 2 for all y ∈ Y, z, z ′ ∈ Z where h ∈ H. 2) P T (y) = P R ′′′ (y).The assumption 2), can be satisfied since we have full control on the prior P R ′′′ (y) via B and C. Theorem 2. Let the distributions T and R ′′′ be defined on the same domain Z × Y and assumption 1 holds, thenE P T (z,y) [ℓ(h(z), y)] -E P R ′′′ (z,y) [ℓ(h(z), y)] ≤ τ W d (P R ′′′ , P T ) Distribution mismatch, with d as in Eq. 2.Theorem 2 shows that when ℓ • h is τ -Lipschitz then the performance gap between the R ′′′ and T is bounded by the type-1 Wasserstein distance between the two distributions. The Lipschitz coefficient of the composition can be bounded by τ , by penalizing the gradient norm w.r.t z at training time. Thus, for linear fine-tuning, we train the classifiers h R and h T with an additional gradient norm penalty max{0, ∥∇ z ℓ(h(z), y)∥ 2 -τ } to make them conform to the Lipschitz assumption (see App. C.3). Note that constraining the Lipschitz constant restricts the hypothesis class. The trade-off between the Lipschitz constant and the performance of h is empirically evaluated in App. C.3.1. Figure 3 : Task-relatedness (decomposed into its components) produces a small gap to transferability (blue bars). As the task-relatedness between the reference (ImageNet (for CV), DBPedia (for NLP)), and the target tasks (x-axis) increases, the transferability improves. (Note: the label mismatch term is zero in our figures as B is fixed to a sparse matrix, see Sec. 3.4.)Here, we combine the results obtained in Theorem 1 and Theorem 2. The final bound proposed in Theorem 3 is one of our main contributions which explains transferability as a sum of three interpretable and measurable gaps. Theorem 3. Let ℓ be the cross-entropy loss, then under assumptions of Theorems 1 and 2,E P T (z,y) [ℓ(h T (z), y)] ≤ E P R (z,y) [C(y)ℓ(h R (z), y)]Re-weighted reference loss+ H(Y R ′′ |Y R ′ ) Label mismatch + τ W d (P R ′′′ , P T ).The theorem shows that transferability can be decomposed into the loss incurred while transforming the class prior distribution, label space, and feature space of the reference distribution (first two terms) and the residual distance between the distribution obtained after transformations and the actual target distribution (last term). Based on the terms in the upper bound we define task-relatedness as follows. Definition 2. (Task-relatedness). The relatedness between a target and a reference task is defined asE P R (z,y) [C(y)ℓ(h R (z), y)] + H(Y R ′′ |Y R ′ ) + τ W d (P R ′′′ , P T ).A smaller value of the task-relatedness measure implies higher relatedness of the reference and the target tasks. In particular, when the target task is a transformation of the reference task then there exist transformations A, B, and C such that the distribution R ′′′ perfectly matches the distribution of the target task (i.e., W d (P R ′′′ , P T ) = 0). Moreover, when labels are deterministically related (Corollary 3) our bound becomes an equality.Lastly, while we presented an analysis for linear fine-tuning here (for simplicity of presentation), our bounds hold for non-linear classifiers and non-linear feature transformations as well (see App. A.5).The optimization problem for learning the transformations A, B, and C to compute task-relatedness in Theorem 3 is presented below. We use two new variables: inverse of the transformation A, denoted by Ā := A -1 and a transformed reference prior distribution denoted by D(y) := C(y)P R (y). EQUATIONAlg. 1 shows how we solve Eq. 4 (see App. D for additional details of the algorithm). Fig. 8 = P R (y), random B ∈ R K T ×K R 1: Randomly sample n R points (z i R , y i R ) ∼ (Z R , Y R ) as per the class prior D. 2: if Y T is available then 3: Randomly sample n T points (z j T , y j T ) ∼ (Z T , Y T ). 4: else 5:Randomly sample n T points (z j T ) ∼ (Z T ).# Compute pseudo-labels for the target samples z T .7:y j T = arg max y∈Y T Bh R (A -1 z T ) for j = 1, • • • , n T . 8: end if 9: Compute (z i R ′′′ , y i R ′′′ ) = (Az i R , arg max y Be(y i R )), for i = 1, • • • , n R . 10: Assign Y R ′ := Y R and Y R ′′ := Y T . 11:Compute the optimal coupling π * between the distributions R ′′′ and T by minimizingEQUATIONπ∈Π(P R ′′′ ,P T ) i,j π ij d((z i R ′′′ , y i R ′′′ ), (z j T , y j T )) s.t. j π ij = 1 n R ∀i, i π ij = 1 n T ∀j.12: Using π * , solve for A, Ā, B, D using mini-batch SGDmin A, Ā,B,D i,j π * i,j d((z i R ′′′ , y i R ′′′ ), (z j T , y j T )) + 1 n R i D(y i ) P R (y i ) ℓ(h R (z i R ), y i ) + H(Y R ′′ |Y R ′ ) + ∥P T (y) -BD∥ 2 2 + (∥A Ā -I∥ F + ∥ ĀA -I∥ F ).13: Repeat 1 -12 until convergence.the reference task has classes semantically related to the target task, Alg. 1 learns transformations that achieve the smallest gap to transferability. However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: In App. A.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
empirical analysis: 
 Here, we empirically demonstrate the effectiveness of task-relatedness in explaining transferability in various settings. We present additional results in App. C and dataset/experimental details in App. D. Our codes can be found at https://github.com/akshaymehra24/TaskTransferAnalysis.Task-relatedness tightly upper bounds transferability across various architectures, pretraining methods, and datasets. We demonstrate this by using various pre-trained models with architectures such as Vision Transformers (ViT) [20] , ResNet-18/50/101/152 [27] , DistilRoBERTa [34] trained with various pretraining methods including supervised training, adversarial training [48] , SimCLR [11] , MoCo [26] , SwAV [10] , and MAE [25] . We also consider a wide range of target datasets including, CIFAR10/100, Aircraft, Pets, DTD, AG-News, Yelp-5, and SST-5 whose details are in App. D. For this experiment, we fix the reference task to be ImageNet [17] for image classification and to DBPedia for sentence classification tasks and use Alg. 1 to estimate task-relatedness. The results in Fig. 3 and 9 (in the Appendix) show that our bound achieves a small gap to actual transferability.As the task-relatedness between the reference and the target tasks improves, transferability also improves showing that task-relatedness and transferability are strongly correlated. Task-relatedness is also strongly correlated with the accuracy of the end-to-end fine-tuned classifiers on the target task. In Fig. 11 (in the Appendix), we show high Pearson correlation coefficients (≥ -0.57) for task-relatedness and accuracy after fully fine-tuning various pre-trained encoders using data from various target tasks.Highly related reference-target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks. To understand how a reference task affects task-relatedness and eventually transferability, we consider two experiments using convolutional and CLIP-trained models with various character recognition tasks such as MNIST, Fashion-MNIST (FMNIST), SVHN, MNIST-M, and USPS. Of these datasets, SVHN and MNIST-M contain colored images while the rest contain gray-scale images. In the first experiment, we train convolutional models on MNIST, FMNIST, and USPS and measure pairwise transferability. Here we use the reference task to be the same task as that used for training the models. The results in Fig. 4(a) show that transferability to those target tasks is higher for which task-relatedness metric's value is smaller. Specifically, USPS achieves the best transferability (1.23) and the smallest task-relatedness (2.80) when the reference task is MNIST. This is attributed to both datasets containing gray-scale images of digits. On the other hand, when the reference task is unrelated to the target task i.e., the task-relatedness value is high, transferability suffers, e.g., when the reference task is MNIST and the target task is FMNIST. Results in App. C.2.2 show similar results for the sentence classification task.The gap between task-relatedness and transferability is smaller when a reference task performs well with a given encoder. Here we use MNIST and SVHN as two reference tasks and compute the task-relatedness and transferability with USPS and MNIST-M as target tasks, using CLIP (Vit B32) model. A linear classifier trained on top of the embeddings from the CLIP model achieves ≈98% accuracy on MNIST but only ≈61% accuracy for SVHN. Due to this, transferability (USPS:2.02, MNIST-M:2.20) explained using task-relatedness with MNIST as the reference task (USPS:2.46, MNIST-M:2.48) is better than that computed using SVHN (USPS:2.66, MNIST-M:2.65) as the reference, even though MNIST-M is intuitively more similar to SVHN (as both contain colored images of digits). This is evident from the results of PE (Pre-trained Encoder) in Fig. 4(b) .Improving the performance of an encoder on a reference task improves transferability to other related (potentially unseen) tasks. To show this we fully fine-tune the CLIP encoder on MNIST and SVHN tasks, increasing the accuracy of the classifiers for both MNIST and SVHN to 99% and 95%, respectively. Using the representations from these new encoders, we find that the transferability of both related target tasks improves along with task-relatedness (see FFE results in Fig. 4(b) ). Here, Table 1 : Task-relatedness achieves high (negative) Pearson correlation to the accuracy after end-to-end fine-tuning for various tasks. For NCE [56] , Leep [40] , LogMe [61] , SFDA [51] , OT-NCE, OTCE [55] , and H-score [6] positive correlation is better whereas for PACTran [19] and task-relatedness (ours) negative correlation is better. we see that task-relatedness for MNIST-M and USPS is the best when the reference task is SVHN and MNIST, respectively, aligning with our intuition of semantic relatedness between these tasks. This also suggests that transferability on other related tasks can be improved by fully fine-tuning the encoder on these reference tasks. Thus, in scenarios where target tasks are private (such as proprietary Chest X-rays), an encoder trained to work well on related tasks (such as publicly available Chest X-rays) is bound to achieve good transferability.In this section, we show an efficient way of computing task-relatedness which enables its use for estimating transferability after end-to-end fine-tuning. While Alg. 1, accurately estimates taskrelatedness by minimizing the bound in Eq. 3, it could be inefficient due to the requirement of computing and minimizing the Wasserstein distance between distributions at every epoch. Thus, to make the computation efficient, we replace the Wasserstein distance computation in step 11 and 12 of Alg. 1, with mean and covariance matching terms. Specifically, we define the distance between two distributions R ′′′ and T asEQUATIONwhere µ R ′′′ /T :=1 n R ′′′ /T z∈P R ′′′ /T z, Σ R ′′′ /T := 1 n R ′′′ /T z∈P R ′′′ /T (z -µ R ′′′ /T ) T (z -µ R ′′′ /T ),and λ is a regularization coefficient. Using Γ(R ′′′ , T ) in place of W d (R ′′′ , T ), makes the computation of task-relatedness by learning transformations A, B, and C significantly more efficient.Task-relatedness is an effective metric for the pre-trained model selection problem. The goal of this problem is to find a pre-trained model from a model zoo that achieves the best accuracy on a given target task after end-to-end fine-tuning of the model using labeled target data. Since end-to-end fine-tuning is costly (takes almost a day to fully fine-tune a single model on a single target task as shown by [61] ), an effective transferability metric is significantly more efficient to compute and is correlated well with the accuracy after end-to-end finetuning. Using 5 different pre-trained models (supervised ResNet-50/101/152, adversarially pre-trained [48] ResNet50 with ϵ ∈ {0.1, 1}) and ImageNet as the reference task, we show in Table 1 , that task-relatedness achieves a high correlation with the accuracy after end-to-end fine-tuning on the target task. Our results also highlight the instability of various popular SbTE metrics, such as LogMe [61] and NCE [56] which can produce a high negative correlation, and those of PACTran [19] which achieve low correlation values on complex datasets. In comparison, task-relatedness consistently achieves a good correlation for various target tasks. Computationally, it takes a mere 3-4 minutes to learn the transformations to compute task-relatedness, providing a significant computation advantage over end-to-end fine-tuning. We also show that task-relatedness remains highly correlated with end-to-end fine-tuning accuracy even with a limited amount of labeled data from the target task as shown in Fig. 5 unlike other SbTE metrics.Next, we show that task-relatedness can even be estimated without using labels from the target task. 2 show that our task-relatedness estimated via pseudo-labeled target data still achieves a high correlation to transferability on most datasets. For datasets such as Pets and DTD, where transforming the reference task classifier produces high accuracy on the target task, the difference between the pseudo and true labels is small. Consequently, the difference in the correlations with pseudo and true labels is also small. Thus, when the reference and target tasks are related, transferability can be estimated accurately without requiring labels of the target task, showing that task-relatedness is an effective metric even for unsupervised transferability estimation.y pseudo T = arg max y∈Y T Bh R (A -1 (z T )). Results in TableQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: In App. D. 
conclusion: 
 We analyzed TL in terms of the relatedness between the target and a reference task. Our analysis works by transforming the distribution of a reference task to match that of the target. Using this we proved an upper bound on transferability, defined as task-relatedness, consisting of three interpretable terms, namely, the re-weighted reference task loss, label mismatch, and distribution mismatch. We proposed an algorithm to compute task-relatedness and demonstrated its effectiveness at accurately predicting transferability (even without target labels) with SOTA models. Moreover, the high correlation of task-relatedness with accuracy after end-to-end fine-tuning and its efficient computability, makes it an effective metric for transferability estimation.Limitations. We studied transferability using the cross-entropy loss and used Wasserstein distancebased distribution shift analysis due to their popularity. However, due to accuracy being the primary metric of interest in classification tasks and the difficulty of computing the Wasserstein distance with limited samples in a high dimensional representation space, extending the analysis to 0-1 loss and other divergence measures are important directions which are not addressed here and are left for future works. 
acknowledgment: 
 Proof.E P R ′ (z,y) [ℓ(h R ′ (z), y)] = E P R ′ (z,y) [ℓ(h R (z), y)] = y∈Y R P R ′ (y)E P R ′ (z|y) [ℓ(h R (z), y)] = y∈Y R P R (y) P R (y) P R ′ (y)E P R ′ (z|y) [ℓ(h R (z), y)] = y∈Y R P R (y)E P R ′ (z|y) [C(y)ℓ(h R (z), y)] = y∈Y R P R (y)E P R (z|y) [C(y)ℓ(h R (z), y)] (since P R (z|y) = P R ′ (z|y) by construction) = E P R (z,y) [C(y)ℓ(h R (z), y)]. A.2.2 Label transformation (R ′ → R ′′ ) Lemma 2. Let B be a K T × K R matrix with B ij = P (y R ′′ = i|y R ′ = j) and h R ′′ (z) := Bh R ′ (z)and ℓ be the cross-entropy loss. Then,E P R ′′ (z,y) [ℓ(h R ′′ (z), y)] ≤ E P R ′ (z,y) [ℓ(h R ′ (z), y)] + H(Y R ′′ |Y R ′ ), where H(Y R ′′ |Y R ′ ) is the conditional entropy (-y R ′ ∈Y R ′ y R ′′ ∈Y R ′′ P R ′ (y R ′ )B y R ′′ ,y R ′ log(B y R ′′ ,y R ′ )).Proof. Note that P (z) := P R ′ (z) = P R ′′ (z) by construction.E P R ′′ (z,y) [ℓ(h R ′′ (z), y)] = E P (z,y ′′ ) [ℓ(h R ′′ (z), y ′′ )] = E P (z) E P (y ′′ |z) [ℓ(h R ′′ (z), y ′′ )] = E P (z) y ′′ y ′ P (y ′′ , y ′ |z)(ℓ(h R ′′ (z), y ′′ )) (since y ′ ∈ Y R ′ ) = E P (z) E P (y ′′ ,y ′ |z) [ℓ(h R ′′ (z), y ′′ )] = E P (z) E P (y ′ |z) E P (y ′′ |y ′ ) [ℓ(h R ′′ (z), y ′′ )] (since P (y ′′ |y ′ , z) = P (y ′′ |y ′ )) = E P (z) E P (y ′ |z) [ y ′′ ∈Y R ′′ ℓ(h R ′′ (z), y ′′ )B y ′′ ,y ′ ] (since B y ′′ ,y ′ = P (y ′′ |y ′ )) = E P (z,y ′ ) [ y ′′ ∈Y R ′′ ℓ(Bh R ′ (z), y ′′ )B y ′′ ,y ′ ].Since the loss ℓ is the cross-entropy loss, we haveℓ(Bh R ′ (z), y ′′ ) = -log( j∈Y R ′ B y ′′ ,j h j R ′ (z)) ≤ -log(B y ′′ ,y ′ h y ′ R ′ (z)) = -log(B y ′′ ,y ′ ) -log(h y ′ R ′ (z)).Our Corollary 2 below shows that the optimal softmax classifier for domain R ′′ remains optimal for domain R ′′′ too. Corollary 2. Let h R ′′ be the optimal softmax classifier in domain R ′′ then under the assumptions of Lemma3, h R ′′′ (z) = h R ′′ (A -1 (z)) is the optimal softmax classifier in domain R ′′′ . Proof. When h R ′′′ (z) = h R ′′ (A -1 (z)), min h R ′′ E P R ′′ (z,y) [ℓ(h R ′′ (z), y)] = min h R ′′′ E P R ′′′ (z,y) [ℓ(h R ′′′ (z), y)] by Lemma 3, hence if h R ′′ is optimal for R ′′ then so is h R ′′′ for the domain R ′′′ . A.2.4 Three transformations combined (R → R ′′′ ) Theorem 1. Let C := P R ′ (y) P R (y) K R y=1be a vector of probability ratios , B be a K T × K R matrix withB ij = P (y R ′′ = i|y R ′ = j), A : Z → Z be an invertible linear map of features. Let the classifiers h R ′ (z) := h R (z), h R ′′ (z) := Bh R ′ (z), h R ′′′ (z) := h R ′′ (A -1 (z)).Assuming ℓ is the cross-entropy loss, we haveE P R ′′′ (z,y) [ℓ(h R ′′′ (z), y)] ≤ E P R (z,y) [C(y)ℓ(h R (z), y)]Re-weighted reference task loss+ H(Y R ′′ |Y R ′ ) Label mismatch .Proof. Proof. Since y R ′′ := σ(y R ′ ) := arg max y∈Y T (Be(y R ′ )) y we haveEQUATIONE P R ′′ [ℓ(h R ′′ (z), y R ′′ )] = E P (z,y ′′ ) [ℓ(Bh R ′ (z), y ′′ )] = y ′′ ∈Y R ′′ P (y ′′ )E P (z|y ′′ ) [ℓ(Bh R ′ (z), y ′′ )] = y ′ ∈Y R ′ P (σ(y ′ ))E P (z|σ(y ′ )) [ℓ(Bh R ′ (z), σ(y ′ ))] = y ′ ∈Y R ′ P (y ′ )E P (z|y ′ ) [ℓ(h R ′ (z), y ′ )] = E P R ′ (z,y) [ℓ(h R ′ (z), y)].The second last equality follows due to the symmetry of cross-entropy loss, i.e., ℓ(h, y) = -log h y = -log Bh σ(y) = ℓ(Bh, σ(y)).Therefore, we have Proof. Let ω * y denote the optimal coupling for the conditional distributions (P U (z|y), P Q (z|y)) for y ∈ Y and π * denote the the optimal coupling for the joint distributions (P U (z, y), P Q (z, y)). Then, under the definition of our base distance d, π * ((z, y), (z ′ , y ′ )) = 0 when y ̸ = y ′ i.e. no mass from the distribution U belonging to class y can be moved to the classes y ′ ̸ = y of the distribution Q when the class priors of U and Q are the same. Moreover, since ij (ω * y ) ij = 1 and {i,j:yi=y ′ j =k} π * ij = P (y = k) for k ∈ Y we have π * ((z, y), (z ′ , y ′ )) = ω * y (z, z ′ )P (y)1 y=y ′ for every y, y ∈ Y.EQUATIONThen, we can show that the total Wasserstein distance between the joint distributions can be expressed as the sum of conditional Wasserstein distances, as followsW d (P U (z, y), P Q (z, y)) = y,y ′ π * ((z, y), (z ′ , y ′ ))d((z, y), (z ′ , y ′ ))dzdz ′ = y,y ′ π * ((z, y), (z ′ , y ′ ))(∥z -z ′ ∥ 2 + ∞ • 1 y̸ =y ′ )dzdz ′ = y,y ′ ω * y (z, z ′ )P (y)1 y=y ′ (∥z -z ′ ∥ 2 + ∞ • 1 y̸ =y ′ )dzdz ′ = y,y ′ ω * y (z, z ′ )P (y)1 y=y ′ ∥z -z ′ ∥ 2 dzdz ′ (since 1 y=y ′ • 1 y̸ =y ′ = 0) = y,y ′ P (y)1 y=y ′ ω * y (z, z ′ )∥z -z ′ ∥ 2 dzdz ′ = y P (y) ω * y (z, z ′ )∥z -z ′ ∥ 2 dzdz ′ = y P (y)W ∥•∥2 (P U (z|y), P Q (z|y)).Theorem 2. Let the distributions T and R ′′′ be defined on the same domain Z × Y and assumption 1 holds, thenE P T (z,y) [ℓ(h(z), y)] -E P R ′′′ (z,y) [ℓ(h(z), y)] ≤ τ W d (P R ′′′ , P T ) Distribution mismatch, with d as in Eq. 2.Proof.E P T (z,y) [ℓ(h(z), y)] -E P R ′′′ (z,y) [ℓ(h(z), y)] = E P T (y) E P T (z|y) [ℓ(h(z), y)] -E P R ′′′ (y) E P R ′′′ (z|y) [ℓ(h(z), y)] = E P T (y) [E P T (z|y) [ℓ(h(z), y)] -E P R ′′′ (z|y) [ℓ(h(z), y)]] (since P T (y) = P R ′′′ (y)) ≤ E P T (y) [ sup ℓ ′ •h ′ ∈τ -Lipschitz E P T (z|y) [ℓ ′ (h ′ (z), y)] -E P R ′′′ (z|y) [ℓ ′ (h ′ (z), y)]] = E P T (y) [τ W ∥•∥2 (P T (z|y), P R ′′′ (z|y))] (Kantorovich -Rubinstein duality) = τ W d (P R ′′′ , P T ) (Lemma 4).A.4 Final bound (Sec. 3.3) Theorem 3. Let ℓ be the cross entropy loss then under the assumptions of Theorems 1 and 2 we have,E P T (z,y) [ℓ(h T (z), y)] ≤ E P R (z,y) [C(y)ℓ(h R (z), y)]Re-weighted reference task loss+ H(Y R ′′ |Y R ′ ) Label mismatch + τ W d (P R ′′′ , P T ) Distribution mismatch . Proof. Let ℓ • h T , ℓ • h R , and ℓ • h R ′′′ be τ -Lipschitz (from Assumption 1).E P T (z,y) [ℓ(h T (z), y)] ≤ E P T (z,y) [ℓ(h R ′′′ (z), y)] (Optimality difference) ≤ E P R ′′′ (z,y) [ℓ(h R ′′′ (z), y)] + τ W d (P R ′′′ , P T ) (Theorem 2) ≤ E P R (z,y) [C(y)ℓ(h R (z), y)] + H(Y R ′′ |Y R ′ ) + τ W d (P R ′′′ , P T ) (Theorem 1).In our experiments, we enforce the τ -Lipschitz constraint for ℓ • h R and ℓ • h T and verify that the Lipschitz constant of ℓ • h R ′′′ remains close to τ within tolerance. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To extend our analysis to non-linear classifiers/transformations, we allow A : Z → Z to be a nonlinear map and the classifiers h ∈ H non_linear to be also non-linear (such as multi-layer perceptron).In addition to the Assumption 1 (1), which requires ℓ • h R ′′′ to be τ -Lipschitz, we also require that h R ′′′ belongs to the same class H non_linear as h T and h R for any A. This holds for example when h is linear and A is also linear or when h is a multilayer perceptron and A is linear. With this additional assumption, all of our proofs work for any linear/non-linear transformation of the feature space, without any change. Thus, Theorem 1, Theorem 2 and Theorem 3 hold for non-linear classifiers as well. With these extensions, our bounds can be used to explain transferability even with non-linear classifiers.Distributional divergence-based analyses of learning with distribution shifts (under same feature and label sets): Here we review some of the previous works that analyzed the problem of learning under distribution shifts in terms of distributional divergences such as the Wasserstein distance. These analyses apply when the feature and label spaces remain the same between the original and the shifted distribution.Early works [8, 52, 36] showed that the performance on a shifted distribution (target domain) can be estimated in terms of the performance of the source domain and the distance between the two domains' marginal distributions and labeling functions. Specifically, [8] showed that thatE T (h, f T ) ≤ E S (h, f S ) + d 1 (P S , P T ) + min{E P S [|f S (z) -f T (z)|], E D T [|f S (z) -f T (z)|]},where d 1 denotes the total variation distance, f : Z → [0, 1] denotes the labeling function, h : Z → {0, 1} denotes the hypothesis andE P (h, f ) := E z∼P [|h(z) -f (z)|]denote the risk of the hypothesis h. A follow up work [52] , showed a similar result using type-1 Wasserstein distance for all K-Lipschitz continuous hypotheses i.e.,E T (h, f T ) ≤ E S (h, f S ) + 2K • W 1 (P S , P T ) + λ,where λ is the combined error of the ideal hypothesis h * that minimizes the combined errorE S (h, f S ) + E T (h, f T ).Another recent work [32] used a target transformation-based approach and Wasserstein distance to quantify learning in the presence of data and label shifts. Other works [31, 50] also presented an analysis based on Wasserstein distance to understand how the accuracy of smoothed classifies and robustness change in the presence of distribution shifts. Compared to these works the bound proposed in Theorem 2 considers cross-entropy loss (which is a popular choice of the loss function in the classification setting) and uses a joint feature and labels Wasserstein distance rather than only marginal Wasserstein distance. These differences make the bound proposed in Theorem 2 useful in the analysis of transfer learning than those proposed in previous works when we have access to labeled target domain data.Comparison with [56] : The closest work to ours is that of [56] , which showed that transferability to a target task can be related to transferability to another task (source task in their work) and the label mismatch between the two tasks. However, the bound is proposed in a restrictive setting when both the tasks have the same features but different labels (i.e. same images labeled differently between the two tasks). In this setting, [56] , showed that transferability is upper bounded by loss of the source classifier on the source task and the conditional entropy (CE) of the label sets of the two tasks. We significantly extend this analysis to general tasks which is the most commonly used setting in practice (e.g., our analysis allows us to study transfer learning from ImageNet with 1000 classes to CIFAR-100 with 100 unrelated classes, where both tasks have different images). In this setting, our main result in Theorem 3, shows that transferability involves additional terms such as the distribution mismatch term (in the form of Wasserstein distance), the prior mismatch term (in the form of re-weighted source loss) and the conditional entropy between the label sets. Moreover, the bound proposed by [56] is a special case of our bound with C being the vector of all ones (no prior change) and A being Identity (data distributions of two tasks are the same).Comparison with other transferability estimation and model selection methods: The problem of transferability estimation has gained a lot of attention recently, especially due to the availability of a larger number of pre-trained models. Earlier works used models end-to-end fine-tuned on target tasks to evaluate transferability via task-relatedness estimated using the actual target loss [62] and the Fisher information matrix [1] . However, the requirement of models trained on target tasks limits their practical utility. Other works [22, 21, 54] used the representation of the data from the tasks from these end-to-end fine-tuned models and developed similarity scores that achieved high correlation with transferability. However, these approaches are not practical since they require models end-to-end fine-tuned on the target task. Our work does not depend on models trained on target tasks as shown in our analysis (Theorem 3) making it both theoretically and practically sound.Another line of work [5, 40, 29, 61, 55, 51, 19] focuses on the problem of pre-trained model selection where the goal is to find the best pre-trained classifier from a model zoo that will achieve the highest transferability to a particular downstream task. Thus, the main challenge of this problem is to be able to estimate transferability in a way that is more efficient than fine-tuning the pre-trained models on the downstream tasks. To this end, recent works have proposed several scores that are correlated with the accuracy of the models after fine-tuning them on the target task, which we refer to as score-based transferability estimation (SbTE) methods, in our work.However, unlike our work, the goal of SbTE works is not to propose a universal bound or identify terms that provably govern transferability. Moreover, while the scores proposed in SbTE works correlate well with transferability, they are only meaningful in a relative sense. Concretely, a score of 1 (e.g., for LogMe [46] ) on a CIFAR-100 task for a particular model does not indicate whether transferability is good or bad and requires comparison with scores of other pre-trained models on the same target task. On the other hand, our upper bound directly approximates transferability, e.g., an upper bound of 1 on the CIFAR-100 task for a model implies that transfer learning will incur an average cross-entropy loss of less than 1 implying a highly accurate transfer. Our results in Figs. 3 , and 9 attest that our upper bound is indeed a good estimate of the transferability.Another disadvantage of the scores proposed in these works is that they cannot be compared across target tasks, unlike our upper bound. As observed from Fig. 4 of LogMe [46] , scores for CIFAR-10 are lower than scores for CIFAR-100 on the same pre-trained model, but, the transferability to CIFAR-10 is better than that to CIFAR-100. On the other hand, from our Fig. 9 , the upper bounds on CIFAR-10 are lower than those of CIFAR-100 implying better transferability of classifiers pre-trained on ImageNet to CIFAR-10. Thus, our work is more suitable for estimating the absolute performance on various target tasks given a particular pre-trained classifier.Thus, while the goal of our work differs fundamentally from SbTE approaches the problem addressed in this work is of significant practical importance. In Sec. 4.3, we show that task-relatedness estimated via Alg. 1 achieves competitive performance compared to popular SbTE methods on this problem.Comparison with task transfer learning approaches: This approach focuses on explaining the transfer performance based on the relationship between tasks. For instance, works such as [28, 57, 24] study the problem of few-shot learning (FSL) where a model is trained on data from related training tasks and is adapted to an unseen task using only a few samples. Different from these works, we focus on the setting where a pre-trained encoder trained on some pre-training dataset is adapted with enough samples/gradient steps to a downstream task. This downstream target task may or may not have any relation to the pre-training task unlike [28, 57, 24] .Concretely, [28] proposed a model-agnostic metric called Task Attribute Distance to gauge the success of transfer. Our work, on the other hand, defines task-relatedness based on the similarity of the representations of reference and target tasks in the representations of the pre-trained model (and is model dependent) rather than relying on the attribute information, which may not be available in TL setting. [57] analyzes the sample complexity for learning a model shared across tasks and adapting it to a new target task and shows task diversity to be a crucial component for the success of transfer in the FSL setting. Our work on the other hand does not assume access to shared tasks or restrict the number of samples required for fine-tuning on the target task. Moreover, their notion of task diversity requires access to a set of training tasks that may not be available in the TL setting, making our notion of task-relatedness more practical for TL. Lastly, [24] proposes a notion of task-relatedness for the FSL setting, allowing to utilize all the data from available training tasks to help learn a model on a new task with a few gradient steps. This notion is model-agnostic and defined over the sample space (X × Y) unlike our measure which is defined in the representation space of the model whose transferability needs to be evaluated.Thus while task-relatedness is at the core TL, the notions proposed by [28, 57, 24] are suitable for task transfer setting where as our notion is more suitable for the inductive TL setting.C.1 Small-scale experiment to evaluate Alg. 1 1) Our algorithm produces transformations that achieve a better value for the bound compared to naive transformations. 2) Having the same number of classes in the reference task as in the target task leads to the best value of the bound.3) It is only marginally better to have semantically related classes in the reference task.We evaluate Alg. 1 for computing task-relatedness and predicting the transferability of a ResNet-18 model to CIFAR-10 in two settings. In the first setting, we consider a reference task that comprises data from 10 classes chosen at random and 10 classes that are semantically related to the labels of CIFAR10 from ImageNet [17] (see App. D) whereas in the setting we select reference task with 20 classes. We compare the cross-entropy loss on CIFAR10 obtained after linear fine-tuning (transferability) with our bound, by using various transformations including those learned via Alg. 1.Using 20 reference classes Figure 6 : Transformations optimized using Eq. 4 produce a better taskrelatedness than obtained by naively chosen transformations, primarily due to decreased distribution mismatch. The presence of the same number of classes in the reference as those in the target achieves the smallest task-relatedness value. Semantically related classes in the reference task help but only marginally.We test 3 different cases. The first is FixedAll: where all transformations are fixed (A is fixed to the Identity matrix, B is fixed to a random permutation matrix in the setting with 10 classes, and to a matrix such that two reference classes match a single target class in the setting with 20 classes, D is fixed to the reference prior).The second is LearnedA: where we use Alg. 1 to learn only A while B, D are fixed as in FixedAll, and lastly LearnedAll: where all the transformations are learned via Alg. 1. Our results in Fig. 6 , show that in both settings using FixedAll, our bound is marginally better when the reference task has classes semantically related to the target task. In all cases, the bound becomes significantly better in the LearnedA setting due to learning the transformation A that transforms the reference distribution to better match the target and decrease the distribution mismatch term of the bound. Lastly, by learning all the transformations, in the LearnedAll setting, we achieve the best value for the bound, regardless of whether there are randomly chosen or semantically related classes in the reference task. Moreover, we see that the value of the bound estimated when the reference task has 10 source classes is better in all cases. This is due to smaller re-weighted reference loss since learning a 20-way classifier is more challenging than learning a 10-way classifier, especially with the Lipschitz constraint. In the setting with 20 classes, Alg. 1, prefers to retain data from 10 of the 20 reference task classes, Fig. 7 (right), reducing the re-weighted reference loss, makes B sparse, reducing the label mismatch term, and aligns the reference and target distributions, via A, reducing the distribution mismatch term.Overall, our results show that 1) learning D prefers to retain the data from the same number of reference classes as those present in the target, 2) the B matrix eventually becomes sparse, making the label mismatch term zero, and 3) there is only a small difference in task-relatedness between LearnedA and LearnedAll settings. Based on these insights, we use Alg. 1 in the LearnedA setting, with the reference task containing the same number of randomly sampled classes from ImageNet as the same number of classes in the target task (since it may not always possible to find semantically similar classes for all target tasks) and fix B to be a random permutation matrix, for all other experiments in the paper. We select classes for the reference task from ImageNet due to its diversity and the fact that it has a large number of classes. However, any dataset can be used to define the reference task (e.g., we used Digits datasets for the reference task in Sec. 4.2).Next, we evaluate the effect of knowing the exact matching between the labels of the reference and the target tasks in comparison to a random permutation. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in a setting where only A is learned and B is set to an identity matrix and when B is set to a random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9).We find that the upper bound obtained when B is fixed to identity is only marginally better than the case when B is a random permutation. Specifically, the difference between the bound when B is fixed to a random permutation and when B is an identity matrix is 0.10 for the MNIST→USPS task and 0.17 for the USPS→MNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term. While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related such as for FMNIST and MNIST. Thus, fixing B to a random permutation matrix yields a reliable estimate of transferability in most cases.Here we use the setting considered in App. C.1 where we consider 20 randomly selected classes from ImageNet as the reference task and consider the transfer to CIFAR-10. We plot the results of using different transformations using t-SNE to show how various transformations affect the upper bound in Theorem 3. Our results in Fig. 7 (left) show that when no transformations are learned (FixedAll), the 20 random reference task classes do not overlap with the 10 target classes leading to an increased Wasserstein distance which in turn leads to a larger upper bound. By learning the transformation A (LearnedA), Fig. 7 (center) shows a significantly better alignment between the classes of the reference and target which leads to a decreased Wasserstein distance and hence a tighter upper bound. Moreover, by learning all the transformations (LearnedAll), Fig. 7 (right) shows that not only do the distributions align well but also the prior of the reference is changed to only keep 10 reference classes to match the prior of the target distribution providing a further improvement in the upper bound. This clearly shows the effectiveness of our proposed optimization algorithm in learning various transformations to minimize the upper bound. Here we provide details of the experiment presented in Sec. 4.2 about the effect of the reference task on task-relatedness and transferability. Similar to the results presented in Fig. 4 of the main paper, the results in Fig. 10 show that when the reference and the target tasks are related then task-relatedness is good as in the case when the reference task is MNIST and target is USPS or vice versa achieving a smaller gap to transferability. When the target tasks are unrelated to the reference task data then both the transferability and task-relatedness are low. E.g., when the reference task is MNIST and the target is FMNIST or vice-versa. Task-relatedness is also correlated to transferability, i.e., a model trained on MNIST achieves better transferability to USPS than FMNIST.In this section, we use sentence classification NLP task to demonstrate further the effect of the reference task on task-relatedness. For this experiment, we first fine-tune the entire DistilRoBERTa [34] model distilled on OpenAI's WebText dataset, using a subsample of 10,000 points from the DBPedia dataset. We then use these fine-tuned models to evaluate the transferability to AG news, SST-5, and Yelp datasets. The results in Fig. 12 show that transferability on AG News is the smallest among the three datasets. This coincides with the task-relatedness value obtained after learning the transformations which explains why transfer from DBPedia to AG News is more successful compared to other target tasks. This observation is reasonable, especially considering that both DBPedia and AG News have structured information. Moreover, since DBPedia is related to Wikipedia, the terms and entities appearing in AG News are more related to those appearing in DBPedia in comparison to terms/entities appearing in SST-5 and Yelp which consist of movie reviews and reviews collected from Yelp.For our experiments, in this section, we follow a similar setting of fixing B to be a random permutation matrix, C to the prior of the reference task, and only learn the transformation A. We sample 10,000 points from DBPedia belonging to the same number of classes as those present in the target task (e.g., for AG News we sample data from 4 randomly selected classes of DBPedia) and use this data as the reference task data to train h R with gradient norm penalty (τ = 0.02). All experiments are run for 3 random seeds and average results are reported in Fig. 12 . To use the bound Theorem. 3, it is required that the loss be τ -Lipschitz continuous w.r.t. z in the input domain Z. To enforce this, while learning the weights of the softmax classifier (linear fine-tuning) for the reference task or the target, we add the gradient norm penalty as used in previous works [52, 4] and solve the following optimization problemmin h 1 N i ℓ(h(z i ), y i ) + ρ max y max{0, ∥∇ z ℓ(h(z i ), y)∥ 2 -τ } 2 (ρ ≈ 10 4 )where ℓ(h(z), y) = -w T y z + log j e w T j z is the cross-entropy loss.Constraining the Lipschitz coefficient of the classifier increases both the target and the reference task cross-entropy loss since the hypothesis set is being restricted. The smaller the τ is, the larger the loss becomes. On the other hand, the smaller τ makes the distribution mismatch term in Theorem 3 also smaller. Since the bound is the sum of the reference task loss and the distribution mismatch (and label mismatch), there is a trade-off determined by the value of τ . We illustrate the effect of the values of τ on the empirical and predicted transferability. As mentioned previously, we train both the classifier for the reference task h R and the target h T with an additional penalty on the gradient norm to make them τ -Lipschitz. In Fig. 13 , we present results of varying the value of τ for the transfer to the Pets dataset with ImageNet as the reference task. For this experiment, we selected 37 random classes from ImageNet and only learned the transform A by keeping B fixed to a random permutation and C fixed to the uniform prior over reference task classes. We observe that the performance of linear fine-tuning degrades as we decrease the value of τ but explainability through the bound improves since the distribution mismatch term (dependent on τ ) decreases in the bound. However, making τ too small is not preferable since it leads to an increase in the first term of the bound (re-weighted reference task loss) increasing the bound overall. Moreover, it also leads to a degradation in the accuracy after linear fine-tuning. For our experiments, we use τ = 0.02 since it doesn't decrease the accuracy of fine-tuning significantly and leads to a small gap between empirical and predicted transferability. All codes are written in Python using Tensorflow/Pytorch and were run on an Intel(R) Xeon(R) Platinum 8358 CPU with 200 GB of RAM and an Nvidia A10 GPU. Implementation and hyperparameters are described below. Our codes can be found in the supplementary material. We report an average of three independent runs for experiments in Sec. 4.2 and 4.3.In our work, we used the standard image classification benchmark datasets along with standard natural language processing datasets1 .Aircraft [35] : consists of 10,000 aircraft images belonging to 100 classes.CIFAR-10/100 [30] : These datasets contain 60,000 images belonging to 10/100 categories. Additionally, we created two subsets of CIFAR100 with the first 25 (small CIFAR-100-S) and 50 (medium CIFAR-100-M) classes.DTD [13] : consists of 5,640 textural images belonging to 47 categories.Fashion MNIST [60] : consists of 70,000 grayscale images belonging to 10 categories.Pets [43] : consists of 7049 images of Cats and Dogs spread across 37 categories.ImageNet [17] : consists of 1.1 million images belonging to 1000 categories.Yelp [63] Details of the optimization problem in Eq. 4 and Alg. 1. In Step 5 of the algorithm, we use the network simplex flow algorithm from POT [23] to compute the optimal coupling. Since computing the Wasserstein distance over the entire dataset can be slow, we follow [16] and compute the coupling over batches. Note that the base distance defined in Eq. 2 is non-differentiable. Thus, we use a differentiable approximation d((z, y), (z ′ , y ′ )) := d f eatures (z, z ′ ) + ν • ∥e(y) -e(y ′ )∥ 2 (with ν = 10 8 ) where (z, y) and (z ′ , y ′ ) are samples from the domains R ′′′ and T and e(•) denotes the one-hot embedding of the labels in Step 5/6 . The first three terms in Step 6 of our algorithm correspond to the terms in the objective of Eq. 4 while the two additional terms are added to penalize the constraints of class prior matching P T (y) = BD and invertibility of the matrix A, respectively as required by Theorem 3. We use the softmax operation to ensure B and D are a valid probability matrix and vector.In our experiments, in Sec. 4.1, we used pre-trained models available from Pytorch for ResNet18/50, along with publicly available pre-trained models provided in the official repositories of each training method. For each experiment, we subsample data from the ImageNet dataset belonging to the same number of classes as those present in the target dataset and use this data to train the linear layer on top of the representations extracted from the pre-trained model along with a gradient norm penalty (Reference task classifier). To speed up the experiments, we use only 10,000 points from the subsample of ImageNet for training the linear classifier and computing the transfer. For evaluation, we use a similar subsample of the validation dataset of ImageNet containing all the samples belonging to the subsampled classes. Fine-tuning on this dataset takes about 0.05 seconds per epoch for the task of transfer from ImageNet to Pets with the ResNet-18 model (we run the fine-tuning for a total of 5000 epochs).Along with training the linear classifiers with a gradient norm penalty (with τ = 0.02), we standardize the features extracted from the pre-trained models to remove their mean (along each axis) and make them have a unit standard deviation (along each axis). While standardizing the features do not have a significant impact on the loss of the classifiers, including it makes it easier to match the distributions of the reference task and target data after transformations. Since our optimization problem transforms the reference task distribution to match the distribution of the target by solving the optimization problem in Eq. 4 by working on mini-batches, the size of the batch must be greater than the dimension of the representation space of the pre-trained encoder. E.g., for ResNet18 models which have a representation dimension of 512, we use a batch size of 1000 and for ResNet50 models which have a representation dimension of 2048, we use a batch size of 2500. Having a smaller batch size than the dimension could lead to a noisy gradient since for that batch the transformation can achieve a perfect matching, which may not generalize to data from other batches or unseen test data.While computing the transformations, we apply the same augmentation (re-sizing and center cropping)/normalization to the training data as those applied to the test data. Along with this, we extract the features of the training and test data from the pre-trained model once and use these to train the linear layer. We note that this is done to save the computation time and better results could be obtained by allowing for extracting features after data augmentation for every batch.Finally, for our experiments in Sec. 4.2, the encoders are trained end-to-end on the reference task. This is in contrast to our other experiments where the encoders are pre-trained and data from the reference task is only used for linear fine-tuning. Using these models, task-relatedness is evaluated by fine-tuning a linear layer using the data from the target task as well as the transformations computed by solving Eq. 4. We used τ = 0.2 here. We run the experiments with 3 random seeds and report the average results.For our experiments in Sec. 4.3, we used the official code from [61] to compute the scores for NCE, Leep, and LogMe along with the official code of [51] for SFDA. For PACTran [19] , we also use their official code with the PACTran-Gaussian method with N/K = 100, β = 10N, σ 2 = D/100 where N denotes the number of samples and K denotes the number of classes. This setting is similar to the PACTran-Gauss f ix setting used in their work with the difference that we use N/K = 100 to use a sufficiently large number of samples, especially considering that all our other results for SbTE methods are computed on the full training set. For OTCE, we follow the official code and compute the recommended OT-based NCE score and OTCE score (λ 1 = -0.0001 and λ 2 = -1) using 4000 randomly selected training samples from the two tasks. For the source task, we subsample the same number of classes as the target task and use. For the H-score, we use the official code to compute the H-score [6] .For estimating transferability by solving Eq. 5, we set λ=0.01 for all the experiments in Sec. 4.3.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).All NLP datasets and models are obtained from https://huggingface.co/. 


Decision: The paper addresses task-relatedness in cross-domain cross-task transfer learning. The main contribution is an upper bound on transferability of the representations of pre-trained models to downstream tasks measured in terms of their relatedness to a given reference task. The proposed task-relatedness works without requiring target labels, and the notion of a reference task has been well received.
 
Initial reviews raised several issues, including clarifications of the contributions, limited empirical evidence, and related work comparisons. The rebuttal successfully addressed most of these concerns, leading to two reviewers raising their scores. After careful deliberation, the decision was made to accept this paper - congratulations to the authors! It is crucial, however, that all the promised improvements and clarifications including the related work, sensitivity analysis and presentation clarity, are addressed in the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a way of assessing the transferability of representations from a pre-trained model to a target task by assessing the impact of the pre-trained model's representations on a reference task. By "transforming" the reference task into the target task, the paper produces a bound on the training loss of the target task
strength_weakness: strength_weakness: My main issues with this paper are as follows. 
1.  **Utility of the method** :  I was initially excited by the following sentence in the introduction of the paper -- *"Moreover, unlike previous SbTE metrics, task-relatedness can be estimated even without labeled target data, making it suitable for unsupervised transferability estimation, highlighting the advantage of a reference task as used in our analysis"*. However,  it seems that Algorithm 1 requires access to the target labels $\mathcal{Y}_{T}$. This calls into question this stated advantage over the SbTE methods. Specifically, SbTE methods like TransRate and LEEP do not require  training on a reference task which not only comes with extra overhead but whose selection requires a-priori meta-intuition about which tasks are similar to the target task. Also, none of the empirical evaluations presented actually show that the proposed method is superior to SbTE approaches for choosing models form a zoo. Specifically, the paper presents hard to interpret correlation measures in Table 1 as proof of transferability -- when a superior test would be compare deltas in final training performance of the models the models that each method select from a zoo.
2. **Correctness / Impact of claims**:  
  [a]. From line 238 - 241: *However, since finding data semantically related to the target task may not always be possible we choose a reference task with the same number of classes as the target and fix that matrix B to a random permutation of identity (making the label mismatch term zero) and D to the prior of the reference task, learning only the transformation A, in our experiments.*. This seems to undercut the utility of the analysis presented. Why does this work ? Why even discuss these terms if they are effectively fixed ?  
  [b]. It feels like the insight of *Highly related reference–target task pairs, based on task-relatedness, achieve higher transferability coinciding with the semantic relatedness between tasks.* is just punting the problem of transferability estimation to task similarity estimation (and now calling on practitioner intuition to decide which tasks are highly related). Thus it does not seem that the authors are actually effectively solving the problem of transferability estimation -- they are just transforming it into a different problem.  
  [c].  The authors claim  "tight bounds / small gaps between task related and transferability" but this is based on (imo) just eyeballing the graphs in  Figure 3 and 9. We are not provided with a rigorous way to interpret the gap. Eg -- a very simple way to artificially present a small gap is to just plot the the losses at random init of the classification head for the the reference and target tasks that have the same number of class labels. 
3. **Strength of underlying technical assumptions**:    
  [a].  the K_r > K_t seems like a strong assumption that limits the methods applicability. What if the most similar task to the target task has K_r < K_t.   
  [b]. Also as mentioned above in 238 - 241 it seems the label and class-prior matching component of the algorithm effectively has no influence on the final algo results.


Score updated after discussion
questions: questions: 1. Work focuses  on classification based tasks -- could you discuss how this applies to generative tasks
2. Can you discuss how faithful the estimates / method is when K_r < K_t  ?
3. The analysis is for training error -- how is transferability estimation affected if there is overfitting to the reference task ?
limitations: limitations: N/A -- paper presents some limitations
paper_summary: paper_summary: This paper analyzes transfer learning from the perspective of task relatedness, a model for transforming from a reference task (pretraining task) to the target task is proposed, which consist of prior transform, label transform, and feature transform. 
The task relatedness is then measured as the distribution mismatch of the transformed distribution and the target distribution.
Experiments have shown that the proposed task relatedness tightly upperbounds the transferrability to a range of architectures and datasets.
strength_weakness: strength_weakness: 1. The datasets for evaluation and empirical studies seem to be small in scale, for example, MNIST and CIFAR which is small in the image resolution, or Aircraft and DTD which is small in the number of images.
questions: questions: The appendix has answered most of my questions.
My only question is that for algorithm 1, does this has guaranteed convergence? and practically how long does it take to reach convergence?


Minor:

A possible related work: Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective, ECCV 2022.
limitations: limitations: The paper has discussed its limitations
paper_summary: paper_summary: This paper proposes an analysis that analyzes the transferability of the representations of pre-trained models to downstream tasks in terms of their relatedness to a given reference task. It aims to understand when the knowledge of these pre-trained models can be transferred to obtain high-performing models on downstream target tasks. Their analysis leads to an upper bound on transferability in terms of task-relatedness, quantified using the difference between the class priors, label sets, and features of the two tasks. The experiments results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: (i) This work mentioned that it is the first to analyze transferability for classification tasks under cross-domain cross-task settings. However, there are multiple works [1-5] that have focused on the transferability and generalization of learned representations. Meanwhile, transferability is mainly evaluated by utilizing the performance differences between different domains and tasks with distribution shifts. This is also the case in previous works, but this is not the case for the introduction of motivation in this paper.\
[1] Understanding few-shot learning: Measuring task relatedness and adaptation difficulty via attributes.\
[2] Otce: A transferability metric for cross-domain cross-task representations.\
[3] On the theory of transfer learning: The importance of task diversity.\
[4] An information-theoretic approach to transferability in task transfer learning.\
[5] Task relatedness-based generalization bounds for meta learning.

(ii) (Minor) Although this work focuses more on theoretical exploration, empirical verification on real-world datasets is also necessary, such as ablation studies on the three proposed terms, trade-off experiments (performance vs. efficiency vs. memory footprint) due to the introduction of additional computations, and comparisons with baselines on transferability. 

(iii) (Minor) The formatting and layout of some formulas and tables need to be adjusted to make the reading clearer and more intuitive while some equations with writing errors, such as Eq. 3, L527-528, the proof of Corollary 1. At the same time, due to the large number of formulas and physical quantities, it may be better to construct a table to explain the symbols and definitions.

(iv) (Minor) The track chosen by the paper is not that matched to its content.
questions: questions: Please see Weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: This paper considers transfer learning in a general cross-domain, cross-task setting. It introduces the concept of "task-relatedness" into measuring whether transfer learning can be successful. Specifically, task-relatedness consists of (1) a reference loss term for measuring the difference in class prior distribution between the target and source tasks; (2) a label mismatch term for measuring the conditional entropy between the label distributions; and (3) a distribution mismatch term  for measuring the difference in feature distributions. To make computing task-relatedness practical, the paper proposes several approximation schemes. It shows the metric is highly correlated with model transferrability in practice and can be used for selecting pretrained models.
strength_weakness: strength_weakness: 1. Limited novelty: the idea of using distribution distances to measure transferability is not new. As the related work section has pointed out, using the Wasserstein distance to estimate domain difference has been studied by OTDD [1]. Works like OTCE [2] also uses the proposed metric to do pretrained model selection. One argument the paper makes is that the proposed task-relatedness works without requiring target labels. While this does add some practical significance, the proposed method is a simple pseudo-labeling scheme without theoretical justification.
2. Limited evaluation: the baselines considered in the experiments are not comprehensive, e.g., OTCE is not evaluated, and other recent OT-based works like [3] are not discussed. There's also no comparison for different pseudo-labeling schemes for the missing-label setting. Besides, there's no ablation study showing the contribution of each of the three terms proposed for computing task-relatedness.

[1] Geometric Dataset Distances via Optimal Transport. David Alvarez-Melis. Nicolò Fusi.

[2] OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations. Yang Tan, Yang Li, Shao-Lun Huang.

[3] Wasserstein Task Embedding for Measuring Task Similarities. Xinran Liu, Yikun Bai, Yuzhe Lu, Andrea Soltoggio, Soheil Kolouri.
questions: questions: 1. Class-prior transformation assumes the reference task has more classes than the target task. Is this too strong an assumption? Does the proposed method applies to the settings where the target task has more labels?

2. In Figure 3, the rankings of transferability for different target tasks are the same across all models. This is perhaps because the same reference dataset is used, and the experiments do not show the effect of varying model backbones. It would be better if the paper can evaluate multiple reference datasets on multiple models. (Figure 4 does look at different reference datasets but the model is fixed here.)

3. Does the proposed metric applyy to non-classification tasks like regression or auto-regressive prediction?
 
4. What's the max cross-domain distance the theory holds? I’d imagine that when the domain gap is large, the theory breaks. Have you studied that?
limitations: limitations: See weaknesses.

Comments: title: Response by Authors
Response
Invitation for further discussion
Response by Authors
Response by Authors
Response by Authors
Response
comment: We sincerely appreciate the reviewer for engaging in the discussion and are glad that our responses clarified their concerns. We are encouraged that they find practical value in the ability of our method to estimate transferability without target labels. 
Following the suggestions of the reviewer, we will include the discussion and experiments for the initialization of transformations, presented here, in the paper to further improve the quality of our work.
Thanks for your clarifications 

>`For computing the results in Table 2, the matrix B is fixed to a permutation matrix (similar to our other experiments). Thus, there is no variability due to the matrix B.`
I think the target label free case is clearer now, but I do still think it's important to evaluate the sensitivity of the final result to the initial (random) value of B.
I think I see the value of the method at least in the case of no target labels present.

>`to use for computing task relatedness, a practitioner can estimate the upper bound using a few benchmark/publicly available datasets and select the reference task as the one that minimizes the upper bound, similar to the approach suggested by the reviewer of treating the reference task as a variable.`

As I mentioned in my initial review, this seems like punting the problem to a new problem of reference task selection and requiring practitioner intuition in selecting the reference task.
And then there is the case of the computational overhead (if you have to run the algo on several tasks ) of this approach compared to other methods that do not require reference task 


My questions have been sufficiently clarified by the authors. I will raise my score from 3 --> 6.
Dear reviewer, 

We sincerely appreciate your time and effort in reviewing our manuscript. Through our rebuttal we clarified your concern around making Alg. 1 suitable for transferability estimation when target labels are unavailable. We also justified the reason for reporting the correlation coefficient for the problem of transferability estimation in line with all prior SbTE works [5, 34, 25, 54, 49]. Lastly, our latest response also provided clarification and more empirical evidence showing that fixing some of the transformations (such as fixing the matrix B to a random permutation) eases the combinatorial nature of the optimization problem in Eq. 3 without significantly affecting the value of the transferability estimate.  

As the discussion period is about to end, it would be appreciated if you could let us know of any other concerns we can address at this point. If you believe our responses sufficiently addressed your concerns we would humbly request you to re-evaluate our paper in the light of our discussion and update your rating of our work. 

Thank you immensely for your contributions and thoughtful consideration.

Regards
Authors
We appreciate the reviewer's response to our rebuttal. We would be happy to address any other concerns the reviewer has.
I appreciate the authors' detailed response. I'll raise my score to a 5.
We thank the reviewer for their response to the rebuttal.

While we agree that transfer learning is a broad research topic, our paper specifically works in the inductive transfer learning setting where the focus is to leverage an inductive bias (a pre-trained model) to improve performance on a target task. Due to this, our analysis specifically proposes a model-dependent way of measuring the relatedness between a reference and a target task. The works such as [1,3,5] provided by the reviewer deal with the problem of task transfer learning. Their goal is to identify the relationship between tasks, regardless of the model, to explain the transfer performance.

E.g., the task attribute distance proposed by [1] is independent of the model being used and only depends on the tasks. (This terminology of inductive and task transfer learning is based on a previous work LogMe[54] which has the same setting as ours.). Thus, the setting of our work differs significantly from the setting in [1,3,5].

We thank the reviewer for bringing this up. We will add a discussion to the paper and clarify the type of transfer learning being studied in the paper along with the mention of the relevant papers from the area of task transfer learning.

We would be happy to address any further concerns the reviewer may have.
We thank the reviewer for their response to the rebuttal.

> Are ... of B.

The reviewer’s intuition is correct that the pseudo labels are computed using the $B$ matrix provided at the initialization of Alg. 1 for the first time. But as Alg. 1 progresses the matrix $B$ will also be updated. As a result, the pseudo labels of the target data are also updated. This can be thought of as an additional step in Alg 1 as follows:

Step 2a: If target labels ($y_T^j$) are not available then compute $y_T^j = arg max_{y \in Y_T} Bh_R(A^{-1}x_T)$ for all $j = 1, …, n_T$.

For computing the results in Table 2, the matrix B is fixed to a permutation matrix (similar to our other experiments). Thus, there is no variability due to the matrix B.

> I would ... mistaken.

The reviewer's intuition is correct. The choice of reference task affects the upper bound as we demonstrated, in detail, in Sec. 4.2. Specifically, when the reference and target tasks are more related (in terms of our task-relatedness metric) the upper bound is smaller.

However, finding the most related reference task for every target task may not be possible in practice. Since our analysis does not impose any restriction on the choice of the reference task to use for computing task relatedness, a practitioner can estimate the upper bound using a few benchmark/publicly available datasets and select the reference task as the one that minimizes the upper bound, similar to the approach suggested by the reviewer of treating the reference task as a variable. 

> I would like ... problem.

This seems to be a misunderstanding. SbTE methods as per [54] produce a score for each model (ideally without end-to-end fine-tuning on the target) that correlates well with end-to-end accuracy. This allows selecting the top-performing model by simply evaluating these scores. Thus, all prior works report correlation coefficients similar to our results in Table 1. 

Moreover, scores produced by SbTE methods do **not** numerically approximate the accuracy after end-to-end fine-tuning. For example, the results in Fig. 4 of [54] show that the value of SbTE metrics (y-axis of the plot) such as LEEP [34], NCE [50], and LogMe [54] lies in the range [-0.8,-0.3], [-0.45, -0.25], and [0.935, 0.95] respectively for the Aircraft dataset whereas the end-to-end fine-tuning accuracy (x-axis of the plot) lies in the range ~[72.5, 87.5] for the various models. 

It would be great if the reviewer could clarify their question if we have misunderstood them.  

> It is very ... suffice.

The experiment in Appendix C.1 is a small-scale experiment as it is in the same setting as in Sec 4.1 with the ResNet-18 model. It is intended to be an ablation study to understand how different transformations affect the upper bound.

The label mismatch term of Theorem 3 is minimized with a sparse B matrix corresponding to a one-to-one mapping between the classes of the reference and target tasks. However, due to the combinatorial nature of the problem finding the exact permutation is infeasible. Regardless of that, the linear transformation $A$ can align the distributions of the representation of the target to any permutation of the reference classes quite accurately. This is primarily due to the high dimensionality of the representation space where the linear transformation is applied. As a result, the value of the bound differs only by little for the different permutations of the reference task’s classes. This is indeed a very interesting outcome that makes our bound usable without requiring the exact semantic matching between the classes of the reference and the target classes. 

Below, we present an empirical evaluation to show that the difference in the bound computed with true vs random permutation is small. We use MNIST as the reference task and USPS as the target task (and vice-versa). We compare our results in a setting where only $A$ is learned and $B$ is set to an identity matrix and when $B$ is set to a random permutation matrix. Note that the identity matrix corresponds to the correct mapping between the classes of MNIST and USPS tasks (both contain digits from 0 to 9). 

We find that the upper bound obtained when $B$ is fixed to identity is only marginally better than the case when $B$ is a random permutation. Specifically, the difference between the bound when $B$ is fixed to a random permutation and when $B$ is an identity matrix is 0.10 for the MNIST→USPS task and 0.17 for the USPS→MNIST task. The primary reason for the decrease in the upper bound comes from the reduced distribution mismatch term. 

While the upper bound improves slightly when the ideal matching between the labels is known, such a mapping may not be known when the labels of the tasks are not related, e.g., for FMNIST and MNIST. Thus, fixing $B$ to a random permutation matrix yields a reliable estimate of transferability in most cases.


We would be happy to address any further concerns the reviewer may have.
Thank you for the responses. But my main concerns still exist, for example, the author mentioned that transfer learning is a setting uniquely considered in this article, but meta-learning can be considered a branch of transfer learning; the concept of task similarity is difficult to distinguish from the one in this article, and the descriptions like "can be used or not" and "may not" is vague. Therefore, the confusion in Weakness 1 is difficult to eliminate from the response. At the same time, by reading the responses of other reviewers, I found some issues that were previously overlooked. Therefore, after carefully checking all the responses, I tend to maintain my score.
Thanks for your detailed response to my review.  Please see my responses below

> `A: While Alg. 1 requires labels of the target task, we discuss in Lines 329-347 how to use Alg.`

Hmm -- this is very confusing / seems contradictory. First, note that Algo 1, uses a random permutation matrix as B. 
Next, in lines in Lines 329-347 requires you to use B to compute the pseudo-labels. But 156 - 166 requires using target labels to specify B. So are you saying that to compute pseudo labels you use a random B ? 
Please clarify if that is not the case, and if that is the case, what are the error bars on the results in Table 2 ? I would imagine some non-trivial sensitivity to the choice of B.

> `B: .. We emphasize that the reference task does not need to be similar to the target task to use our methodology for estimating transferability. Any benchmark/publicly available dataset can serve as the reference dataset and its use is not a bottleneck for our analysis/approach.`

This is also confusing to me -- and seems a bit "free-lunchy".  Theorem 3 on line 213 clearly gives **an upper bound** that depends on the label and distribution.  I would imagine that the choice of the reference tasks affects the looseness of this upper bound -- we can simply think of the reference task as a variable that we are minimizing the upper bound over to make it tighter.
Thus, the quality of your transferability estimate does depend on the *appropriate* choice of reference task. 
Please clarify if I am mistaken.

> `While the best way to evaluate transferability is to fine-tune a model end-to-end on the target task, it is computationally prohibitive.` 

I would like to respectfully push back on this. You could have set up much smaller scale experiments to validate that the transferability scores are actually good for a reasonably sized model selection problem.

> `We ablate the use of different transformations and measure the effect of learning each of them in Appendix C.1. Our experiments show that minimizing the conditional entropy by learning transformation B prefers a sparse mapping between the classes of the reference and the target tasks. .... Based on this, we use a reference task with the same number of classes as the target task and fix the matrix B to be a random permutation of the identity matrix.`

As you mention in Appendix C.1 this is done on a toy problem -- it's a big leap to assume that this is the right thing to do for a more realistic problem without a bit of preliminary / small scale experimentation. Next, even if there is a sparse 1-1 mapping between classes, it is very strange to me that you decided that the exact mapping is not important and rather a random one would suffice. Basically, I understand your restriction to the set of `permutation matrices` but not your assumption that any random one of these matrices would suffice