Title: FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated Learning

Transfer Learning
Abstract: The performance of Transfer Learning (TL) significantly depends on effective pretraining, which not only requires extensive amounts of data but also substantial computational resources. As a result, in practice, it is challenging to successfully perform TL at the level of individual model developers. Federated Learning (FL) addresses these challenges by enabling collaboration among individual clients through an indirect expansion of the available dataset, distribution of the computation burden across different entities, and privacy-preserving communication mechanisms. Despite several attempts to devise effective transferable FL approaches, several important issues remain unsolved. First, existing methods in this setting primarily focus on optimizing transferability within their local client domains, thereby ignoring transferability over the global learning domain. Second, most approaches focus on analyzing indirect transferability metrics, which does not allow for accurate assessment of the final target loss and extent of transferability. To address these issues, we introduce two important FL features into the model. The first boosts transferability via an exchange protocol between the clients and the server that includes information about cross-client Jacobian (gradient) norms. The second feature promotes an increase of the average of the Jacobians of the clients at the server side, which is subsequently used as a local regularizer that reduces the cross-client Jacobian variance. A rigorous analysis of our transferable federated algorithm, termed FedGTST (Federated Global Transferability via Statistics Tuning), reveals that increasing the averaged Jacobian norm across clients and reducing its variance ensures tight control of the target loss. This insight leads to the first known upper bound on the target loss of transferable federated learning in terms of the source loss and source-target domain discrepancy. Extensive experimental results on datasets including MNIST → MNIST-M and CIFAR10 → SVHN suggest that FedGTST significantly outperforms other relevant baselines, such as FedSR. For example, on the second source-target dataset pair, we improve the accuracy of FedSR by 9.8% and that of FedIIR by 7.6% when the backbone used is LeNet.

FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning

Abstract

The performance of Transfer Learning (TL) significantly depends on effective pretraining, which not only requires extensive amounts of data but also substantial computational resources. As a result, in practice, it is challenging to successfully perform TL at the level of individual model developers. Federated Learning (FL) addresses these challenges by enabling collaborations among individual clients through an indirect expansion of the available dataset, distribution of the computational across different clients, and privacy-preserving communication mechanisms. Despite several attempts to design effective transferable FL approaches, several important issues remain unsolved. First, existing methods primarily focus on optimizing transferability within local client domains, thereby ignoring transferability across clients. Second, most approaches focus on analyzing indirect transferability metrics, which does not allow for accurate assessment of the final target loss and the degree of transferability. To address these issues, we introduce two important FL features into the model. The first boosts transferability via an exchange protocol between the clients and the server that includes information about cross-client Jacobian (gradient) norms. The second feature promotes an increase of the average of the Jacobians of the clients at the server side, which is subsequently used as a local regularizer that reduces the cross-client Jacobian variance. A rigorous analysis of our transferable federated algorithm, termed FedGTST (Federated Global Transferability via Statistics Tuning), reveals that increasing the averaged Jacobian norm across clients and reducing the Jacobian variance ensures tight control of the target loss. This insight leads to an upper bound on the target loss of transferable FL regarding the source loss and source-target domain discrepancy. Empirically, experiments on public benchmarks show that FedGTST significantly outperforms other baselines, such as FedSR.

1 Introduction

Transfer Learning (TL) has received significant interest in the machine learning community due to its ability to extract representative features from source tasks and use them to improve the generalization capability on related target domain problems [48, 52]. In addition to boosting the performance of a target domain model, TL also reduces the computational cost of fine-tuning the target domain model. Nevertheless, effective source pretraining in TL is practically challenging for individual model developers because it requires access to large datasets as well as significant computational resources [35]. To resolve this problem, one can leverage Federated Learning (FL), which refers to decentralized learning protocols used in mobile and IoT devices [18]. FL not only increases access to multiple datasets in a decentralized manner and alleviates the computational burden of individual clients, but it also protects the privacy of local data [32]. As a result, a number of recent works have outlined methods for transferable FL, including FedADG (Federated Adversarial Domain Generalization) [55], FedCDG (Federated Contrastive Domain Generalization) [53], FedSR

(Federated Simple Representations) [34], FedIIR (Federated Implicit Invariant Relationships) [12], FedCCST (Federated Cross-Client Style Transfer) [4], FedMM (Federated Adversarial Domain Adaptation) [42] and StableFDG (Stable Federated Domain Generalization) [36]. Despite the promising preliminary findings provided by the techniques above, several combinations of important issues remain unsolved across the spectrum of methods.

For privacy leakage , the limitations of existing methods include: 1. FedADG forces each client source domain to align its representation distribution with that of the target domain, and therefore violates data privacy because source domains are given access to the target domain in order to perform the alignment; 2. FedCCST boosts global transferability by increasing local diversity to avoid local overfitting. It therefore requires clients to share their local representations with each other and this information is subsequently used for local data augmentation. This is a direct violation of FL privacy constraints; 3. StableFDG expands local data diversity by sharing style statistics (i.e., representations, including means and variances). This clearly leads to the leakage of local privacy-sensitive information.

For local overfitting , the shortcomings of a group of the methods above are as follows: 1. FedSR learns a simple representation through regularization during local training, by exploiting the similarity between the representation and the data, given the labels. However, since the regularized local training relies completely on local structures (i.e., local models, representations, labels, data), it leads to overfitting of local distributions, and thus has limited capability to learn cross-client invariant features, which is key for global transferability; 2. FedCDG uses a contrastive local regularizer on representations generated by various samples within the same class. This leads to overfitting in the local domain since no cross-client information is exploited.

For communication complexity , we observe that: 1. FedIIR is suboptimal. Although it mitigates the problem of privacy violation and avoids local overfitting by adding a local regularizer capturing the distance between the local gradient and the global gradient, it requires communicating gradients between the clients and the server and therefore doubles the communication cost compared to baselines (additionally, FedIIR performs well for a large number of clients, but offers average performance when this number is small); 2. Similar communication complexity problems are faced by FedCCST and StableFDG, which rely on communicating styles (i.e., representations); 3. FedMM requires significant communication overhead for adaptation using distribution-matching techniques [58], which involves solving an intractable non-convex-non-concave optimization problem [56].

Finally, prior works mostly lack explicit theoretical analyses of global transferability: they do not tend to quantify the performance/loss of the pretrained model fine-tuned on the target domain.

In summary, perhaps the most important unresolved problem with known transferable FL models (with the exception of FedIIR) is that they use centralized TL approaches during local training, and do not fully exploit features specific to federated learning (for details, see also the discussion in Section 2).

Our contributions. We describe what is, to the best of our knowledge, the first approach to federated transfer learning termed Federated Global Transferability via Statistics Tuning (FedGTST) that simultaneously alleviates the above issues faced by existing methods. Our main contributions can be summarized as follows.

We use a new regularizer that encodes cross-client statistics and forces the local training process to tune the global statistics in a 'direction' that improves global transferability rather than just local transferability. This is achieved through subtractions of global norms of Jacobians (gradients) communicated by the server.

We suggest to only communicate scalars, more precisely, Jacobian norms , which introduces a negligible communication overhead in the overall model exchange protocol.

We ensure that our communication schemes do not allow uncontrolled access to data and thereby ensure data privacy.

We rigorously prove that even though only small discrepancies among local gradients may exist upon regularization, transferability can be low as regularization can impede the growth of the gradient norm. To boost the Jacobian norm, we implement specialized protocols at both the client and server levels. Finally, we establish relevant bounds on the transferability loss for this setting.

The main technical insights provided by our analysis are as follows. Two FL-specific factors, a small cross-client Jacobian variance and larger cross-client Jacobian norm are indicative of good transferability. These factors are direct performance indicators, unlike indirect factors (e.g., feature invariance) which only suggest improved transferability. Our findings are based on the first direct measure of transferability, which equals the loss on the target domain incurred by the pretrained federated model. The FL-specific factors govern the bounds on the loss and therefore allow one to control them for better transferability. We validate these findings through extensive experiments which show that FedGTST outperforms methods such as FedSR and FedIIR by as much as 10% .

2 Related Work

FL is a machine learning paradigm in which multiple entities collaborate to train a global model without sharing their local data (see [15] for a comprehensive review). FL has gained significant attention due to its potential to address privacy concerns while enabling large-scale collaborative learning [44]. Relevant to this work, [32] proposed the Federated Averaging (FedAvg) algorithm, which aggregates model updates from multiple client devices to train a global model. Another relevant line of work [51] introduced FedProx, a federated optimization algorithm that incorporates proximal terms to handle non-iid data distributions. FL methods nevertheless still face several challenges. One challenge is dealing with highly heterogeneous local datasets [26] , for which the recent work FedImpro[46] proposed leveraging aggregated feature distributions to address client drift. Another challenge is the communication overhead incurred during the aggregation of model updates [2]. Minimizing communication costs while ensuring convergence and data privacy remain active topics of research in FL. Also, many FL solutions primarily emphasize performance in the client domain without considering the performance of the model on unseen domains.

TL is a powerful machine learning technique that allows models to leverage knowledge gained from one task to improve performance on another related task [35]. TL has been widely adopted in various domains such as computer vision, natural language processing, and speech recognition, where labeled data may be scarce or expensive to acquire [47, 48]. A common approach in TL involves fine-tuning a pre-trained model on a target task using a small amount of labeled data, which often leads to improved generalization and faster convergence compared to training from scratch [52]. Recent works in TL have focused on developing more effective algorithms, such as domain adaptation methods that address the discrepancy between the source and target domains [11]. Additionally, TL techniques have been used to handle tasks with limited amounts of labeled data through techniques like semi-supervised and self-supervised learning [40]. TL still faces challenges such as negative transfer, where information from the source task actually degrades performance on the target task; and, it requires careful selecting of appropriate pretrained models and transfer strategies for specific tasks and domains [35]. Current TL methods often require that one entity possesses knowledge of all data, violating the privacy requirements of FL. Moreover, we comment on Gradient Matching in TL in Appendix I.1

Transferable Federated Learning (TFL) is an emerging research area at the intersection of FL and TL. One of earliest contributions to the field, FedADG [55], encourages the transferablity of FL through adversarial local training. However, the work does not provide theoretical guarantees, and existing studies [50] indicate that adversarial robustness does not necessarily lead to better transferability. Other methods, such as FedSR [34] and FedCDG [53], enhance transferability by adapting standard representation learning from a single-agent to a federated setting; they do not incorporate FL-specific features (i.e., instructions provided by the server, cross-client model properties etc). Note that although FedSR has successfully included centralized invariant feature learning into FL, it uses centralized methods locally and then shares information with the global model, and thereby does not fully exploiting FL capabilities. Thus, using FedSR, each client can learn very different representations that are hard to aggregate at the central server. More precisely, FedSR does not communicating information that can help improve the transferability of the global model. Among all the previously discussed methods (FedADG, FedCDG, FedSR, FedIIR, FedCCST, and StableFDG), FedIIR is the closest to our approach and may be seen as a special case of our method which has better performance, smaller communication complexity and comes with provable global transferability guarantees. Furthermore, we discuss the distinctions and connections between TFL and Generalization of FL, a topic potentially relevant to TFL, in Appendix I.2.

3 Preliminaries

General Supervised Learning Settings . We denote the data space by X , the feature space by Z , and the label space by Y . A model h : X → Y typically takes the form h = g ◦ f , where f : X → Z is a feature extractor and g : Z → Y is a classifier. Denote the function class for the entire model, the feature extractor and the classifier by H , F , G , respectively, so that h ∈ H , f ∈ F , g ∈ G . Denote the weights of model ψ ∈ { f, g, h } as w ψ . Given a loss function l : Y × Y → R and a domain distribution D over X × Y , the population loss L D ( h ) is defined as

General Framework of TFL . In TL, the two typical learning phases are: a) pretraining on the source domain; and, b) finetuning on the target domain. In the context of TFL, pretraining is conducted via FL over source (local) domains, while the global model is trained and then finetuned on the target domain during the second phase. In both phases, supervised learning is performed with full access to the labels. More details are provided next.

Pretraining Phase in TFL: FL on Source (Local) Domains . The source domain is a composition of the agents' local domains, {D ( k ) } , with k ∈ [ K ] denoting the client index and K representing the total number of clients. The source loss is defined as the standard federated loss on the source domains.

Let h ∗ = g ∗ ◦ f ∗ be an optimal global solution for the objective (2). In FL approaches, the problem solution is the result of the central server's aggregation of local models into a global one. We denote the local solutions involved in creating the optimal global solution ψ ∗ ( ψ ∈ { f, g, h } ) by { ψ ∗ ( k ) } ; through averaging aggregation, we obtain the optimal global weights w ∗ ψ = 1 K ∑ k w ∗ ( k ) ψ .

Finetuning Phase in TFL: Supervised Finetuning on the Target Domain. Upon obtaining the optimal pretrained global solution h ∗ = g ∗ ◦ f ∗ , the pretrained feature extractor f ∗ is fixed and applied to the target domain D T . The target loss is defined as the loss on the target domain D T , i.e.,

Through finetuning, a new classifier g ∗ T := arg inf g ∈G L tgt ( g ◦ f ∗ ) is determined by minimizing the target objective (3).

Transferability Assessment . With a slight abuse of notation, we define the optimal target loss as

We formally define the measure of transferability of TFL as the optimal target loss L ∗ tgt , as it directly reflects the performance of a transferable model on the target domain. A smaller L ∗ tgt , or a tighter bound on it, indicates better transferability.

4 Theoretical Bounds on the Target Loss

4.1 A General Bound Based on Discrepancy/Divergence

We start with Definitions 1 and 2 borrowed from existing TL studies that characterize the domain discrepancy, and then propose a new domain divergence tailored to Transferable FL (TFL), including the cross-client discrepancy in Definition 2 and the source-target discrepancy in Definition 3.

Definition 1 ( G , F -discrepancy [50]) . Given a classifier class G , a feature extractor class F , the source domain D S and the target domain D T , with a slight abuse of notation, the ( G , F ) -discrepancy is defined as

Remark. The G , F -discrepancy has also been used in the analysis of domain adaptation [50].

We next adapt the H -discrepancy to measure the cross-client domain discrepancy in transferable FL (Definition 2), and tailor the G , F -discrepancy (Definition 1) to measure the source-target discrepancy in TFL (Definition 3).

Definition 2 (Cross-Client Divengence for TFL) . Given a model class H and federated local domains D fed S = {D ( k ) } k ∈ [ K ] , with d H ( · , · ) defined as Definition 5, the intra-client discrepancy is defined as

̸

Note that d H ( D S ) equals the average of H -discrepancies over all local domain pairs and therefore measures the intra-discrepancy on the non-iid distributed source domains. When source domains are iid across clients, we have d H ( D S ) = 0 .

Definition 3 (Source-Target Discrepancy for TFL) . Given a classifier class G , a feature extractor class F , federated local domains D fed S = {D ( k ) } k ∈ [ K ] , and the target domain D T , with slight abuse of notation, the federated ( G , F ) -discrepancy is defined as

Based on the two TFL-specific domain discrepancy definitions (i.e., Definition 2 and 3), we derive a general bound on the TFL loss in Theorem 1. The TFL-specific source-target discrepancy (Definition 3) is further used in Theorem 2 which presents a bound on the target loss using cross-client statistics. For our theoretical analyses, we need the following common assumptions.

Assumption 4.1 (Convexity and Smoothness) . We assume that the loss function l satisfies two conditions: (1) l is convex w.r.t. w h ; (2) l is Lipschitz smooth for w h with a constant α > 0 .

Remark. Assumption 4.1 is easy to meet in practice, and it arises in many linear models (linear regression, SVM etc).

Theorem 1 (Bound Based on TFL-specific Domain Discrepancy) . Under Assumptions 4.1 (Convexity and Smoothness), the optimal target loss is bounded as

where h ∗ ( k ) denotes the optimal local model of client k (see Section 3).

Semantic Interpretation . In Theorem 1 , the optimal target loss L ∗ tgt is bounded by the sum of three terms on the RHS of Equation 7: (1) the averaged optimal local loss 1 K ∑ K k =1 [ L D ( k ) ( h ∗ ( k ) )] ; (2) the intra-discrepancy of the source domain d H ( D fed S ) ; (3) the discrepancy between the source and target domains, d G , F ( D fed S , D T ) . Therefore, the bound on the optimal target loss can be tightened by making the optimal source loss smaller, and by lowering the intra-source and source-target discrepancy. The later two terms can be controlled through regularization as detailed below.

Tightening the Bound via Regularization over F . Given two feature extractor function classes F 1 and F 2 , if F 1 ⊂ F 2 , we have H 1 ⊂ H 2 where H 1 = G × F 1 and H 2 = G × F 2 . From Definition 2 and 3, it is straightforward to see that d H 1 ( D fed S ) ≤ d H 2 ( D fed S ) and d G , F 1 ( D fed S , D T ) ≤ d G , F 2 ( D fed S , D T ) . This indicates that any general regularizer on F can lead to a decrease in the latter two terms of the RHS of Theorem 1. However, shrinking the expressive power of F will inevitably increase the optimal source loss, which is the first term on the RHS of the expression in Theorem 1. Therefore, using general regularization, one has to trade-off the optimal source loss , TFL-specific cross-client discrepancy , and TFL-specific source-target discrepancy .

Based on Theorem 1 and the follow-up discussion, we aim to answer the question: how should one design a practical regularizer that can inherently tighten our bound on TFL? We give an answer to this question in Section 4.2. There, Theorem 2 shows that a transferability-boosting pretraining regularization should decrease the cross-client Jacobian variance while at the same time increase the cross-client averaged Jacobian norm .

4.2 Practical Bounds Based on Cross-Client Statistics

The goal of FL is to estimate the optimal solution f ∗ by updating the global model through multiple federated rounds 1 . Denote the total number of rounds by P and the output global model after round P by f P . Denote the target loss under f P instead of f ∗ as ̂ L ∗ tgt (which is an estimate of L ∗ tgt ). At the end of local training during any round p ≤ P , let h ( k ) p with weight w ( k ) p represent the model of client k , and let h p with weight w p represent the global model. Denote the Jacobian (gradient) of the loss l w.r.t the model weights at domain k as J ( k ) ( w ) = E D ( k ) S ∇ w h l ( h ( x ) , y ) | w h = w .

Throughout the remainder of the manuscript, we use J ( k ) p as shorthand for J ( k ) ( w p ) . Furthermore, we denote the learning rate of agent k at round p by λ ( k ) p . We make a single-step local update assumption (Assumption 4.2) and use the definitions for cross-client statistics (Definition 4) to derive bounds exploiting the cross-client statistics from Lemma 4.1 and Theorem 2).

Assumption 4.2 (Single-Step Local Update) . During local training, all clients perform one step of gradient descent (GD) to update their model for transmission, w ( k ) p +1 = w p -λ ( k ) p +1 · J ( k ) ( w p ) . This is a common assumption in the FL literature [30, 32]. 2

Definition 4 (Cross-Client Statistics) . At federated round p , given K clients with local Jacobians { J ( k ) p } k ∈ [ K ] , we define the cross-client averaged Jacobian norm ∥ J p ∥ 2 and the cross-client Jacobian variance , respectively, as

Note that we assumed the loss function to have a α -Lipschitz continuous gradient. When the gradient is large, α is also large, and to make β 1 ( λ ) ≥ 0 , λ has to be close to 0 . In this case, the absolute value of the second term can be small and that of the third term can be large.

Lemma 4.1 (Loss Bound Using Cross-Client Statistics) . Under Assumptions 4.2 and 4.1, and the cross-client statistics defined in Definition 4, after P rounds of federated pretraining one has

where h 0 is the initial global model and β 2 ( λ ) = αλ 2 2 , β 1 ( λ ) = λ -β 2 ( λ ) .

Note that during the training process , a large Jacobian norm can promote transferability (the Jacobian is expected to be small at the end of training).

Interpretation of Key Terms . Lemma 4.1 shows that the target loss of the finetuned pretrained model (LHS) is bounded by a sum (RHS) involving four key terms: 1) L src ( h 0 ) , the initial source loss ; 2) ∥ J p ∥ 2 , the cross-client averaged Jacobian norm ; 3) σ 2 p , the cross-client Jacobian variance ; 4) d G , F ( D fed S , D T ) , the TFL-specific source-target domain divergence . Since L src ( h 0 ) is fixed throughout the pretraining process, and d G , F ( D fed S , D T ) can be reduced using general regularization, we focus on analyzing the two tunable cross-client statistics, ∥ J p ∥ 2 and σ 2 p .

Influence of the Cross-Client Statistics on the Bound . We note that during pretraining, both the coefficients β 1 ( λ p +1 ) and β 2 ( λ p +1 ) have to be positive (see how to ensure these constraints in Appendix D). Therefore, a larger ∥ J p ∥ 2 and a smaller σ 2 p tighten the upper bound, indicate a lower target loss, and thus suggest better model transferability.

Coefficients Quadratic w.r.t. the Learning Rates . Lemma 4.1 involves coefficients β 1 ( λ ) and β 2 ( λ ) that are quadratic in the learning rate λ ; thus, we can further tighten the bound by optimizing the learning rates across different rounds (Assumption 4.3). The tightened bound is given in Theorem 2.

Assumption 4.3 (Optimal Learning Rates Across Rounds) . In each round p ( 2 ≤ p ≤ P ), we use an optimal learning rate for local training λ ∗ p = K ·∥ J p -1 ∥ 2 2 α · ∑ k ∥ J ( k ) p -1 ∥ 2 2 . A similar analysis and assumptions on optimal learning rate has also been used in [7, 10, 13, 30, 49].

Theorem 2 (Tightened Bound Based on Cross-Client Statistics) . By optimizing the bound in Lemma 4.1 with respect to the learning rates λ as governed by Assumption 4.3, and under Assumption 4.1 (Convexity and Smoothness), the estimated optimal target loss is bounded as follows:

Semantic Interpretation . Theorem 2 indicates that by optimizing the learning rates at each round, the bound for ̂ L ∗ tgt can be made smaller through 1) a smaller source loss L src ( h 0 ) ; 2) a larger cross-client average Jacobian norm ∥ J p ∥ 2 ; 3) a smaller cross-client Jacobian variance σ 2 p , and 4) a smaller source-target domain divergence d G , F ( D fed S , D T ) . These are consistent with Lemma 4.1.

Intuitive Explanation . (a) Increasing the Cross-Client Averaged Jacobian Norm ∥ J p ∥ . In early training, it is crucial to avoid a small J p , as it can 'trap' clients in local minima or cause overfitting of local distributions. Note that increasing J p will not induce an excessively large second term (i.e., a term that → ∞ ) in Theorem 2, as increments are constrained through the Lipschitz condition ∥ J 1 -J 2 ∥ ≤ α ∥ x 1 -x 2 ∥ . In the final stages of training, ∥ J p ∥ naturally decreases due to convergence. (b) Decreasing the Cross-Client Jacobian Variance σ p . A small σ p promotes domain similarity, preventing local overfitting and enhancing global model transferability. (c) Trade-Off Between Increasing ∥ J p ∥ and Decreasing σ p . A larger ∥ J p ∥ induces more substantial local updates but can increase variance σ p +1 . Thus, Theorem 2 underscores balancing this trade-off: enlarging ∥ J p ∥ while maintaining a small σ p .

Challenges in Regularization of the Jacobians . Certain regularization of the Jacobians can induce local Jacobian alignment and reduce σ 2 p (e.g., the regularization of the alignment between local gradients and the global gradient proposed in FedIIR [12]). However, local regularization can naturally impede the growth of local Jacobian norms { J ( k ) p } k ∈ [ K ] and subsequently prevent ∥ J p ∥ 2 from increasing. Thus, such prior regularization may not necessarily improve transferability. As an example, consider the extreme case where σ 2 p = 0 . Then, the bound in Theorem 2 takes the form ̂ L ∗ tgt ≤ L src ( h 0 ) -∑ P -1 p =0 2 α ∥ J p ∥ 2 2 + d G , F ( D fed S , D T ) , where a small ∥ J p ∥ 2 can severely degrade the bound. This indicates that boosting transferability by prior regularization, such as controlling σ 2 p itself, has limitations, and tuning ∥ J p ∥ during the pretraining stage is of crucial importance. We provide a solution to this problem in Section 5.

5 Our Algorithm

Algorithmic Solution for the Theoretical Challenges . From Lemma 4.1 and Theorem 2, we can see that certain round-wise FL-specific statistics, the cross-client averaged Jacobian norm ∥ J p ∥ 2 and cross-client Jacobian variance σ 2 p control the bound on the target loss. The challenge is that, while σ 2 p can be reduced using straightforward techniques (i.e., such as gradient alignment from FedIIR [12]), such techniques unavoidably prevent J p from increasing properly. We therefore propose our FedGTST approach which reduces σ 2 p while enlarging ∥ J p ∥ 2 . A round-wise description of FedGTST is given in Algorithm 5.

Tuning the Cross-Client Jacobian Variance σ 2 p . The cross-client Jacobian variance is controlled via regularization at the local client level (Line 5 of Algorithm 5). The local clients, upon receiving a guide norm γ from the server (explanation deferred to subsequent sections), implement a regularized local training protocol. While client k during standard local training uses the objective L D ( k ) ( h ) , during regularized local training he/she/they use

instead, where ξ is the penalty coefficient for the regularization term. This type of regularization intuitively aligns each local Jacobian norm ∥ J ( k ) ∥ with the guide norm γ , which results in a σ 2 J that is smaller than that of standard FL (i.e., FedA VG).

Algorithm 1 FedGTST (Round p )

1: Randomly select a set ϕ ⊂ [ K ] for regular training.

2: while k ≤ K , Client k should do

3: Receive the guide norm γ p -1 and global model h p -1 from the server.

4: Initialize the local model to h p -1 .

5: Update the local model to h ( k ) p by training with L D ( k ) ( h ) + ξ · ( ∥ J ( k ) ( h ) ∥ 2 -γ ) 2 .

S

6: Calculate the surrogate Jacobian norm γ ( k ) p = γ reg, ( k ) p = ∥ J ( k ) p ∥ 2 .

7: if k ∈ ϕ then

8: Train with L D ( k ) ( h ) to obtain Jacobian norm γ std, ( k ) p .

9: Update the surrogate Jacobian norm γ ( k ) p = max ( γ reg, ( k ) p , γ std, ( k ) p ) .

10: end if

11: Transmit the model h ( k ) p and norm γ ( k ) p to the server.

12: end while

13: The server aggregates the client models into a global model h p and sets the guide norm to γ p = max k ( γ ( k ) p ) .

14: The server broadcasts h p and γ p to all clients.

Tuning the Cross-Client Averaged Jacobian Norm ∥ J p ∥ 2 . The cross-client averaged Jacobian norm can be increased via an exchange protocol that includes: 1) clients calculating surrogate norms for transmission; 2) the server computing and broadcasting a guide norm; 3) clients performing local alignment using the guide norm.

Surrogate Norms . A decrease in the local Jacobian norm ∥ J ( k ) ∥ prevents the cross-client averaged Jacobian norm ∥ J ∥ from growing. We mitigate such a decrease by forcing a small portion of the clients to implement both regularized training and standard training (Line 1), to generate a pair of Jacobian norms γ reg, ( k ) and γ std, ( k ) . The client than chooses the larger norm, γ ( k ) = max ( γ reg, ( k ) , γ std, ( k ) ) as a surrogate norm for transmission (Lines 6 to 10).

Server Guide Norm . The clients send their local Jacobian norms γ ( k ) := ∥ J ( k ) ∥ 2 to the server, and the server broadcasts the largest norm received as its guide norm γ := max k γ ( k ) (Line 13).

Local alignment . The local regularizer from Equation (10) reduces the variance, but can also force an increase of the averaged Jacobian norm. It forces all local Jacobian norms to align with the guide norm γ (Line 5). Since γ has been boosted both at the clients' and server levels, the alignment leads to larger local Jacobian norms.

Communication Cost. Most TFL methods, as already described, require large communication overheads between the clients and the server. For example, FedIIR requires exchanging Jacobians, which have the same dimension as the model weights; FedGTST only requires exchanging norms.

6 Experiments

6.1 Experimental Setting

Transfer tasks. We investigate three transfer tasks utilizing fully-annotated data: a) MNIST [9] to MNIST-M [11], b) CIFAR-10 [19] to SVHN [33], and c) cross-domain transfer in DomainNet [37]. All transfer tasks have been used as benchmarks in existing TL research [3, 8, 11, 22, 28, 50] (also, see Appendix G). Moreover, DomainNet is a standard large-scale dataset for TFL studies, also used by FedSR.

Non-iid Distributed Source (Local) Domains . The pretraining phase is conducted via FL on source (local) domains. Marginal distribution shift is an important phenomenon in FL [27] since the access to classes (or categories) is not the same for all participating entities. However, some federated datasets tested by existing transferable FL methods do not reflect marginal distribution shifts when constructing source local domains (i.e., in the Rotated-MNIST benchmark in FedSR[34], all clients have access to all classes). To address this issue, we employ the following methods for constructing non-iid local domains:

For MNIST or CIFAR-10, unless otherwise specified, we follow the approach in [27] and let each client have access to only a subset of the categories. The category selection rule and data sampling method is described in Appendix H. Besides, we run additional experiments for Dirichlet sampling, a method commonly used in FL [1, 20, 23, 24, 29, 45, 46, 54].

For DomainNet, which compromises six distinct domains, we follow the leave-one-out strategy used in FedSR: one domain is designated as the target domain, while the remaining five domains serve as source domains, with each assigned to an individual client.

Federated System Size . Large system sizes are inherent to FL systems, where the number of clients can be as large as 100 [25, 27]. In such a case, it is more challenging for the global model to achieve good performance. However, existing TFL methods typically use a very small number of domains ( ≤ 5 for FedSR and StableFDG). In contrast, we allow the system size to cover a broad range of values, including 10 (small), 50 (medium), and 100 (large) clients.

Evaluation matrix and backbones . We measure domain transferability via acc tgt , the accuracy of a pretrained model finetuned on the target domain. For backbone selection, we use both LeNet [21] and ResNet18[14] to represent different levels of backbone complexity.

Baselines . We consider the following TFL algorithms as our main baselines: FedAVG, FedSR, and FedIIR. Furthermore, we also compare our findings to Scaffold [16], an advanced FL approach used to address data heterogeneity. We do not report results for FedADG, FedCDG, FedGTST, and StableFDG since they do not ensure privacy and/or underperform compared to the main baseline.

Settings for Pretraining (FL on source local domains) . 1) Local epochs . To conform with our theoretical assumptions, unless specified otherwise, we set the local epoch of each client to 1 . We also investigate the system performance with 10 local epochs in Appendix C. 2) Number of participants per round . We allow 50% of the clients to participate in each round (e.g., if K = 50 , there are 25 participants per round). 3) Number of clients performing standard local training . To boost the Jacobian norm, a subset of clients conducts standard local training (we set the subset size to 10% of the total number of clients).

Additional Settings . Unless specified otherwise, for local training on the source datasets we use the Adam [17] optimizer with coefficients ( β 1 , β 2 ) = (0 . 9 , 0 . 999) (note that these β values are not to be confused with the coefficients from our theoretical analysis). Our initial learning rate equals 0 . 01 and then decays by a factor of 10 per 50 rounds, with an early stop trigger of 10 rounds. We apply the standard cross-entropy loss. We also set the pretraining batch size to 256 for MNIST → MNIST-M and to 128 for CIFAR10 → SVHN. For both tasks, we use a finetuned learning rate 0 . 005 , with weight decay 0 . 0001 . All results reported in Section 6.2 are averaged over three runs.

6.2 Results

Transferability results . FedGTST exhibits significantly improved transfer performance when compared to baselines across a range of tasks, system sizes, and backbone architectures. For the example of CIFAR10 → SVHNwith 100 clients and a LeNet backbone, FedGTST outperforms FedIIR by 7 . 6% and FedSR by 9 . 8% . The results for a small, medium and large federated system ( K = 10 , K = 50 and K = 100 ) are reported in Tables 1, 2 and 3, respectively.

Discussion. Besides the significant performance gain of FedGTST over baseline methods, we also observe that 1) for transfer tasks in which the backbone and method are fixed, transferability generally decreases with the system size. Importantly, FedGTST improves the baselines more significantly

for large system sizes; 2) when the system size, transfer task and the method are fixed, the more 'complex' the backbone (ResNet18 vs LeNet), the better the transferability.

Additional results . We defer reporting and discussing additional results in Appendix C, which include (a) an investigation on hyper-parameter sensitivity the convergence speed , and cross-client statistics , (b) results regarding DomainNet , Dirichlet Sampling , and Scaffold .

7 Limitations

Increased Local Computational Cost . While FedGTST only induces a negligible communication overhead among clients and the server, we note that at the level of a small subset of clients we need to conduct both regularized training and standard training to boost ∥ J p ∥ 2 . This increases the local computational burden but to a very small extent. Nevertheless, in future works, we will explore algorithms with lower local computational costs.

Potentially Loose Bounds . Although existing studies have added to our understanding of transferable federated learning, our work is the first to derive a bound on a direct measure of transferability (the target loss). We believe that the bound can be tightened.

8 Conclusion

We introduced FedGTST, a federated learning algorithm aimed at enhancing global transferability. Inspired by theoretical insights, FedGTST integrates cross-client information on averaged Jacobian norms and Jacobian variance. Our work addresses key challenges in existing methods, such as privacy violations and an overemphasis on local transferability. Experimental results demonstrate significant performance improvements over baseline models.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations (ICLR) , 2021. URL https://openreview.net/forum?id=B7v4QMR6Z9w .

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, David Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046 , 2019.

Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems , 29, 2016.

Junming Chen, Meirui Jiang, Qi Dou, and Qifeng Chen. Federated domain generalization for image recognition via cross-client style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 361-370, 2023.

Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science , 519:103-126, 2014. ISSN 0304-3975. doi: https://doi.org/10.1016/j.tcs.2013.09.027. URL https://www.sciencedirect.com/ science/article/pii/S0304397513007184 . Algorithmic Learning Theory.

Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. Journal of Machine Learning Research , 20(1):1-30, 2019.

Laizhong Cui, Xiaoxin Su, Yipeng Zhou, and Jiangchuan Liu. Optimal rate adaption in federated learning with compressed communications. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 1459-1468. IEEE, 2022.

Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV) , pages 447-463, 2018.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461 , 2020.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research , 17(59):1-35, 2016.

Yaming Guo, Kai Guo, Xiaofeng Cao, Tieru Wu, and Yi Chang. Out-of-distribution generalization of federated learning via implicit invariant relationships. In International Conference on Machine Learning , pages 11905-11933. PMLR, 2023.

Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425 , 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjuneta Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 , 2019.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5132-5143. PMLR, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jakub Koneˇ cný, H Brendan McMahan, Daniel Ramage, Peter Richtárik, Ananda Theertha Suresh, et al. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575 , 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning at scale. In International Conference on Machine Learning (ICML) , 2022. URL https://proceedings.mlr.press/v162/lai22a.html .

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

Seunghun Lee, Sunghyun Cho, and Sunghoon Im. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15252-15261, 2021.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10713-10722, 2021. doi: 10.1109/CVPR46437. 2021.01057. URL https://openaccess.thecvf.com/content/CVPR2021/papers/Li_ Model-Contrastive_Federated_Learning_CVPR_2021_paper.pdf .

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR , pages 10713-10722, 2021. doi: 10.1109/CVPR46437.2021. 01057. URL https://openaccess.thecvf.com/content/CVPR2021/papers/Li_ Model-Contrastive_Federated_Learning_CVPR_2021_paper.pdf .

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10713-10722, June 2021.

Tianqing Li, Anup Sahu, Ameet S Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning , pages 97-105. PMLR, 2015.

Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. In Advances in Neural Information Processing Systems , volume 34, pages 6072-6084, 2021.

Evelyn Ma, Praneet Rathi, and S Rasoul Etesami. Local environment poisoning attacks on federated reinforcement learning. arXiv preprint arXiv:2303.02725 , 2023.

Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430 , 2009.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

A Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing Systems , 35:38831-38843, 2022.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359, 2010.

Jungwuk Park, Dong-Jun Han, Jinho Kim, Shiqiang Wang, Christopher Brinton, and Jaekyun Moon. Stablefdg: Style and attention based learning for federated domain generalization. Advances in Neural Information Processing Systems , 36, 2024.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision , pages 1406-1415, 2019.

Mohammad Pezeshki, Chengxu Le Lan, Thibaut Durand, Guillaume Lajoie, Simon LacosteJulien, Aaron Courville, and David Lopez-Paz. Understanding hessian alignment for domain generalization. arXiv preprint arXiv:2308.11778 , 2023.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18347-18377. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/ v162/rame22a.html .

Sebastian Ruder, Ivan Vulic, and Anders Søgaard. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research , 65:569-631, 2019.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems , 33, 2020. URL https://par.nsf.gov/biblio/10283431 .

Yan Shen, Jian Du, Han Zhao, Zhanghexuan Ji, Chunwei Ma, and Mingchen Gao. Fedmm: A communication efficient solver for federated adversarial domain adaptation. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 1808-1816, 2023.

Yuge Shi, Jeffrey Seely, Philip H. S. Torr, Siddharth Narayanaswamy, Awni Y. Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations (ICLR) , 2022. URL https://openreview.net/ forum?id=_7w5JXy8VJM .

Virginia Smith. Federated learning: strategies for improving communication efficiency. IEEE Signal Processing Magazine , 34(6):36-43, 2017.

Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. arXiv preprint arXiv:2206.02465 , 2022.

Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, and Xiaowen Chu. Fedimpro: Measuring and improving client update in federated learning. In The Twelfth International Conference on Learning Representations (ICLR) , Vienna, Austria, May 7-11 2024. OpenReview.net.

Lisa Torrey and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques , pages 242-264. IGI global, 2010.

Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data , 3(1):1-40, 2016.

Xidong Wu, Feihu Huang, Zhengmian Hu, and Heng Huang. Faster adaptive federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1037910387, 2023.

Xiaojun Xu, Jacky Y Zhang, Evelyn Ma, Hyun Ho Son, Sanmi Koyejo, and Bo Li. Adversarially robust models may not transfer better: Sufficient conditions for domain transferability from the view of regularization. In International Conference on Machine Learning , pages 24770-24802. PMLR, 2022.

Qiang Yang, Yang Liu, Tianjian Chen, Yu Tong, et al. Federated optimization: Distributed optimization beyond the datacenter. IEEE Signal Processing Magazine , 36(4):49-59, 2019.

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems , 27, 2014.

Xinhui Yu, Dan Wang, Martin McKeown, and Z Jane Wang. Contrastive-enhanced domain generalization with federated learning. IEEE Transactions on Artificial Intelligence , 2023.

Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. DENSE: Data-free one-shot federated learning. In Advances in Neural Information Processing Systems , volume 35, pages 1911219126, 2022. URL https://papers.nips.cc/paper_files/paper/2022/hash/ 868f2266086530b2c71006ea1908b14a-Abstract-Conference.html .

Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization. arXiv preprint arXiv:2111.10487 , 2021.

Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing Systems , pages 8568-8579, 2018.

Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference on machine learning , pages 7523-7532. PMLR, 2019.

Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. The Journal of Machine Learning Research , 23(1):15356-15404, 2022.

A Additional preliminaries

Definition 5 ( H -discrepancy [6, 57]) . Given a model function class H and two data distributions D S , D T , the H -discrepancy between D S , D T is defined as

Remark. This type of discrepancy, as well as the related H -divergence, have been frequently used in the analysis of domain adaptation [5, 31, 57], and we follow this trend.

B Proof of our bounds

Lemma B.1 (Theorem 2.5 in [50]) . Suppose we perform pretraining on the source domain D S to obtain f and g at the server, and fine-tune the model on the target domain D T to obtain a new classifier g T . We then have

Equation (12) shows that the transfer loss can be upper bounded by the sum of the loss on the source domain and the divergence between two domains.

Lemma B.2. With Assumption 4.1 (Convexity and Smoothness) and Definition 2 (Cross-Client Divergence), we have

Proof. The loss function l is assumed to be convex w.r.t. the parameters of the model h , where the parameters are denoted by w h . Based on the aggregation rule in Section 3, we have w h = 1 K ∑ k w ( k ) h , where w h and w ( k ) h stands for the weights for global model h and that for local model h ( k ) , respectively. We therefore have

̸

where 1 K 2 ∑ k 1 ,k 2 d H ( D ( k 1 ) , D ( k 2 ) ) ≤ 1 K ( K -1) ∑ k 1 = k 2 d H ( D ( k 1 ) , D ( k 2 ) ) = d H ( D S ) as in Definition 2. Here ( a ) follows from the convexity assumption for l w.r.t the parameters, while ( b ) follows

from Lemma B.1 based on the argument below:

Theorem 3 (Theorem 1) . Under Assumptions 4.1 (Convexity and Smoothness), the optimal target loss is bounded by

where h ∗ ( k ) denotes the optimal local model of client k (see Section 3).

Proof. Since we have

Using Lemma B.2, we have

Lemma B.3 (Bound on round-wise source loss) . Suppose the learning rates of all clients at round p are the same, i.e., λ ( k ) p = λ p , ∀ k ∈ [ K ] , p ∈ [ P ] . Under Assumptions 4.2 and 4.1, we have that

where J p = 1 K ∑ k J ( k ) p , σ 2 p = 1 K ∑ k ∥ ∥ ∥ J ( k ) p ∥ ∥ ∥ 2 2 -∥ ∥ ∥ 1 K ∑ k J ( k ) p ∥ ∥ ∥ 2 2 , and β 1 ( λ ) = λ -β 2 ( λ ) , β 2 ( λ ) = αλ 2 2 .

Proof. Following the same proof idea as for Lemma B.2, we have

By definition, we also have

From the update rule of GD, we can write

In (18), ( x i , y i ) are sampled from D ( k 2 ) , and not to be confused with ( x, y ) sampled from D ( k 1 ) . Define ∆ := -λ p +1 · J ( k 2 ) p , we can upper bound the loss in (19) by

The inequality holds since l is assumed to have α -Lipschitz continuous gradient. Combining (17) and (20), we have

Next, by (16) and (21), we have

The last equality follows from the definition of J p .

Define σ 2 p := 1 K ∑ K k =1 ∥ ∥ ∥ J ( k ) p ∥ ∥ ∥ 2 2 -∥ J p ∥ 2 2 . Then, we have

which completes the proof.

Lemma B.4 (Lemma 4.1) . Under Assumptions 4.2 and 4.1, and the cross-client statistics defined in Definition 4, after P rounds of federated pretraining one has

where h 0 is the initial global model and β 2 ( λ ) = αλ 2 2 , β 1 ( λ ) = λ -β 2 ( λ ) .

Proof. From Lemma B.3, we have

Following the same proof as the one outlined for Theorem 1, we obtain

Therefore,

C Additional Experimental Results

Computing resources . We used an NVIDIA GeForce RTX 3090 Ti GPU with a memory of 24247MiB. One run of experimental evaluation takes approximately 6 hours for CIFAR10 → SVHN for the small client setting.

Fraction of Clients Participating in FL Rounds . Besides the results generated using 20% of the participating clients reported in the main text, we also report the results pertaining to 10% and 100% participating clients in Table 4. A larger fraction of participants helps with improving transferability, i.e., as expected, 100% participation outperforms the setting with 10% participation. However, we note that for 20% of participating clients we already reach a performance comparable to that involving 100% of the participants.

Convergence Results. Convergence results are plotted in Figure 1. We observe that a model pretrained via FedGTST not only offers better transferability than baselines, but also tends to converge faster during the finetuning stage (i.e., the green lines in all plots always converge faster than the grey dashed lines).

Local Epochs . Besides the results for single local client epochs presented in main text, here we also report the results for multiple local epochs in Table 5. A smaller number of local epochs tends to improve transferability more than a larger number of epochs, which is consistent with the intuition that FL pretraining can avoids local overfitting when using fewer epochs.

Cross-Client Statistics . We plot the cross-client averaged Jacobian norm ∥ J p ∥ 2 and the cross-client Jacobian variance σ 2 p in Figure 2. The observation is that FedGTST leads to a significantly larger ∥ J p ∥ 2 and a significantly smaller σ 2 p compared to FedAVG. A more detailed explanation of the findings (i.e., the reason for truncating the x-axis in the left plot, the FedGTST coefficient selection approach and faster convergence results in the right plot) is available in the caption of Figure 2.

Additional Dataset: DomainNet . Following FedSR, we apply a leave-one-out strategy, where one domain is treated as the target domain and the other five are source domains, allocated to five

FinetunePerformance onTargetDomain

FinetunePerformanceonTargetDomain

0.60

0.60

0.55

0.55

trn_acc

tst_acc

0.50

0.50

EJ

E.J

0

0.45

0

0.45

5e-4

5e-4

1e-4

0.40 -

1e-4

0.40 -

1e-3

1e-3

0.35 -

20

0.35-

10

FinetuneEpochs

40

50

0

10

FinetuneEpochs

20

30

40

50

FinetunePerformanceonTargetDomain

FinetunePerformance onTargetDomain

0.64

0.66 -

S.J

S.J

0.64 -

0

0.62

0

0.62 -

1e-4

0.60

1e-4

0.60-

5e-4

5e-4

1e-3

1e-3

0.58-

0.56

0.56-

0.54

0.54 -

0.52-

0.52 -

0.50

0

10

20

30

40

0

10

20

30

40

Finetune Epochs

Finetune Epochs

individual clients. Results are reported in Table 6, where the target domains listed are: C (Clipart), I (Infograph), P (Painting), Q (Quickdraw), R (Real), and S (Sketch).

Additional Baseline: Scaffold . FedGTST consistently outperforms Scaffold, as presented in Table 7 and 8.

Dirichlet Sampling . We set the concentration parameter to 0.5 and the number of parties to 10 by default. The results in Table 9 and 10 indicate that FedGTST still outperforms others when individual domains are constructed via Dirichlet sampling.

FedAvg

8

350

JNB_e-3

300

NormofMeanGradient

250

JNB_coef=1e-3

JNB_coef=5e-4

4

JNB_coef=1e-4

200

FedAVG

150

2

100

0

50

0

20

40

60

80

100

0

20

40

60

80

100

federated traininground

Federatedrounds

D Discussion of the Positivity of Coefficients Requirement for Cross-Client Statistics

Lemma D.1 (Bound on round-wise source loss) . Suppose the learning rates of all clients at round p are the same: λ ( k ) p = λ p , ∀ k ∈ [ K ] , p ∈ [ P ] . When Assumption 4.2 and 4.1 hold, we have that

Remark. Lemma D.1 provides an upper bound of current-round cross-client loss (the proof is given in Appendix B), indicating that a smaller current-round cross-client loss (LHS) may be influenced by factors such as a small last-round cross-client loss , a large cross-client average Jacobian norm , and a small cross-client Jacobian variance (RHS). More precisely, the LHS is the cross-client current-round loss , which is upper bounded by three terms on the RHS,

The Loss Term : L src ( h (0) p ) is the last-round cross-client loss .

The Variance Term : σ 2 p measuring the variance of local gradients across all nodes.

The Norm Term : ∥ J p ∥ 2 2 measures the squared cross-client average Jacobian norm .

Positivity of the Coefficients . It is straightforward to see that β 2 ( λ p +1 ) = α ( λ p +1 ) 2 2 , the coefficient in front of σ 2 p , is always positive. Therefore, we focus on β 2 ( λ p +1 ) , the coefficient in front of ∥ J p ∥ 2 . Denote the RHS of Equation 25 by B , so that L src ( h p +1 ) ≤ B . To allow such a bound to be used as an indicator that the source loss is decreasing with the number of training rounds, we require

B ≤ L src ( h p ) , since only in this way can the bound give L src ( h p +1 ) ≤ B ≤ L src ( h p ) . Thus, we require β 1 ( λ p +1 ) to be positive, since otherwise B ≤ L src ( h p ) cannot be meet. We describe in what follows that a positive β 1 ( λ p +1 ) is indeed possible in practice.

Realistic scenarios in which β 1 ( λ p +1 ) > 0 . To require β 1 ( λ p +1 ) > 0 is equivalent to require λ p +1 < 2 α . This requirement can be easily met since for any model of a known mathematical form based on a second-order differentiable loss, we can easily get the lower bound for α once we observed all training data. Using linear regression as an example, where l ( x, y ; w ) = ( wx -y ) 2 , we have d 2 l dw 2 = 2 x , therefore, in this case we can simply control λ p +1 ≤ 1 max x ∈X ∥ x ∥ to approximately guarantee λ p +1 < 2 α . This then ensures β 1 ( λ p +1 ) > 0 .

E Optimal Learning Rate

Choosing the Optimal Learning Rate . Lemma B.3 shows that the upper bound of federated loss at round p is a quadratic function w.r.t the learning rate λ p . Therefore, a good learning rate at each round needs to be chosen to minimize the upper bound. For simplicity of notation, we use B p +1 ( λ p +1 ) as a shorthand for the upper bound shown in (15).

The following two observations are in place for B p +1 ( λ p +1 ) :

When 0 < λ p +1 < 2 K ·∥ J p ∥ 2 2 α · ∑ k ∥ J ( k ) p ∥ 2 2 ≤ 2 α , it holds that L src ( h (0) p +1 ) ≤ B p +1 ( λ p +1 ) < L src ( h (0) p ) , indicating that the federated loss is decreasing with the number of rounds.

By minimizing B p +1 ( λ p +1 ) w.r.t. λ p +1 , we have

F Stochastic Gradient Descend

Assumption 4.2 on one step of gradient descent can be extended to stochastic learning with batch sampling. The additional randomness in sampling would require incorporating the variance of batch sampling into the generalization bound. This variance term, being independent of the algorithm design, was omitted in our theoretical analysis.

G Dataset Description

MNIST comprises 60 , 000 28 × 28 grayscale images of handwritten digits (0 through 9); MNIST-M is created by combining MNIST digits with the patches randomly extracted from color photos of BSDS500 as their background, containing 59 , 001 training images. The CIFAR-10 dataset consists of 60 , 000 32 × 32 colour images from 10 classes. SVHN contains 73 , 257 32 × 32 colored digits obtained from house numbers in Google Street View images.

H Non-iid FL Models

For the source domains MNIST and CIFAR-10, we only allow each local client to have access to two out of ten classes (e.g., for the digit dataset (0-9), one client may only have access to say digits 3 and 6). We let each client randomly chooses their labels and samples following a uniform distribution. See the example in Fig. 3

I Additional Related Work

I.1 Gradient Matching in Transfer Learning

Discussion. Gradient matching in transfer learning focuses on aligning gradients between source and target domains to facilitate better domain adaptation. Shi et al. [43] demonstrate that matching gradients across domains enhances domain generalization by making the learned representations more resilient to domain shifts. Extending this idea, Rame et al. [39] introduce the concept of invariant gradient variances, which helps maintain generalization performance even for out-of-distribution settings. The work in Pezeshki et al. [38] further highlights the impact of Hessian alignment, showing that aligning the Hessians of the source and target domains can significantly boost generalization in gradient-based methods.

Challenges. Applying gradient alignment techniques directly to FL presents several challenges: (1) Privacy leakage -these methods can potentially compromise data privacy by necessitating access to the target domain from source domains; (2) Local overfitting -clients train their models on local data, which can lead to overfitting within their specific domains, reducing the global model's generalization capabilities.

Our contribution. To address the issues described above, we propose communication schemes that ensure data privacy by preventing unrestricted access to client data. Furthermore, our approach promotes global transferability by focusing on improving generalization across all clients rather than solely enhancing local domain performance.

I.2 Distinctions and Connections between Generalization and Transferability in FL

Both approaches address the challenge of non-iid data, and improving transferability may potentially enhance generalization across diverse local domains. However, they employ distinct models and

evaluation datasets: (a) Generalization of FL targets performance on heterogeneous source testing datasets , while transferable FL aims for strong performance on a target dataset that may significantly differ from source training datasets . (b) Heterogeneous FL utilizes the pretrained model for evaluation, whereas Transferable FL assesses the finetuned model.

The methodologies also diverge. Although reducing cross-client variance is linked to better generalization, our method distinguishes itself from traditional heterogeneous FL by enforcing a large average Jacobian norm | J p | in the early stages. While a larger | J p | may hinder generalization due to increased local updates and model variance, it enhances transferability by preventing premature local convergence.

Additionally, adversarially robust models offer an example where improving transferability may come at the expense of generalization. As shown in [41], adversarially robust models tend to have better transferability. However, since these models are designed to perform well against adversarial examples or perturbations, they do not necessarily exhibit lower generalization error on clean test data. In this scenario, transferability can be unrelated to or even conflict with generalization.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction reflect the facts that 1) we theoretically established that certain cross-client statistics can boost transferability of federated learning, and that 2) based on our theoretical findings, we proposed a novel transferable federated learning method which significantly outperforms baselines.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in a separate section, Limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provided necessary assumptions need for our theoretical analysis in the main text but deferred the proofs to the Appendix due to space limitations. We nevertheless discussed the intuition behind our findings in the main text.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The datasets are existing open-source benchmarks, and we have included our code in the Supplementary materials document.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets are existing open-source benchmarks, and we have attached our code in a Supplementary materials document.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details of the experimental work are discussed in the Experimental Setting in the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experimental results are test accuracies, and we reported the standard deviations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Sufficient information about the compute resources (type of compute workers, memory, time of execution) is provided in the Experiments section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted to produce the paper conformed, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is neither a negative or positive societal impact of the work performed. It still may have a technological and academic impact based on its results and findings. There should be no problems with the proper use of the pertinent methods/software.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no risk in this domain.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original papers that produced the dataset used in our studies are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper will not lead to the release of new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this paper, the authors proposed a method to address the problem of transfer learning under FL. Specifically, after a thorough theoretical analysis, the authors observe that the cross-client averaged Jacobian norm and its variance control the bound on the target loss. Therefore, they propose FedGTST to properly increase the norm while reduce its variance. The proposed method achieves promising results on several benchmark datasets.

STRENGTHS:
1. The structure and writting of the paper is clear.
2. The proposed method is effective, and the executed experiments are thorough.
3. The theoretical analysis is sound.

WEAKNESSES:
1. The adopted benchmark datasets are a bit easy. Since the authors already used ResNets in the experiments, comparing on more complex datasets, e.g., PACS, DomainNet would make the experimental results more convicing.
2. From the definition introduced in Section 3, only the classifier will be optimized during the federated communication. Is the proposed method also effective when the feature extractors are also optimized during FL?
3. Is the proposed method also applicable to settings when there exists mutliple source domains?

QUESTIONS:
Please refer to the Weaknesses section.

LIMITATIONS:
The authors have discussed the limitations of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses transferability of models pre-trained using Federated Learning (FL). Through a thorough theoretical analysis, the authors identify cross-client averaged Jacobian norm and cross-client Jacobian variance as a key factor influencing transferability. To retain good transferability,  FedGTST controls cross-client averaged Jacobian norm through a guide norm determined by the server while the cross-client Jacobian variance is controlled through local regularisation at the clients based on the guide norm. Different from existing works, FedGTST prevents privacy leakage, and maintains communication efficiency by only communicating additional scalars.\n\nSTRENGTHS:\n- Thorough theoretical analysis quantifying loss on target domain in terms of the constituents from FL training.\n- Proposed algorithm is communication efficient, satisfying a crucial requirement of federated settings.\n- Proposed algorithm requires clients to transfer only scalar values in addition to locally updated parameters, preventing any further privacy leakage beyond sharing model parameters.\n- Empirical results cover different model sizes, networks with up to 100 clients and non-IID local domains, simulating diverse FL setups.\n\nWEAKNESSES:\n1) The variance of cross-client Jacobian appears to be a property of the non-IID local domains. For instance, when clients have IID data, the variance would be inherently low and thus naturally promote good transferability. Thus, when considering non-IID local domains, the proposed algorithm should be compared for transferability against advanced FL algorithms that address data heterogeneity and therefore would naturally control cross-client Jacobian variance, e.g. FedAdam, Scaffold and FedNova. FedAvg seems to be a weaker baseline.\n\n2) While the theoretical analysis identifies cross-client Jacobian as a key influencing factor for transferability, intuitive explanations regarding the same are missing. Why does inflating norms help transferability? Does a higher or lower average norm during the initial stages affect the generalisation error of the FL pre-training and consequently transferability? If this is true, then the quest reverts back to learning good FL models under non-IID data instead of transferability. It would be great if authors could provide detailed insights/discussion regarding this.\n\n3) Certain assumptions in the analysis seem too strong and could be impractical – i) one local step (assumption 4.2) and ii) Gradient Descent (GD) instead of Stochastic Gradient Descent (SGD) (assumption 4.2). FL hinges on the idea of executing multiple local steps to achieve communication efficiency. Furthermore, most standard FL/ML algorithms execute SGD due to the well known computationally expensive nature of GD which evaluates a single gradient on the entire local dataset. The authors should clarify the implications of loosening these two assumptions on the theoretical analysis.\n\nMinor comments:\n\n1) The paper has several typos making reading comprehension difficult:\n- Line 136: “does not communicating information that …” \n- Line 319: “We do not report results for … FedGTST …”\n- Line 8 in Algorithm: $\\gamma_p^{unl, (k)}$\n- Line 523 (Incorrect reference): “Convergence results are plotted in Figure C”. \n\n2) Convexity assumption is certainly not easy to meet in practice for today’s ML. Neural network loss functions are non-convex which are the heart of most of the modern ML.\n\nThe reviewer is happy to reconsider the rating once the authors address/clarify the above issues.\n\nQUESTIONS:\n1) Does the experimental setup consist of 1 local step or 1 local epoch (similarly in Table 5)?.  The theoretical analysis assumes one local step, however the experiments mention one local epoch. \n2) On the same lines, does the experimental evaluation use GD or SGD? Can the authors show empirical evaluations with SGD and multiple local epochs if not already?\n3) What is the effect of removing the surrogate Jacobian norm evaluation with standard training? Can the authors show an ablation study where there are no clients computing $\\gamma_p^{std, (k)}$ and clients only send $\\gamma_p^{reg, (k)}$ ?\n\nLIMITATIONS:\n1) Missing important non-IID FL baselines (comment (1) in weaknesses).\n2) Lacking discussion of cross-client average Jacobian norm and its relation to data heterogeneity (comment (2) in weaknesses).\n3) Certain assumptions seem impractical (comment (3) in weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents FedGTST, a novel federated transfer learning approach that achieves transferability over the global learning domain and accurate assessment of the degree of transferability. The proposed method utilizes a new regularizer that encodes cross-client statistics and adjusts the local training process towards global transferability. This is achieved by subtractions of global norms of Jacobians communicated by the server. The authors also use two FL-specific factors, cross-client Jacobian variance and cross-client Jacobian norm as direct performance indicators. The proposed method only communicates Jacobian norms, which is a scalar, thus achieving reduced communication overhead and ensuring privacy. The authors also provide theoretical analysis and experimental results demonstrating the effectiveness of FedGTST over existing baselines.\n\nSTRENGTHS:\nOverall this paper is well written and tecchnically sound. The authors present a detailed literature review of existing work related to Federated Transfer Learning and point out three major drawbacks: privacy leakage, local overfitting and huge communication bottleneck. \n\nFedGTST has the following contributions:\n1. The studied problem is novel and may have practical usage in real-world applications.\n2. It uses a novel regularizer that focuses on the cross-client statistics to enhance the global transferability.\n3. It introduces the cross Jacobian variance and norm as the direct indicator of the transferability.\n4. It communicates only the scalar to reduce communication overhead and protect the privacy.\n\nThis paper also provides detailed and comprehensive analysis on the literature of Federated Transfer Learning, which eases the understanding for readers. Rigorous theoretical analysis and extensive experiments are also included to supports the proposed method.\n\nWEAKNESSES:\nWeaknesses\n1. The non-iid setting is not common. Most of FL methods utilize the dirichlet sampling to simulate the Non-IID distribution [1-7].\n2. Estimating the quality of the solution f* and using idea from the domain generalization is interesting. Ref [7] also considers measuring the local training through the lens of the domain gap, which can be added for discussion.\n3. The hyper-parameters are not given, which reduces the reproducibility of this work.\n4. The assumption 4.3 require the optimal learning rate, which may require delicate hyper-parameter tuning.\n5. It would be great if the authors add conceptual explanations for the proposed method.\n\n\n[1] Model-contrastive federated learning. In CVPR 2021.\n[2] No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data. In NeurIPS 2021.\n[3] Federated Learning Based on Dynamic Regularization. In ICLR 2021\n[4] Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. In ICML 2022.\n[5] DENSE: Data-Free One-Shot Federated Learning. In NeurIPS 2022.\n[6] FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In ICML 2022.\n[7] FedImpro: Measuring and Improving Client Update in Federated Learning. In ICLR 2024\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes, the authors discussed the limitations of this work, including local computational cost and potentially loose bound.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to improve the transferability of pre-trained models under federated learning. It established a cross-domain generalization error bound under federation. Motivated by this, the authors devise a regularization method leveraging cross-client statistics to minimize the generalization error upper bound. They evaluate the performance of the proposed method on several cross-domain transfer learning tasks.\n\nSTRENGTHS:\n- The reviewer appreciates the theoretical analysis provided in this paper, despite that this can only be achieved on a convex model. The authors did a good job of connecting these theoretical results to their method design. \n- The proposed method is more efficient in terms of communication cost than existing works for cross-domain generalization.\n\nWEAKNESSES:\n- Some important experimental settings are missing. It is not clear that (1) how the pre-trained models on source dataset are obtained (from centralized or federated pre-training?), (2) the availability of target domain data or labels. These important details are absent from the main text, appendix and code description, making it hard to understand the context of this work. The reviewer can only infer the task from some fragment of information. For example, given the evaluated experiment on CIFAR10 -> SVHN, the reviewer assumes the target domain data and labels are available for finetuning. Based on this, the task of interest is more like an inductive transfer learning task rather than domain generalization.\n- If transfer learning under federation is the main focus of this work, then the datasets considered are too small for a comprehensive comparison of evaluated methods. For example, finetuning pre-trained models learned from ImageNet on downstream datasets like Stanford Cars, and Oxford Pets [1] are frequently used in the existing literature. Otherwise, it makes no sense to transfer learn features from a small dataset to the other small dataset. \n- In terms of the compared baselines, it could be better to consider existing transfer learning approaches, since it is a direct adaptation to federated learning.\n- There remains a gap between the generalization bound and transferability measurement. In fact, it remains not clear why cross-client statistics can be introduced as a transferability indicator.\n- From Algorithm 1, I can only guess the proposed method prevents local overfitting by preventing a very small Jacobian norm. If the proposed method works in such a way, then existing works use L2 norm [1] of feature map for regularization should also be considered as baselines.\n\nQUESTIONS:\nIn addition to the above weakness points, here are the additional minor issues:\n- In the introduction, Line 46 to Line 74 seem too verbose when presenting the limitations of the existing methods. It is suggested to give a distilled summarization of these methods and defer the detailed description to the appendix or related work, and highlight the advantages of the proposed method. Besides, if these methods are not compared or simply focus on different tasks with this work, then it is not necessary to discuss them here.\n- Line 299 “We use two sets of source-domain dataset pairs for FTL” Do you mean “source-target” dataset?\n- Line 307 “we let each client have access to only part of the labels.” This indeed to some extent confusing. I suppose this intends to refer to a semi-supervised cross-domain generalization task with part of the “labels”. After checking Appendix G, I know each client holds a fractional of “classes” with full annotations. \n- Line 341, “Besides the significant performance game of FedGTST over baseline methods”. Do you mean performance “gain”?\n- Line 523, Figure C, this link is broken. \n- Line 524, “FedGTST does not only offer better transferability”. -> Remove “does”.\n- As mentioned in the weakness, to figure out the task and evaluation metric, I checked the code provided. Frustratingly, I did not find much useful instruction, and the title of the README file remains the title of another paper “On the Convergence of FedAvg on Non-IID Data”. For a submission targeting such a venue, the authors should at least provide a decent description of their code implementation and evaluation criteria for reproduction.\n- It is recommended to discuss existing works leveraging Jacobian matching in transfer learning. \n- Theoretical insights (Line 266) could be moved to Section 4.\n\nLIMITATIONS:\nLimitations are discussed sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}