FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning
Authors: Evelyn Ma
Chao Pan
S. Rasoul Etesami
Han Zhao
Olgica Milenkovic
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Transfer Learning
Primary area: learning_theory
TL;DR: This paper presents FedGTST, a federated learning algorithm that improves global transferability using cross-client statistics, significantly outperforming existing methods.

Abstract: The performance of Transfer Learning (TL) significantly depends on effective pretraining, which not only requires extensive amounts of data but also substantial computational resources. As a result, in practice, it is challenging to successfully perform TL at the level of individual model developers. Federated Learning (FL) addresses these challenges by enabling collaboration among individual clients through an indirect expansion of the available dataset, distribution of the computation burden across different entities, and privacy-preserving communication mechanisms. Despite several attempts to devise effective transferable FL approaches, several important issues remain unsolved. First, existing methods in this setting primarily focus on optimizing transferability within their local client domains, thereby ignoring transferability over the global learning domain. Second, most approaches focus on analyzing indirect transferability metrics, which does not allow for accurate assessment of the final target loss and extent of transferability. To address these issues, we introduce two important FL features into the model. The first boosts transferability via an exchange protocol between the clients and the server that includes information about cross-client Jacobian (gradient) norms. The second feature promotes an increase of the average of the Jacobians of the clients at the server side, which is subsequently used as a local regularizer that reduces the cross-client Jacobian variance. A rigorous analysis of our transferable federated algorithm, termed FedGTST (Federated Global Transferability via Statistics Tuning), reveals that increasing the averaged Jacobian norm across clients and reducing its variance ensures tight control of the target loss. This insight leads to the first known upper bound on the target loss of transferable federated learning in terms of the source loss and source-target domain discrepancy. Extensive experimental results on datasets including MNIST → MNIST-M and CIFAR10 → SVHN suggest that FedGTST significantly outperforms other relevant baselines, such as FedSR. For example, on the second source-target dataset pair, we improve the accuracy of FedSR by 9.8% and that of FedIIR by 7.6% when the backbone used is LeNet.

FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning
Evelyn Ma, Chao Pan, Rasoul Etesami, Han Zhao, Olgica Milenkovic University of Illinois rrtana-Champaige (pingm, chaopan2, etesami1, hanzhao, milenkov)@ill pealllinoi.... edu
Abstract
The performance of Transfer Learning (TL. significantly depends effective pretraining, which not only requires extensive amounts of data but also substantial computational resources, As result, in practice, it is challenging to successfully perform TL at the level of individual model developers. Federated Learning (FL) addresses these challenges by enabling collaborations among individual clients through an indirect expansion of the available dataset, distribution of the computa- tional across different clients, and Despite several attempts to design effective transferable FL approaches, several important issues remain unsolved. First, existing methods primarily focus on opti- mizing transferability within local client domains, thereby ignoring transferability across clients. Second, most approaches focus on analyzing indirect transferability metrics, which does not allow for accurate assessment of the final target loss and the degree transferability. To address these issues, we introduce two important FL features into the model. The first boosts transferability via exchange protocol between the clients and the server that includes information about cross- client Jacobian (gradient) norms. The second feature promotes increase of the average the Jacobians of the clients at the server side, which is subsequently used as a local regularizer that reduces the cross-client Jacobian variance. A rigorous analy- sis of our transferable federated algorithm, termed FedGTST (Federated Global Transferability via Statistics Tuning), reveals that increasing the averaged Jacobian norm across clients and reducing the Jacobian variance ensures tight control of the target loss. This insight leads to an upper bound on the target loss of transferable FL regarding the source loss and source-target domain discrepancy. Empirically, experiments on public benchmarks show that FedGTST significantly outperforms other baselines, such as FedSR.
communication mechanisms.
Introduction
Transfer Learning (TL) has received significant interest in the machine learning community due to its ability extract representative features from source tasks and use them improve the generalization capability on related target domain problems [48 52]. In addition to boosting the performance of a target domain model, TL. also reduces the computational cost of fine-tuning the target domain model. Nevertheless, effective source pretraining in TI. is practically challenging for individual model developers because requires access large datasets as well as significant computational resources [35). To resolve this problem, one can leverage Federated Learning (FL), which refers to decentralized learning protocols used in mobile and loT devices [18]. FL not only increases access to multiple datasets in decentralized manner and alleviates the computational burden of individual clients, but it also protects the privacy of local data [32]. As a result, a number of recent works have outlined methods for transferable FL, including FedADG (Federated Adversarial Domain Generalization) [55]. FedCDG (Federated Contrastive Domain Generalization) [53), FedSR
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(Federated Simple Representations) [34], FedlIR (Federated Implicit Invariant Relationships) [12] FedCCST (Federated Cross-Clies Style Transfer)
4. FedMM (Federated Adversarial Domain
Adaptation) 42 and StableFDG (Stable Federated Domain Generalization) [36] Despite the
promising preliminary findings provided by the techniques above, several combinations of important issues remain unsolved across the spectrum of methods.
For privacy leakage, the limitations of existing methods include: 1. FedADG forces each client source domain to align its representation distribution with that of the target domain, and therefore violates data privacy because souree domains are given access to the target domain in order to perform the alignment; 2. FedCCST boosts global transferability by increasing local diversity to avoid local overfitting It therefore requires clients to share their local representations with each other and this information is subsequently used for local data augmentation. This is a direct violation of FL. privacy constraints; StableFDG expands local data diversity by sharing style statistics (i.e., representations, including means and variances). This clearly leads to the leakage of local privacy-sensitive information.
For local overfitting, the shortcomings of a group of the methods above are as follows: 1. FedSR learns simple representation through regularization during local training, by exploiting the similarity between the representation and the data, given the labels. However, since the regularized local training relies completely on local structures (i.e., local models, representations, labels, data), it leads to overfitting of local distributions, and thus has limited capability to learn cross-client invariant features, which is key for global transferability; FedCDG uses contrastive local regularizer on representations generated by various samples within the same class. This leads to overfitting in the local domain since no cross-client information is exploited.
For communication complexity, we observe that: 1. FedlIR is suboptimal. Although it mitigates the problem of privacy violation and avoids local overfitting by adding a local regularizer capturing the distance between the local gradient and the global gradient, it requires communicating gradients be- tween the clients and the server and therefore doubles the communication cost compared to baselines (additionally, FedIIR performs well for large number of clients, but offers average performance when this number is small); 2. Similar communication complexity problems are faced by FedCCST and StableFDG, which rely on communicating styles (i.e., representations); 3. FedMM requires significant communication overhead for adaptation using distributioon-matehee optimization problem [56]
techniques (58)
which involves solving an intractable nit-convex-non-conee
Finally, prior works mostly lack explicit theoretical analyses of global transferability: they do not tend to quantify the performance/loss of the pretrained model fine-tuned on the target domain.
In summary, perhaps the most important ved problem with known transferable FL models (with the exception of FedIIR) is that they use centralized TL approaches during local training, and do not fully exploit features specific to federated learning (for details, see also the discussion in Section 2)
Our contributions. We describe what is, to the best of our knowledge, the first approach to federated transfer learning termed Federated Global Transferability via Statistics Tuning (FedGTST) that simultaneously alleviates the above issues faced by existing methods. Our main contributions can be summarized as follows.
1. We use a new regularizer that encodes cross-client statistics and forces the local training process tune the global statistics in "direction" that improves global transferability rather than just local transferability. This is achieved through subtractions of global norms of Jacobians (gradients) communicated by the server.
2. We suggest to only communicate scalars, more precisely, Jacobian norms, which introduces a negligible communication overhead in the overall model exchange protocol.
3. We ensure that communication schemes do not allow uncontrolled access data and thereby ensure data privacy.
4. We rigorously prove that even though only small discrepancies among local gradients may exist upon regularization, transferability can be low as regularization can impede the growth of the gradient norm. To boost the Jacobian norm, we implement specialized protocols at both the client and server levels. Finally, we establish relevant bounds on the transferability loss for this setting.

---

The main technical insights provided by our analysis are as follows. Two FL-specific factors, small cross-client Jacobian variance and larger cross-client Jacobian norm are indicative of good transferability. These factors are direct performance indicators, unlike indirect factors (e.g., feature invariance) which only suggest improved transferability.. Our findings are based on the first direct measure of transferability, which equals the loss on the target domain incurred by the pretrained federated model. The FL-specific factors govern the bounds on the loss and therefore allow one to control them for better transferability. We validate these findings through extensive experiments which show that FedGTST outperforms methods such as FedSR and FedlIR by as much as 10%.
2 Related Work
FL is a machine learning paradigm in which multiple entities collaborate to train global model without sharing their local data (see [15] for a comprehensive review). FL has gained significant attention due to its potential to address privacy concerns while enabling large-scale collaborative learning [44]. Relevant to this work, [2] proposed the Federated Averaging (FedAvg) algorithm, which aggregates model updates from multiple client devices to train a global model. Another relevant line of work [51] introduced FedProx, federated optimization algorithm that incorporates proximal terms to handle non-iid data distributions. FL methods nevertheless still face several challenges. One challenge is dealing with highly heterogeneous local datasets (26) for which the recent work FedImpro([46) proposed leveraging aggregated feature distributions to address client drift. Another challenge is the communication overhead incurred during the aggregation of model updates (2). Minimizing communication costs while ensuring convergence and data privacy remain active topics of research in FL. Also, many FL solutions primarily emphasize performance in the client domain without considering the performance of the model on unseen domains.
TL is a powerful machine learning technique that allows models to leverage knowledge gained from one task to improve performance on another related task 35 TL has been widely adopted in various domains such as computer vision, natural language processing, and speech recognition, where labeled data may be scarce or expensive to acquire [47] 48). A common approach in TL involves fine-tuning pre-trained model on a target task using small amount of labeled data, which often leads to improved generalization and faster convergence compared to training from scratch [52]. Recent works in TL have focused developing more effective algorithms, such as domain adaptation methods that address the discrepancy between the source and target domains [1. Additionally, TL techniques have been used to handle tasks with limited amounts of labeled data through techniques like semi-supervised and self-supervised learning [4). TL still faces challenges such as negative transfer, where information from the source task actually degrades performance on the target task; and, it requires careful selecting of appropriate pretrained models and transfer strategies for specific tasks and domains (35). Current TL methods often require that one entity possesses knowledge of all data, violating the privacy requirements of FL. Moreover, we comment on Gradient Matching in TL in Appendix [.]
Transferable Federated Learning (TFL) is an emerging research area at the intersection of FL and TL. One of earliest contributions to the field, FedADG (53), encourages the transferablity of FL through adversarial local training. However, the work does not provide theoretical guarantees, and existing studies [50] indicate that adversarial robustness does not necessarily lead better transferability. Other methods, such as FedSR (34 and FedCDG [53] enhance transferability by adapting standard representation learning from single-agent
federated setting; they do not incorporate Fl.-specific features (i.e., instructions provided by the server, cross-client model properties etc). Note that although FedSR has successfully included centralized invariant feature learning into FL, it uses centralized methods locally and then shares information with the global model, and thereby does not fully exploiting FL capabilities. Thus, using FedSR, each client can learn very different representations that are hard to aggregate at the central server. More precisely, FedSR does not communicating information that can help improve the transferability of the global model. Among all the previously discussed methods (FedADG, FedCDG, FedSR, FedlIR, FedCCST, and StableFDG), FedIRR is the closest to our approach and may be seen as special case of our method which has better performance, smaller communication complexity and comes with provable global transferability guarantees. Furthermore, we discuss the distinctions and connections between TFL. and Generalization of FL, topic potentially relevant to TFL, in Appendix 1.2
3

---

3 Preliminaries
General Supervised Learning Settings. We denote the data space by X, the feature space by 2, and the label space by y. A model h : X' + ) typically takes the form h = g f. where f : - 2 is a feature extractor and g : 2 - 3/ is a classifier. Denote the function class for the entire model, the feature extractor and the classifier by H, F, G. respectively, so that h E H, f E F,g g E G. Denote the weights of model v E {S.9, ht as wy. Given loss function / 3 x 3 + IR and domain distribution D over X' x V the population loss (h) is defined as
Lp(h) == ((,,))> (h(x).
General Framework of TFL. In TL. the two typical learning phases are: a) pretraining on the source domain; and, b) finetuning on the target domain. In the context of TFL, pretraining conducted via FL over source (local) domains, while the global model is trained and then finetuned on the target domain during the second phase. both phases, supervised learning performed with full access to the labels. More details are provided next.
Pretraining Phase in TFL: FL on Source (Local) Domains. The source domain is a composition of the agents' local domains, {D(()), with l [K] denoting the client index and K representing the total number of clients. The source loss is defined as the standard federated loss on the source domains.
K
1
Lore o
Σ K
Diw
(g- f).
(2)
Let h² - "0 o f be an optimal global solution for the objective (2). In FL approaches, the problem solution the result of the central server's aggregation of local models into a global one. We denote the local solutions involved in creating the optimal global solution v (1b E {f by (k));)); };
through averaging aggregation, we obtain the optimal global weights w = K
1 Σ. w
Finetuning Phase in TFL: Supervised Finetuning on the Target Domain. Upon obtaining the optimal pretrained global solution h" g" f", the pretrained feature extractor f* is fixed and applied to the target domain Dr. The target loss is defined as the loss on the target domain Dr i...
Lnge f*) LoD, (g f*)-
(3)
Through finetuning, new classifier 9 == arg inf Lige (g o f*) is determined by minimizing the target objective 3
Transferability Assessment. With a slight abuse of notation, we define the optimal target loss as
Ligt == ( f*).
We formally define the measure of transferability of TFL the optimal target loss Ligt. as it directly reflects the performance of a transferable model on the target domain. A smaller Ligt, or a tighter bound on it, indicates better transferability.
4 Theoretical Bounds on the Target Loss
4.1 A General Bound Based on Discrepancy/Divergence
We start with Definitions T and [borrowed from existing TL studies that characterize the domain discrepancy, and then propose a new domain divergence tailored to Transferable FL (TFL), including the cross-client discrepancy in Definition and the source-targe discrepancy in Definition 3
Definition 1 (G, Fdiiccerpyyyyyy (50)). Given classifier class G, a feature extractor class F, the source domain Ds and the target domain Dr, with slight abuse of notation, the (G, -ddiscrrppyyy
defined as
dg.=(Ds, Dr) = sup inf Lpo JeF gEg
inf LD (g gEg
Remark. The G, F-dieerrpyyyyy has also been used in the analysis of domain adaptation [50].
We next adapt the H-discrepancy to measure the cross-client domain discrepancy in transferable
FL. (Definition 2p. and tailor the 9, F-discreppaccy (Definition T to measure the source-target discrepancy in TFL (Definition 3)

---

Definition 2 (Cross-Cliem, Divengence for TFL). Given model class H. and federated local domains D hEEK [K(K) with d><(:;)) defined as Definition the intra-client discrepancy is defined as
dx(D fed
Σ du (DS lo #22
K(K II)
Note that dx (Ds) equals the average of H-diicrepanncies over all local domain pairs and therefore measures the intra- ciira-diccrepaccy on the non-iid distributed source domains. When souree domains are iid across clients, we have dx (Ds) = O.
Definition 3 (Source-Targe) Discrepancy for TFL). Given classifier class G, feature extractor class F, federated local domains D (D(*) ke[x], and the target domain Dr, with slight abuse of notation, the federated F) -discrepancy defined as
dD fed (r))
1 Σ (D() Dr). kEIK]
(6)
K
Based on the two TFL-specific domain discrepancy definitions (i.e., Definition 2 and 3), we derive a
general bound on the TFL loss in Theorem [] The TFL-specific source-target discrepancy (Definition 3) is further used in Theorem 2 which presents bound on the target loss using cross-client statistics. For our theoretical analyses, we need the following common assumptions.
Assumption 4.1 (Convexity and Smoothness). We assume that the loss function satisfies two conditions: (I) l convex w.r.t wh: (2) I is Lipschitz smooth for wi with a constant O.
is easy to meet in practice, and it arises in many linear models (linear regression, SVM etc).
Remark. Assumption
Theorem 1 (Bound Based on TFL-specific Domain Discrepancy). Under Assumptions 4.1 and Smoothness), the optimal target loss is bounded as
Convexity
LDct (n^(t)] + (x(ed) + dg. Dr),
Li I
K
where (1) denotes the optimal local model of client (see Section
Semantic Interpretation. In Theorem the optimal target loss Ligt is bounded by the sum of three
terms on the RHS of Equation 7 (1) the averaged optimal local loss * ΣK [InDue) (h~(+))]: (2)
the intra-discrepancy of the source domain dx (1D) (3) the discrepancy between the source and
target domains, dg.r(Dfedd Dr)) Therefore, the bound on the optimal target loss can be tightened by making the optimal source loss smaller, and by lowering the intra-source and source-target discrepancy. The later two terms can be controlled through regularization as detailed below.
Tightening the Bound via Regularization over F. Given two feature extractor function classes Fi and Fo. if F F2, we have Hi H2 where Hi G Fi and Hz G F2. From Definition
2 and it straightforward to see that dx, (Df < du, (D fedy and dg.r, (D) Dr) s
dg.F, (D Dr). This indicates that any general regularizer on F can lead to a decrease in the latter two terms of the RHS of Theorem However, shrinking the expressive power of F will inevitably increase the optimal source loss, which is the first term on the RHS of the expression in Theorem [ Therefore, using general regularization, one has to trade-off the optimal source loss, TFL-specific cross-client discrepancy, and TFL-specific source-targe target discrepancy.
Based on Theorem and the follow-up up discussion, we aim to answer the question: how should one design a practical regularizer that can inherently tighten our bound on TFL. We give an answer to this question in Section 4.2 There, Theorem Shooss shows that a transferability regularization should decrease the cross-client Jucobian variance while at the same time increase the cross-client averaged Jacobian norm.
pretraining
5

---

4.2 Practical Bounds Based on Cross-Client Statisties
The goal of FL to estimate the optimal solution f by updating the global model through multiple federated rounds |
Denote the total number of rounds by P and the output global model after round P by fr. Denote the target loss under fr instead of f Lige (which is an estimate of Ligt)- At the end of local training during any round p S P. let hig with weight wp represent the model of client k, and let hp with weight wp represent the global model. Denote the Jacobian (gradient) of the loss ( w.r.t the model weights at domain as ( (())() (w) = E D V., [(,(()))) ())
Throughout the remainder of the manuscript,
J.hh) as shorthand for J(H) (wp)) Furthermore,
we denote the learning rate of agent k at round p by X We make a single-step local update assumption (Assumption 4.2) and use the definitions for cross-client statistics (Definition 4 derive bounds exploiting the cross-client statistics from Lemma 4. and Theorem 2))
Assumption 4.2 (Single-Step Local Update). During local training, all clients perform one step of gradient descent (GD) update their model for transmission, (k)
up 2 ((0)(w)) This
a common assumption in the FL literature (30 32
Definition 4 (Cross- Client Statisties). At federated round p. given K clients with local Jacobians (J/h) )е[K]] we define the cross-client averaged Jucobian norm || ll and the cross-client Jacobian variance, respectively,
K Σ (k)
and of
(k)
1 K
J (
(8)
K
K
Note that assumed the loss function to have a-Lipschitz continuous gradient. When the gradient is large, a is also large, and to make B1 (A) 2 0, À has to be close to O. In this case, the absolute value of the second term can be small and that of the third term can be large.
Lemma 4.1 (Loss Bound Using Cross-Client Statistics). Under Assumptions 4.2 and 4.1 and the cross-client statistics defined in Definition 4 after P rounds of federated pretraining one has
P-1
P-1
Ligt S Loro (ho)
Σ (1
Σ B2(Ap+1)0z + dg DT):
(9)
p=0
p=0
where ho is the initial global model and B2(A) = ax² 2 31(A) A
Note that during the training process, a large Jacobian norm can promote transferability (the Jacobian is expected to be small at the end of training).
Interpretation of Key Terms. Lemma 4.1 shows that the target loss of the finetuned pretrained model (LHS) is bounded by sum (RHS) involving four key terms: 1) Lsre(ho), the initial source loss; 2) || the cross-client averaged Jacobian norm; 3) of the cross-client Jacobian variance;
4) (D fed Dr), the TFL-specific source-target domain divergence. Since (hre is fixed throughout the pretraining process, and dG.=(D) DT) can be reduced using general regularization, focus analyzing the two tunable cross-client statistics, || and oz.
Influence of the Cross-Client Statistics on the Bound. We note that during pretraining, both the coefficients 31 (Ap+1) and (2(Ap+1) have to be positive (see how to ensure these constraints in Appendix D Therefore, a larger || and smaller o² tighten the upper bound, indicate lower target loss and thus suggest better model transferability.
Coefficients Quadratic w.r.t. the Learning Rates. Lemma 4.1 iivolles coefficients 31 (A) and B2 (A) that are quadratic in the learning rate X; thus, we can further tighten the bound by optimizing the learning rates across different rounds (Assumption 4.3).
The tightened bound is given in Theorem 2
'One federated round comprises local model initialization. local training, local model transmission, global model aggregation, and global model broadcasting
We briefly discuss the Stochastic Gradient Descent approach in Appendix F

---

Assumption 4.3 (Optimal Learning Rates Across Rounds), In each round 12 P), we use an similar analysis and assumptions on
optimal learning rate for local training X a a. 1
K-
11
optimal learning rate has also been used in 1Z 10 13
49%.
Theorem 2 (Tightened Bound Based on Cooss-Client Statistics). By optimizing the bound in Lemma 4.1 with respect to the learning rates X as governed by Assumption 4.3 and under Assumption 4.1
(Convexity and Smoothness), the estimated optimal target loss is bounded as follows:
2 211 a11 2211-111))
Li I Lore (ho)
dg. Dr))
Semantic Interpretation. Theorem inniicttes that by optimizing the learning rates at each round,
the bound for Ligt can be made smaller through 1) smaller souree loss (hoe(h)); 2) a larger cross-client average Jacobian norm || 3) a smaller cross-client Jacobian variance oz. and 4)
smaller source-target domain divergence dg. fed Dr). These are consistent with Lemma 4.1
Intuitive Explanation. (a) Increasing the Cross- Client Averaged Jacobian Norm || In early training. it is crucial to avoid a small Jp. as it can "trap" clients in local minima or cause overfitting of local distributions. Note that increasing Jp will not induce an excessively large second term (i.e., a term that + 20) in Theorem 2 as increments are constrained through the Lipschitz condition x2ll-
[|J J2ll S a ||| - In the final stages of training, [| naturally decreases due convergence. (b) Decreasing the Cross- Coiss-Clent Jacobian Variance dp. A small dp promotes domain similarity. preventing local overfitting and enhancing global model transferability. (c) Trude-Off Between Increasing |plll and Decreasing pp. A larger || induces more substantial local updates but can increase variance pp+1- Thus, Theorem underscores balancing this trade-off: enlarging || ||ll while maintaining small Op.
Challenges in Regularization of the Jacobians. Certain regularization of the Jacobians can induce local Jacobian alignment and reduce of (e.g., the regularization of the alignment between local gradients and the global gradient proposed in FedlIR [22. However, local regularization can naturally impede the growth of local Jacobian norms {Jp [R((K] and subsequently prevent |Jplla from increasing. Thus, such prior regularization may not necessarily improve transferability. As an example, consider the extreme case where of = 0. Then, the bound in Theorem 2
takes the form
Trigt S. T.are (ho) - Ep- 2 2115/112 + (D Dr), where a small ||Jpll2 can severely degrade the bound. This indicates that boosting transferability by prior regularization, such as controlling of itself, has limitations, and tuning | during the pretraining stage is of crucial importance. We provide a solution to this problem in Section
a
5 Our Algorithm
Algorithmic Solution for the Theoretical Challenges. From Lemma 4.1 and Theorem we can see that certain round-wise FL-specific statistics, the cross-client averaged Jacobian norm |Jall and cross-client Jacobian variance of control the bound on the target loss. The challenge is that, while o3 can be reduced using straightforward techniques (i.e., such as gradient alignment from FeduR [12,, such techniques unavoidably prevent Jp from increasing properly. We therefore propose our FedGTST approach which reduces of. while enlarging || A round-wise description of FedGTST is given in Algorithm
Tuning the Cross-Client Jacobian Variance o². The cross-client Jacobian variance is controlled via regularization at the local client level (Linee 5 Algorithm 5). The local clients, upon receiving a guide norm y from the server (explanation deferred to subsequent sections), implement a regularized local training protocol. While client & during standard local training uses the objective LDon) (h), during regularized local training he/she/they use
Ingg (h) E - (1)12))))))))))) -))
(10)
instead, where& & is the penalty coefficient for the regularization term. This type of regularization intuitively aligns each local Jacobian norm [}](k] || with the guide norm 7, which results in o that is smaller than that of standard FL (i.c., FedAVG).
7

---

Algorithm FedGTST (Roundy) p)
Randomly select a & C [K] for regular training.
2: while k S K, Client k should do
3: Receive the guide norm 7p-1 and global model hp-1 from the server.
4: Initialize the local model to hp-1-
5: Update the local model to (6) by training with D (h) + (1) -2)
6: Calculate the surrogate Jacobian norm 7/p = 7/p if' & E @ then
(k)
(k)
|| 1l2.
7:
Train with LDw (h) to obtain Jacobian norm std,(k)
8:
Yp
(k)
reg.(k) std.(k)
9: Update the surrogate Jacobian norm "ip max
7p
7p
10: end
11: Transmit the model hg and norm %/ the server.
12:
end while
13: The server aggregates the client models into global model hp and sets the guide norm to maxx (7p (k))
The server broadcasts hp and 7p all clients.
Tuning the Cross-Client Averaged Jacobian Norm |-ppl22- The cross-client averaged Jacobian norm can be increased via an exchange protocol that includes: 1) clients caleulating surrogate norms for transmission; 2) the server computing and broadcasting a guide norm; 3) clients performing local alignment using the guide norm.
Surrogate Norms. A decrease in the local Jacobian norm [| prevents the cross-client averaged Jacobian norm || from growing. We mitigate such a decrease by forcing a small portion of the clients to implement both regularized training and standard training (Line D to generate pair of Jacobian norms (kg)(k) and sttd,(k), The client than chooses the larger norm, 7(()) mar( (k). (td)(k))
as surrogate norm for transmission (Lines to
Server Guide Norm. The clients send their local Jacobian norms -/( := (k) ||2 to the server, and the server broadcasts the largest norm received as its guide norm y max (8) (Line 13
Local alignment. The local regularizer from Equation (10) reduces the variance, but can also force an increase of the averaged Jacobian norm. It forces all local Jacobian norms to align with the guide norm y (Line 5) Since y has been boosted both the clients' and server levels, the alignment leads to larger local Jacobian norms.
Communication Cost. Most TFL methods, already described, require large communication overheads between the clients and the server. For example, FedlIR requires exchanging Jacobians, which have the same dimension the model weights; FedGTST only requires exchanging norms.
Experiments
6.1 Experimental Setting
Transfer tasks. We investigate three transfer tasks utilizing fully-annotated data: a) MNIST 9 to
MNIST-M [TI. b) CIFAR-10 [19 to SVHN 333 and c) cross-domain transfer in DomainNet [37
All transfer tasks have been used as benchmarks in existing TL research [3] I 22 28 50 (also,
Appendix G). Moreover, DomainNet is standard large-scale dataset for TFL studies, also used by FedSR.
Non-iid Distributed Source (Local) Domains. The pretraining phase is conducted via FL on source (local) domains. Marginal distribution shift is an important phenomenon in FL (2 since the access to classes (or categories) is not the same for all participating entities. However, some federated datasets tested by existing transferable FL methods do not reflect marginal distribution shifts when constructing source local domains (i.e., in the -NISTT benchmark in FedSR [34], all clients have access all classes). To address this issue, we employ the following methods for constructing non-iid local domains:

---

For MNIST or CIFAR-10, unless otherwise specified, we follow the approach in [27] and let each client have access to only a subset of the categories. The category selection rule and data sampling method is described in Appendix H Besides, we run additional experiments
for Dirichlet sampling, a method commonly used in FI. 20 23
29
46
For DomainNet, which compromises six distinct domains, we follow the leave-one-out strategy used in FedSR: one domain is designated as the target domain, while the remaining five domains serve as source domains, with each assigned an individual client.
Federated System Size. Large system sizes are inherent to FL. systems, where the number of clients can be as large as 100 27 In such a case, is more challenging for the global model to achieve good performance. However, existing TFL methods typically use very small number of domains (S 5 for FedSR and StableFDG). In contrast, we allow the system size to cover a broad range of values, including 10 (small), 50 (medium), and 100 (large) clients.
Evaluation matrix and backbones. We measure domain transferability via acCagt, the accuracy of a pretrained model finetuned on the target domain. For backbone selection, we use both LeNet [2]] and ResNet18 to represent different levels of backbone complexity.
Baselines. We consider the following TFL. algorithms as our main baselines: FedAVG, FedSR, and FedIIR. Furthermore, also compare our findings Scaffold (16), advanced FL approach used to address data heterogencity. We do not report results for FedADG, FedCDG, FedGTST, and StableFDG since they do not ensure privacy and/or underperform compared to the main baseline.
Settings for Pretraining (FL on source local domains). 1) Local epochs. To conform with our theoretical assumptions, unless specified otherwise, we set the local epoch of each client to 1. We also investigate the system performance with 10 local epochs in Appendix([] [C] 2) Number of participants per round. We allow 50% of the clients to participate in each round (e.g., if K = 50, there 25 participants per round). 3) Number of clients performing standard local training. To boost the Jacobian norm, subset of clients conducts standard local training (we set the subset size to 10% of the total number of clients).
Additional Settings. Unless specified otherwise, for local training on the source datasets we use the Adam [17 optimizer with coefficients ($1,3g) = (0.9, 0.999) (note that these 3 values are not to be confused with the coefficients from our theoretical analysis). Our initial learning rate equals 0.01 and then decays by a factor of 10 per 50 rounds, with an early stop trigger of 10 rounds. We apply the standard cross-entropy loss. We also set the pretraining batch size to 256 for MNIST - MNIST-M and to 128 for CIFAR10 - SVHN. For both tasks, we use a finetuned learning rate 0.005, with weight decay 0.0001. All results reported in Section 6.2 are averaged over three runs.
6.2 Results
Table 1: Target accuracy (%) of the finetuned model pretrained on a small number of clients (K 10). FedGTST outperforms other methods across both tasks and both backbones; for the example MNIST to MNIST-M with LeNet backbone, FedGTST outperforms FedlIR and FedSR by around 4%.
MNIST- MNIST-M
CIFARIO - SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
FedAVG
73.8
81.610.2
64.410.5
72.011.0
73.0
FedSR
75.0+10.9
80.611..1
65.9+10.6
71.3+10.4
73.2
FedIIR
74.5+10.3
82.7+0.7
66.2+1.0
73.8+1..2
74.3
Fed-GTST
76.2.10.9
82.3+0.55
70.1
74.5:L0.3
75.8
Transferability results. FedGTST exhibits significantly improved transfer performance when compared to basclines across range of tasks, system sizes, and backbone architectures. For the example of CIFARIO-
with 100 clients and a LeNet backbone, FedGTST outperforms FedIIR by 7.6% and FedSR by 9.8%. The results for a small, medium and large federated system (K 10. K 50 and K = 100) are reported in Tables [ and 3 respectively.
Discussion. Besides the significant performance gain of FedGTST over baseline methods, we also observe that 1) for transfer tasks in which the backbone and method are fixed, transferability generally decreases with the system size. Importantly, FedGTST improves the baselines more significantly

---

Table 2: Target accuracy (%) of the finetuned model pretrained on medium number of clients (K' 50). FedGSTS outperforms other methods across both tasks and both backbones; for both the examples of MNIST-MNISTAN and CIFARIO-SSV with a ResNet18 backbone, FedGTST outperforms FedIIR and FedSR by around 5%.
MNIST MNIST-M
CIFARIO
SVHN ResNet
Method
Average
LeNet
ResNet
LeNet
FedAVG
=5.5
72.4+0.2
59.1
65.5+t.1
65.1
FedSR
64.0=10.6
73.1
58.9=0.1
66.8
65.7
FedlIR
64.7:10.2
73.911..
59.720.3
66.2:t0.1
66.1
Fed-GTST
6..0
79.21004
63.5110.1
71.1
70.7
Table 3: Target accuracy (%) of the finetuned model pretrained large number of clients (K = 100). FedGTST outperforms other methods across both tasks and for both backbones; for CIFARIO-SVHHN with ResNet18 and LeNet backbone, FedGTST outperforms FedlIR and FedSR by 7%.
MNIST
MNIST-M
CIFARTO
- SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
FedAVG
48.710.1
61.1
41,241.22
51.7.00.5
50.7
FedSR
49.2=00.2
59.8
42.6+0.1
52.0=0.3
50.9
FedIIR
51.9=0.4
61.3=1.11
44.8+0.4
55.8=0.2
53.4
Fed-GTST
57.5+1.3
67.621.2
52.410.1
63.1-0.2
60.2
for large system sizes; 2) when the system size, transfer task and the method fixed, the more "complex" the backbone (ResNet18 vs LeNet), the better the transferability.
Additional results. We defer reporting and discussing additional results in Appendix C which include (a) an investigation on hyper-parameter sensitivity the convergence speed, and cross-client statistics, (b) results regarding DomainNet, Dirichlet Sampling, and Scaffold.
7 Limitations
Increased Local Computational Cost. While FedGTST only induces negligible communication overhead among clients and the server, we note that at the level of a small subset of clients we need to conduct both regularized training and standard training to boost [2ppl22- This increases the local computational burden but to a very small extent. Nevertheless, in future works, we will explore algorithms with lower local computational costs.
Potentially Loose Bounds. Although existing studies have added to our understanding of transferable federated learning, our work is the first to derive bound direct measure of transferability (the target loss). We believe that the bound can be tightened.
Conclusion
We introduced FedGTST, federated learning algorithm aimed at enhancing global transferability. Inspired by theoretical insights, FedGTST integrates cross-client information on averaged Jacobian norms and Jacobian variance. Our work addresses key challenges in existing methods, such privacy violations and an overemphasis on local transferability. Experimental results demonstrate significant performance improvements over baseline models.
Acknowledgement
This work was supported by the Air Force Office of Scientific Research under award number FA9550- 13-1-0 the NSF CARÉER Award under Grant No. EPCN-1944403 as well as the NSF Awards CCF 1956384 and 2402815, and SVCF CZI 2022-249120.
10

---

References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regular- ization. In International Conference on Learning Representations (ICLR), 2021. URL
[2] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp. David Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv: 01046, 2019.
[3] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems, 29, 2016.
[4] Junming Chen, Meirui Jiang, Qi Dou, and Qifeng Chen. Federated domain generalization for image recognition via cross-client style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 361-370, 2023.
[5] Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103-126, 2014. ISSN 0304-3975. doi: https://bbiorrg//0000011...tooo0 wttp://///o..t.......... sci httlaace/rttial////////////73313
URL https: www.:ssceeneeirecteo com/ Algorithmic Learning Theory.
[6] Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. Journal of Machine Learning Research, 20(1):1-30, 2019.
[7] Laizhong Cui, Xiaoxin Su, Yipeng Zhou, and Jiangchuan Liu. Optimal rate adaption in federated learning with compressed communications. JEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 1459-1460 IEEE, 2022.
[8] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty, Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pages 447-463, 2018.
[9] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[10] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13400 2020.
[11] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Journal of machine learning research, 17(59):1-33 2016.
training of neural networks.
[12] Yaming Guo, Kai Guo, Xiaofeng Cao, Tieru Wu, and Chang. -ut-of-distri generaliza- tion of federated learning via implicit invariant relationships. In International Conference Machine Learning, pages 1 1905-11933. PMLR, 2023.
[13] Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv: 1910. 2019.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- juneta Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 2019.
[16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5132-5143. PMLR, 2020.

---

[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Jakub Konečný, H Brendan McMahan, Daniel Ramage, Peter Richtárik, Ananda Theertha Suresh, et al. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv: 1.03575, 2016.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning at scale. In International Conference on Machine Learning (ICML), 2022. URL
https::/proceeediin .mlr wwwps///ww///aa222.... html
[21] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-24) 1998. doi: 10.1 109/5.76679.
[22] Seunghun Lee, Sunghyun Cho, and Sunghoo Im. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15252-1526), 2021.
[23] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 10713-10722, 2021. doi: 10.1 10.1109/CVP446437,
2021.01057. URL / thecvf. com/ /otp////oooteee////////////////p Mbo.aLCoonnaattteoeeeaaacaaaaann
pdf
[24] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learn- ing. In CVPR, pages 10713-10722, 2021. 01057.
doi:
10.11090CVPR46437.2000
URL htttps://openaccess thecvf.... wtt/ httm://nntent///P/0////aarss///s
CVPR Lawpin__cVPP_221_aappeee 2021_paper_pppp
[25] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10713-10722, June 2021.
[26] Tianqing Li, Anup Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3))50-60, 2020.
[27] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02.130 2019.
[28] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 975105. PMLR, 2015.
[29] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. In Advances in Neural Information Processing Systems, volume 34, pages 6072-6084, 2021.
[30] Evelyn Ma. Praneet Rathi, and S Rasoul Etesami. Local environment poisoning attacks on federated reinforcement learning. arXiv preprint 2303::23003.0225,, 2023.
[31] Yishay Mansour, Mchryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0900..4453 2009.
[32] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiinne learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
Hampson,and
[33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
12

---

[34] A Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing Systems, 15:38831-388444 2022.
[35] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1355-135, 2010.
[36] Jungwuk Park, Dong-Jun Han, Jinho Kim, Shigiang Wang. Christopher Brinton, and Jackyun Moon. Stablefdg: Style and attention based learning for federated domain generalization. Advances in Neural Information Processing Systems, 36, 2024.
[37] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1406-1415, 2019.
[38] Mohammad Pezeshki, Chengxu Le Lan, Thibaut Durand, Guillaume Lajoie, Simon Lacoste- Julien, Aaron Courville, and David Lopez-Paz. Understanding hessian alignment for domain generalization. arXiv preprini arXiv:2308. 11778, 2023.
[39] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 18347-18377. PMLR, 17-23 Jul 2022. URL attpss:/proceeddingg mlr..mmrr... presp v162/rame22a. html
[40] Sebastian Ruder, Ivan Vulic, and Anders Søgaard. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569-631, 2019.
[41] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do ad- versarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33, 2020. URL htttt::://ppat nsf gow htor/biblio/002233333
[42] Yan Shen, Jian Du, Han Zhao, Zhanghexuan Ji, Chunwei Ma, and Mingehen Gao. Fedmm: A communication efficient solver for federated adversarial domain adaptation. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 1818-1816, 2023.
[43] Yuge Shi, Jeffrey Seely, Philip H. S. Torr, Siddharth Narayanaswamyy Awni Y. Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations (ICLR), 2022. URL. / net/
Dorum?id=__75JXK8VVJ
[44] Virginia Smith. Federated learning: strategies for improving communication efficiency, IEEE Signal Processing Magazine, 34(6):36 2017.
[45] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. arXiv preprint arXiv:2206.0244 02465, 2022.
[46] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, and Xiaowen Chu. Fedimpro: Measuring and improving client update in federated learning. In The Twelfth International Conference on Learning Representations (ICLR), Vienna, Austria, May 7-11 2024. OpenReview.net.
[47] Lisa Torrey and Jude Shavlik. Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pages 242-264. IGI global, 2010.
[48] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
13

---

[49] Xidong Wu, Feihu Huang. Zhengmian Hu, and Heng Huang. Faster adaptive federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10379-
10387, 2023.
[50] Xiaojun Xu, Jacky Y Zhang, Evelyn Ma, Hyun Ho Son, Sanmi Koyejo, and Bo Li. Adversarially robust models may not transfer better: Sufficient conditions for domain transferability from the view of regularization. In International Conference Machine Learning, pages 24800-24802. PMLR, 2022.
[51] Qiang Yang, Yang Liu, Tianjian Chen, Yu Tong, et al. Federated optimization: Distributed optimization beyond the datacenter. IEEE Signal Processing Magazine, 36(4):49-59, 2019.
[52] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems, 27, 2014.
[53] Xinhui Yu, Dan Wang, Martin McKeown, and Jane Wang. Contrastive-enhanced domain generalization with federated learning. IEEE Transactions on Artificial Intelligence, 2023.
[54] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chun- hua Shen, and Chao Wu. DENSE: Data-free one-shot federated learning. In Advances in Neural Information Processing Systems, volume 35. pages 19112- 19126, 2022. URL https: htttss:/ppapers.niipp www.. wwwp..ccp...cpppp.eeeecccppppppa falpeerrrfll/eeeeeeeeeeeeeeeeeee fiwes/ilpppppeeee ptille//paprr//22//aaaaase 8681 28812666660022220011000111144-23 htm
[55] Liling Zhang, Xinyu Lei, Yichun Shi, Hongyu Huang, and Chao Chen. Federated learning with domain generalization. arXiv preprint 1xXiv:2111.00887 2021.
[56] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In Advances in Neural Information Processing Systems, pages 8568-8579, 2018
[57] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference machine learning, pages 7523-7532. PMLR, 2019.
[58] Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoff's in invariant representation learning. The Journal 2022.
of Machine Learning Research,
14

---

A Additional preliminaries
Definition (H-discrepancy 6 57). Given model function class H. and two data distributions Ds. Dr, the H-discrepanny between Ds, Dr is defined
(Ds, Dr) == sup NEH
LDs (h)l
(11)
Remark. This type of discrepancy, as well as the related H--divergence, have been frequently used in analysis of domain adaptation
and we follow this trend.
B Proof of our bounds
Lemma B.1 (Theorem 2.5 in (50). Suppose we perform pretraining the source domain DDs to obtain f and at the server, and fine-tune the model on the target domain Dr to obtain new classifier gr. We then have
(9r f*) Lorelg" dg. Dr).
(12)
Equation 12 shows that the transfer loss can be upper bounded by the sum of the loss on the source domain and the divergence between two domains.
Lemma B.2. With Assumption 4.1 (Convexity and Smoothness) and Definition (Cross-Cliee Divergence), we have
Laoc (A*1ht)]- + dx(
Lore S
(13)
K
Proof. The loss function / is assumed to be convex w.r.t the parameters of the model h, where parameters are denoted by wh. Based on the aggregation rule Section 3 we have wp. = (k) stands for the weights for global model h and that for local model 1 Σx w/y where wh and wi, K
l(kk, respectively. We therefore have
Lere(h") K
LiDin (h*)
b1
Σ
EDiip Z (x, wf) v)
K
h1
Σ A1
E
h x, K
1
Σ wh
(A2z) ,
(a)
E E I (hr (Zz, (o( .1)
K2
Lock) (n=(k2))
R1.
(8)
TTpinn (nolket) drs (Ditt), Dibss)]
K2
kiike
Σ
Liite (Bothu)) + K²
Σ dre (D(t)), Diit)) ki.k2
LI(A) (nothi)) + dx (Ds):
bn
where 1 Ek, M2 du (D(1)), ((2))) S K(R-1) Enitke du (D(m)), D(ka)) = (Ds) as in Defi- K2
nition Here (a) follows from the convexity assumption for I w.r.t the parameters, while (b) follows
15

---

from Lemma B.1
based on the argument below:
LD()
(h*(k2)
LIDEE) (hrlees) D((e) (h^(t2)
+ D(t (nelte))
$ LDbnn (h~lkea) Den (n-(ts)
+ LD(n)) (n=(ba)
s dH (D(k)), D(ka)) + DDE (ha(ka)).
Theorem (Theorem) Dp. loss is bounded by
Under Assumptions 4.1
Convexity and Smoothness), the optimal target
K E [
(n*(t)] + (n( dg. Dr),
Lig
(14)
where I (k) denotes the optimal local model of client k (see Section
Proof. Since
Ligt LDD (97 fz) S Lic o + dg ,rr)
we have
Lgg S K Σ LIw of) dg. = Dr)
(isrc(h)) + (D Dr).
Using Lemma
B.2
have
Ligt S Lero(h") + (D fed Dr)
TuDeto (n>(tt)] + dz (D( fod) + do.r(D) Dr). (D))
K
Lemma B.3 (Bound on round-wise source loss), Suppose the learning rates of all clients round p are the same, ie., Ap Ap, Vk E [K].2 E [P]. Under Assumptions 4.2 and 4. we have that
Lare (Mp+1) I Torre (h₂)) (Ap Jy + 132(Apt)o
(15)
where Jp Ex (k) of Eか )
1
E
(k) J
and B1 (A) x B2(A)
R
R
ox²
2
Proof. Following the same proof idea as for Lemma B.2 we have
Ture (hp+1)
S
TeD(tt
(kzg)
(16)
K2
By definition, we also have
(1 (k2)
Execyi (n (z, w(tt)) v9)
L Lock)
(17)
From the update rule of GD, write (1 (x 10f .2) 1
h
Up Apt1
V wn Tupcta) (h)
1
h
wp Apt1
V Ee (h(xi, w).
(18)
w=
=1 1
(r, wp Apt1 Jphai)
(19)
16

---

In 18 (ot, V4) are sampled from D(aa), and not to be confused with (a, 3) sampled from ((k1),
Define A == -Apt1 _/ upper bound the loss in (19) by
I (h (2, .2) 1 (h (z, wp + +)),))
S 1 (h (z, wp) v) + V (h (2, w),,0) 10m
A
A.
(20)
The inequality holds since / is assumed to have a- a-pisshhit continuous gradient. Combining I7 and (20). we have
LDDA (Mott)
I Ep(() (hh (x, up). .3) - [Vrvl (/ (ar, (I, w) v) w=wp] 1
+ A
a 2
A
D(c (kp) + [s A A
(21)
Next, by 16 and 21 we have
Lirr (hp+1) 5
Liiike (1oti)
R1 A02
LDWA (hpp + [J(ki]]
A + A A
K2
k
TuDUNN (Mp) - Apt1 [sbnn]] T1
k1 Azz
( [||_||aa)
27
+ (Ap+1)2
Lave (hp) 11+11152112 + 2K
(Ap+1)² 2.
Σ
((kz)
(22)
222
The last equality follows from the definition of Jp.
Define of =
1 K Ex= 1(( || 11 Then, we have
Lave (hp+1)= S Lorc (hp) Apt1l1
a
(Ap+1) -07 + +111111))
2
(2+z2)2
|| (Ap+1)
Lore (hxp) Aptz
og
2
-Lore (htp B1 (Ap+t)) + (2((ttt)
(23)
which completes the proof.
Lemma B.4 (Lemma 4.1. Under Assumptions 4.2 and 4.1 and the cross-client statistics defined in Definition
after P rounds of federated pretraining one has
P-1
Digt S Lurr (ho)
Σ p=1
Br B2(Ap+)) + dg. + Dr), (24)
p=0
where ho is the initial global model and B2(A) a²²
(((A) = - B2()))
Proof. From Lemma B.3 we have
Lirre (hp) Lorc (hp-1) -hpp-))-8((A))1
112 11 32(Ap)07_1
P-
P-1
< Loro (ho) -
Σ
31 + Σ 32 12
0=
0=0
17

---

Following the same proof as the one outlined for Theorem we obtain
(hr) S (hrr (D Dr).
Therefore,
Ligt- = Legt (hp) S (hpp) + dD ,Dr)
S Lono (ha) (1pt Σ
B2 B2(Ap+10 dD.=(D) Dr).
C Additional Experimental Results
Computing resources. We used an NVIDIA GeForce RTX 3090 Ti GPU with memory of 24247MiB. One run of experimental evaluation takes approximately hours for CIFARIO - SVHN for the small client setting.
Fraction of Clients Participating in FL Rounds. Besides the results generated using 20% of the participating clients reported in the main text, we also report the results pertaining to 10% and 100% participating clients in Table A larger fraction participants helps with improving transferability, ie., expected, 100% participation outperforms the setting with 10% participation. However, note that for 20% of participating clients we already reach performance comparable to that involving 100% of the participants.
Method FedAVG
Coefficient
10% participants
100% participants
NA
0.52 0.01
0.56 0.03
le-4
0.52 0.02
0.58: 0.01
FedGTST
5e-4
0.58 0.04
0.63 0.05
le-3
0.53:: 0.01
0.60 0.06
Table 4: Transferability versus the fraction of participating clients in each round. We report the results for CIFAR1O + SVHN the LeNet backbone, with K = 100.
Convergence Results. Convergence results are plotted in Figure 1 We observe that model pretrained via FedGTST not only offers better transferability than baselines. but also tends to converge faster during the finetuning stage (i.e., the green lines in all plots always converge faster than the grey dashed lines).
Local Epochs. Besides the results for single local client epochs presented in main text, here also report the results for multiple local epochs in Table []] A smaller number of local epochs tends to improve transferability more than larger number of epochs, which is consistent with the intuition that FL pretraining can avoids local overfitting when using fewer epochs.
Coeffient NA
local epoch
local epochs
FedAVG
0.54
0.52=0.07
lc-4
0.60
0.52=0.04
FedGTST
5e-4
0.56+0.06
0.58=0.02
le-3
0.58+00.5
0.53++0.02
Table 5: Transferability v.s. local number of epochs. We report results for CIFARIO- with the LeNet backbone, for 10% of participating clients and K = 100.
Cross-Cliem Statisties. We plot the cross-client averaged Jacobian norm Jplll and the cross-client Jacobian variance o3 in Figure The observation is that FedGTST leads to a significantly larger || and a significantly smaller o3 compared to FedAVG. A more detailed explanation of the findings (i.e., the reason for truncating the x-axis the left plot, the FedGTST coefficient selection approach and faster convergence results in the right plot) is available in the caption of Figure 2
Additional Dataset: DomainNet. Following FedSR, we apply leave-one-out strategy, where one domain is treated as the target domain and the other five source domains, allocated to five
18

---

Finetune Performance Target Domain
Finetune Performance Target Domain
0.00
0.50
0.30
0.30
E
E,
0
0
0.48
5e-4
4
le-4
0.99
0.40
1e-3
0.3
Finetune Epochs
Finetune Epochs
Finetune Performance on Target Domain
Finetune Performance on Target Domain
f
0
0.62
0/2
1e-4
1.00
1e-4
5e-4
5e-4
0.58
1e-3
1e.3
0.56
0.4
025
0.0
0.50
Finetune Epochs
Finetune Epochs
Figure 1: Visualization of Convergence Results. We use CIFARIO - SVHN with K = 100 as example. The top two plots correspond to fraction of 10% of participating clients, while the bottom two plots correspond 100% participation. We report the training and accuracy along with finetuned epochs for both settings. The grey dashed lines represent FedAVG, where the coefficient for the regularizer term is set to 0. Other lines represent FedGTST with tuned coefficients.
individual clients. Results are reported in Table where the target domains listed are: C (Clipart), I (Infograph), P (Painting), Q (Quickdraw), R (Real), and S (Sketch).
Leave-One-Our Domain
Model
Average
C
P
Q
R
FedAVG
59.3-F0.7
16.5
44.2
10.8
57.2+-0.8
39.6
FedSR
61.0+0.6
18.6+0.4
45.2+0.5
13.4+00.6
57.6++0.2
41.3
FedGTST 63.9
20.7
47.810.4
15.2+10.5
59.5
43.6
Table 6: Comparison of different federated models intra-domain transfer tasks of DomainNet.
FedGTST consistently outperforms the other methods.
Additional Baseline: Scaffold. FedGTST consistently outperforms Scaffold, as presented in Table
and g
MNIST- MNIST-M
CIFARIO SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
Scaffold
75.6 1 0.8
80.8 - 0.3
66.0 + 0.5
71.1 0.4
73.3 74.5 + 0.3
FedGTST
76.2 0.9
82.3 = 0.5
70.1 + 0.8
75.8
Table 7: Comparison between Scaffold and FedGTST. Number of clients 10.
Dirichlet Sampling. We set the concentration parameter to 0.5 and the number of parties to 10 by default. The results in Table and TO indicate that FedGTST still outperforms others when individual domains are constructed via Dirichlet sampling.
19

---

RedAvg
Grademt
JN ccef 1e-3
250
[40, coef Se-4
p4B coof 10-4
graa
200
FedAvG
150
100
60
100
federated training round
Federated rounds
Figure 2: Cross-client statistics tuning via FedGTST. We use CIFARIO with K = 100 an example. The left plot reports the global Jacobian (gradient) norm versus the index the federated round. The grey dashed line represents FedAVG, while other lines correspond to FedGTST with different coefficients. We truncate the plot to only capture the results of the first 100 rounds, since at the end of training the gradient norm should drop value close to 0 due convergence, and we only interested in observing the hehaviour of Jacobian norms during relative early pretraining stages. We select the best-performing setup from the left plot (the red line with coefficient 1e 3), and then in the right plot, compare its variance during a federated round with that of FedAVG. The blue line represents FedAVG and the yellow line corresponds to FedGTST. The yellow line terminated earlier since all experiments are averaged over 3 runs and aligned with the run that converges the fastest.
MNIST MNIST-M
CIFARIO - SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
Scaffold
52.3 + 0.5
63.1 + 0.3
45.5 + 0.1
55.5 + 0.3
54.1
FedGTST
57.5 + 0.3
67.6
0.2
52.4 + 0.1
63.1
0.2
60.2
Table 8: Comparison between Scaffold and FedGTST. Number of clients is 100.
D Discussion of the Positivity of Coefficients Requirement for Cross-Client Statistics
Lemma D.1 (Bound on round-wise source loss). Suppose the learning rates of all clients round p are the same: x(k)
App vk: E [K], P E [P]. When Assumption and 4. hold, we have that
Lave (hp+1) S Lero (htp) -1 10++1)10 11 32(Ap+1)0f
(25)
where Jp = K
Σx
(k) p op Σか (k)
1/= K Σ
(k) 112 and B1 (A) 322 (2())
a>²
2
Remark. Lemma D. provides an upper bound of current-round cross-client loss (the proof given in Appendix [B). indicating that smaller current-round cross-client loss (LHS) may be influenced by factors such as small last-round cross-client loss, a large cross-client average Jacobian norm, and small cross-client Jacobian variance (RHS). More precisely, the LHS is the cross-client corrent-round loss, which is upper bounded by three terms on the RHS.
1) The Loss Term: Lere
(1100) is the last-round cross-client loss.
2) The Variance Term: oi measuring the variance of local gradients across all nodes.
3) The Norm Term: || measures the squared cross-client average Jacobian norm.
Positivity the Coefficients. is straightforward to see that (2((pp11 a()p+1)² the coefficient 2
front of o is always positive. Therefore, we focus on 32()n+1), the coefficient front of || Denote the RHS of Equation 25 by B, so that [.gre(hp+1) < B. To allow such bound to be used indicator that the source loss is decreasing with the number of training rounds, we require
20

---

MNIST MNIST-M
CIFARIO
SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
FedAVG
74.11:0.6
82.2:10.3
65.2.10.4
7.9
73.5
FedSR
75.5
82.0:H..2
66.1+00.5
7..1=0.3
73.9
FedIIR
75.8+-0.2
82.8-
66.5+009
74.6-0.1
74.9
Fed-GTST
77.3++0.8
82,7+0.4
70.8
75.2 0.2
76.5
Table 9: Comparison of target accuracy (%) across different methods on MNIST and CIFARIO-SVVH tasks. 10 individual domains constructed by Dirichlet sampling.
MNIST
MNIST-M
CIFARTO - SVHN
Method
Average
LeNet
ResNet
LeNet
ResNet
FedAVG
50.4+0.1
63.0#0..3
43.3+0.2
54.6=0.5
52.8
FedSR
52.7=0.2
62.9:10.3
44.7=0.1
56.5=0.3
54.2
FedlIR
54.2-10.4
64.1
47.4-0.4
58.7=0.2
56.1
Fed-GTST 5..
69.2.10.2
55.1
65.6-0.2
62.4
Table 10: Comparison of target accuracy (%) across different methods MNIST and CIFARIO-S tasks. 100 individual domains are constructed by Dirichlet sampling.
< (hp) since only in this way can the bound give Laro(hi-1) S S Lhpe(tp). Thus, we require 31 (Xp+1) to be positive, since otherwise B I (hp) cannot be meet. We describe in what follows that a positive B1 (Ap+1)) is indeed possible practice.
Realistic scenarios in which B1 (Ap-1) > To require (1(p++)) > 0 is equivalent to require Ap+1 < 2. This requirement can be easily met since for any model of a known mathematical form based second-order differentiable loss, we can easily get the lower bound for once observed all training data. Using linear regression as an example, where 1(x, yi w) (wx - y)², we
have d't 2xr, therefore, in this case dw
simply control Ap+1 max.ex ||=]|| to approximately
guarantee Ap+1 2 This then ensures 31 (Ap+1) > 0.
E Optimal Learning Rate
Choosing the Optimal Learning Rate. Lemma B.3 shows that the upper bound of federated loss at round is quadratic function w.n.t the learning rate Ap- Therefore, good learning rate at each round needs to be chosen to minimize the upper bound. For simplicity of notation, we use Bp+1 (Ap+1) as shorthand for the upper bound shown in (15).
The following two observations are in place for (Ap+1):
2K 2K <.
< 2. holds that Lero (n10) I Bp+1(Ap+1) <
When O Ap+1
Lave
hpp
indicating that the federated loss is decreasing with the number of rounds.
By minimizing Bp+ (Ap+1) w.r.t. An41. we have
K
J
K ||
X
Bp+1(Ap-1) Love (h
Σ
| B
2x Σn ||
F Stochastic Gradient Descend
Assumption 4.2 on one step of gradient descent can extended to stochastic learning with batch sampling. The additional randomness in sampling would require incorporating the variance of batch sampling into the generalization bound. This variance term, being independent of the algorithm design, was omitted in theoretical analysis.
21

---

G Dataset Description
MNIST comprises 60, 000 28 28 grayscale images of handwritten digits (0 through 9); MNIST-M created by combining MNIST digits with the patches randomly extracted from color photos of BSDS500 as their background, containing 59, 001 training images. The CIFAR-I0 dataset consists of 60, 000 32 32 colour images from 10 classes. SVHN contains 73, 257 32 x 32 colored digits obtained from house numbers in Google Street View images.
H Non-iid FL Models
For the source domains MNIST and CIFAR-10 10, we only allow each local client to have access to two out of ten classes (c.g.. for the digit dataset (0-9), one client may only have access to say digits 3 and 6). We let each client randomly chooses their labels and samples following a uniform distribution.
See the example in Fig.
Client ID
Figure An example for constructing a non-iid marginal distribution for the Cifar10 dataset allocated to 10 clients. Each client has access to only two labels. We also make sure that no samples are used by more than client.
Additional Related Work
I.1 Gradient Matching in Transfer Learning
Discussion. Gradient matching in transfer learning focuses on aligning gradients between source and target domains to facilitate better domain adaptation. Shi et al. [43 demonstrate that matching gradients across domains enhances domain generalization by making the learned representations more resilient to domain shifts. Extending this idea, Rame et al. [39] introduce the concept of invariant gradient variances, which helps maintain generalization performance even for out-of-distribution settings. The work in Pezeshki et (88 further highlights impact of Hessian alignment, showing that aligning the Hessians of the source and target domains can significantly boost generalization in gradient- based methods.
Challenges. Applying gradient alignment techniques directly to FL presents several challenges: (1) Privacy leakage- these methods can potentially compromise data privacy by necessitating access to the target domain from source domains; (2) Local overfitting- weefitting-clients train their models on local data, which can lead to overfitting within their specific domains, reducing the global model's 's generalization capabilities.
Our contribution. To address the issues described above, propose communication schemes that ensure data privacy by preventing unrestricted access to client data. Furthermore, our approach promotes global transferability by focusing on improving generalization across all clients rather than solely enhancing local domain performance.
I.2 Distinctions and Connections between Generalization and Transferability in FL
Both approaches address the challenge of non-iid data, and improving transferability may potentially enhance generalization across diverse local domains. However, they employ distinct models and
22

---

evaluation datasets: (a) Generalization of FL targets performance on heterogeneous source testing datasets, while transferable FL aims for strong performance on target dataser that may significantly differ from source training datasets. (b) Heterogeneous FL. utilizes the pretrained model for evaluation, whereas Transferable FI. assesses the finetuned model.
The methodologies also diverge. Although reducing cross-client variance is linked to better gener- alization, our method distinguishes itself from traditional heterogeneous FL by enforcing large average Jacobian norm [Jp] in the early stages. While a larger |Jpl may hinder generalization due to increased local updates and model variance, it enhances transferability by preventing premature local convergence.
Additionally, adversarially robust models offer example where improving transferability may come at the expense of generalization. As shown [41], adversarially robust models tend to have better transferability. However, since these models designed to perform well against adversarial examples or perturbations, they do not necessarily exhibit lower generalization error on clean test data. In this scenario, transferability can be unrelated to even conflict with generalization.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction reflect the facts that 1) we theoretically estab- lished that certain cross-client statistics boost transferability of federated learning, and that 2) based on our theoretical findings, we proposed a novel transferable federated learning method which significantly outperforms baselines.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in separate section, Limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: We provided necessary assumptions need for our theoretical analysis in the main text but deferred the proof' to the Appendix due space limitations. We nevertheless discussed the intuition behind our findings in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The datasets are existing open-source benchmarks, and we have included our code in the Supplementary materials document.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The datasets are existing open-source benchmarks, and we have attached our code in a Supplementary materials document.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss wwtpcc/pcccccpcccccooooooop hutee/ggguudeeeoeeeuuuubiiiiiiii wwwaanssssonppoo ssiniioiscicycc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details the experimental work are discussed in the Experimental Setting in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Our experimental results are test accuracies, and we reported the standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Sufficient information about the compute resources (type of compute workers, memory, time of execution) is provided in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: The research conducted to produce the paper conformed, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is neither negative or positive societal impact of the work performed. It still may have technological and academic impact based on its results and findings. There should be no problems with the proper use of the pertinent methods/softtarr.
27

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no risk in this domain.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers that produced the dataset used in our studies are properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper will not lead to the release of new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "3E4TBqUeVJ", "reviewer": "Reviewer_kpC8", "summary": "This work aims to improve the transferability of pre-trained models under federated learning. It established a cross-domain generalization error bound under federation. Motivated by this, the authors devise a regularization method leveraging cross-client statistics to minimize the generalization error upper bound. They evaluate the performance of the proposed method on several cross-domain transfer learning tasks.", "questions": "In addition to the above weakness points, here are the additional minor issues:\n- In the introduction, Line 46 to Line 74 seem too verbose when presenting the limitations of the existing methods. It is suggested to give a distilled summarization of these methods and defer the detailed description to the appendix or related work, and highlight the advantages of the proposed method. Besides, if these methods are not compared or simply focus on different tasks with this work, then it is not necessary to discuss them here.\n- Line 299 “We use two sets of source-domain dataset pairs for FTL” Do you mean “source-target” dataset?\n- Line 307 “we let each client have access to only part of the labels.” This indeed to some extent confusing. I suppose this intends to refer to a semi-supervised cross-domain generalization task with part of the “labels”. After checking Appendix G, I know each client holds a fractional of “classes” with full annotations. \n- Line 341, “Besides the significant performance game of FedGTST over baseline methods”. Do you mean performance “gain”?\n- Line 523, Figure C, this link is broken. \n- Line 524, “FedGTST does not only offer better transferability”. -> Remove “does”.\n- As mentioned in the weakness, to figure out the task and evaluation metric, I checked the code provided. Frustratingly, I did not find much useful instruction, and the title of the README file remains the title of another paper “On the Convergence of FedAvg on Non-IID Data”. For a submission targeting such a venue, the authors should at least provide a decent description of their code implementation and evaluation criteria for reproduction.\n- It is recommended to discuss existing works leveraging Jacobian matching in transfer learning. \n- Theoretical insights (Line 266) could be moved to Section 4.", "limitations": "Limitations are discussed sufficiently.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The reviewer appreciates the theoretical analysis provided in this paper, despite that this can only be achieved on a convex model. The authors did a good job of connecting these theoretical results to their method design. \n- The proposed method is more efficient in terms of communication cost than existing works for cross-domain generalization.", "weaknesses": "- Some important experimental settings are missing. It is not clear that (1) how the pre-trained models on source dataset are obtained (from centralized or federated pre-training?), (2) the availability of target domain data or labels. These important details are absent from the main text, appendix and code description, making it hard to understand the context of this work. The reviewer can only infer the task from some fragment of information. For example, given the evaluated experiment on CIFAR10 -> SVHN, the reviewer assumes the target domain data and labels are available for finetuning. Based on this, the task of interest is more like an inductive transfer learning task rather than domain generalization.\n- If transfer learning under federation is the main focus of this work, then the datasets considered are too small for a comprehensive comparison of evaluated methods. For example, finetuning pre-trained models learned from ImageNet on downstream datasets like Stanford Cars, and Oxford Pets [1] are frequently used in the existing literature. Otherwise, it makes no sense to transfer learn features from a small dataset to the other small dataset. \n- In terms of the compared baselines, it could be better to consider existing transfer learning approaches, since it is a direct adaptation to federated learning.\n- There remains a gap between the generalization bound and transferability measurement. In fact, it remains not clear why cross-client statistics can be introduced as a transferability indicator.\n- From Algorithm 1, I can only guess the proposed method prevents local overfitting by preventing a very small Jacobian norm. If the proposed method works in such a way, then existing works use L2 norm [1] of feature map for regularization should also be considered as baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Our answers are as below.\n* **Experimental Setting:**\n  1. *Federated Pre-training* was explained in the original manuscript.\n     In Lines 148–149, we said: “...federated pretraining where the source domain is a union of the agents’ local domains.” Hence, we indeed used federated pretraining with the source domain comprising data from all local domains. In Appendix G, Line 579, we reiterated the claim: “For the source domains MNIST and CIFAR-10, we only allow each local client to have access to…”\n  2. *Target domain data or labels* are available.\n     Target domain data and labels were indeed available, as stated in Lines 147 and 151. One can see from Equations (1) and (3) that the target loss is defined based both on the data and labels. \n  3. Yes, we are indeed performing inductive transfer learning.\n* **Additional Dataset**. We additionally run experiments on DomainNet, as suggested by another reviewer, to accommodate both requests. DomainNet is a widely used large dataset for both transfer learning and transferable federated learning, such as FedSR. Following FedSR we applied a leave-one-out strategy, where one domain is treated as the target domain and the other five are source domains. The target domains listed in the table are: C (Clipart), I (Infograph), P (Painting), Q (Quickdraw), R (Real), and S (Sketch). The results confirm that FedGTST still outperforms other methods from the literature.\n| Model     | C        | I         | P        | Q        | R        | S        | AVG  |\n|-|-|-|-|-|-|-|-|\n| FedAVG    | 59.3±0.7 | 16.5±0.9  | 44.2±0.7 | 10.8±1.8 | 57.2±0.8 | 49.8±0.4 | 39.6 |\n| FedSR     | 61.0±0.6 | 18.6±0.4  | 45.2±0.5 | 13.4±0.6 | 57.6±0.2 | 51.8±0.3 | 41.3 |\n| FedGTST   | **63.9±0.5** | **20.7±0.3** | **47.8±0.4** | **15.2±0.5** | **59.5±0.6** | **54.3±0.2** | **43.6** |\n\n* **Baseline clarification**. \n   1) We indeed considered adaptations of existing transfer learning approaches to federated learning: FedSR, one of the baselines in our comparative study, is a direct adaptation of transfer learning. We will explain how FedSR performs this adaptation in the revision. \n   2) FedSR also represents an example of using L2 norm regularization on mapped features. Also, we would appreciate it if the reviewer could clarify what reference [1] is, as we do not believe it corresponds to the reference in our submission. Other examples of using L2 regularizers in transfer learning include Li et al., Explicit inductive bias for transfer learning with CNs, etc.\n\n* **Transferability measure, theoretical bound and cross-client statistics**. \n   - *Theorem*: As in Lines 157-158, we use the optimal target loss as the transferability performance measure. More precisely, in Theorem 2, Line 251, the LHS represents the actual transferability measure. Proper tuning of the cross-client statistics influences transferability by controlling the RHS, which is an upper bound on the transferability metric on the LHS.\n  \n   - *Intuition*: Intuitively, a large averaged Jacobian norm in the early stages helps prevent clients from getting stuck in local minima or local overfitting, while a small cross-client Jacobian variance promotes similarity across domains, both of which enhance global transferability. Further clarifications regarding the Jacobian norm are: 1) the increment of the Jacobian is controlled as $\\|J_1 - J_2\\| \\leq \\alpha \\|x_1 - x_2\\|$, ensuring that the second term on the RHS of Theorem 2 cannot be $\\infty$; 2) An average Jacobian norm will naturally decrease in the final stages due to convergence.\n* We appreciate the reviewer pointing out the typos, phrasing and reorganization issues. We will correct these errors in the revised manuscript.\n* Regarding the README file, we sincerely apologize for this oversight. This was an unfortunate copy-paste error that we can easily correct since we always had a ready-to-use instruction manual for the code. Due to space limitation, we can only repeat some parts of our intended README file below:\n\n    Sample run command (i.e., 100 clients with resnet18):\n \n    python main.py --num_clients 100 –back_bone resnet18 \n\n    Our evaluation metric is implemented in the code contained in main.py, Line 207.\n* Please see the discussion below regarding gradient matching in transfer learning, and challenges of applying it directly to federated learning.\n   - **Discussion**: Gradient matching in transfer learning are approaches of aligning gradients between source and target domains. The work *Gradient Matching for Domain Generalization* explores how matching gradients between the source and target domains can lead to better domain generalization by ensuring that the learned representations are robust to domain shifts. *Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization* extends this idea by focusing on invariant gradient variances, which helps in maintaining generalization performance when faced with out-of-distribution samples. *Understanding Hessian Alignment for Domain Generalization* investigates the role of Hessian alignment in gradient-based methods, highlighting how aligning the Hessian matrices of the source and target domains can further improve generalization performance. \nAll these and other suggested works will be cited in the revised manuscript.\n   - **Challenges**: 1) Privacy leakage. Such approaches violate data privacy because source clients are given access to the target domain. 2) Local overfitting. Clients perform training solely on their local models, representations, and labels, thus  intuitively end up overfitting in the local domain. \n   - **Our contribution**: We ensure that our communication schemes prevent uncontrolled access to data, thereby maintaining data privacy, and that our approach enhances global transferability rather than focusing solely on local transferability.\n\n\nWe appreciate the reviewer’s feedback, and we are happy to provide additional clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IXZpxxg6lv", "reviewer": "Reviewer_Z3J8", "summary": "The paper presents FedGTST, a novel federated transfer learning approach that achieves transferability over the global learning domain and accurate assessment of the degree of transferability. The proposed method utilizes a new regularizer that encodes cross-client statistics and adjusts the local training process towards global transferability. This is achieved by subtractions of global norms of Jacobians communicated by the server. The authors also use two FL-specific factors, cross-client Jacobian variance and cross-client Jacobian norm as direct performance indicators. The proposed method only communicates Jacobian norms, which is a scalar, thus achieving reduced communication overhead and ensuring privacy. The authors also provide theoretical analysis and experimental results demonstrating the effectiveness of FedGTST over existing baselines.", "questions": "Please refer to the weaknesses.", "limitations": "Yes, the authors discussed the limitations of this work, including local computational cost and potentially loose bound.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall this paper is well written and tecchnically sound. The authors present a detailed literature review of existing work related to Federated Transfer Learning and point out three major drawbacks: privacy leakage, local overfitting and huge communication bottleneck. \n\nFedGTST has the following contributions:\n1. The studied problem is novel and may have practical usage in real-world applications.\n2. It uses a novel regularizer that focuses on the cross-client statistics to enhance the global transferability.\n3. It introduces the cross Jacobian variance and norm as the direct indicator of the transferability.\n4. It communicates only the scalar to reduce communication overhead and protect the privacy.\n\nThis paper also provides detailed and comprehensive analysis on the literature of Federated Transfer Learning, which eases the understanding for readers. Rigorous theoretical analysis and extensive experiments are also included to supports the proposed method.", "weaknesses": "Weaknesses\n1. The non-iid setting is not common. Most of FL methods utilize the dirichlet sampling to simulate the Non-IID distribution [1-7].\n2. Estimating the quality of the solution f* and using idea from the domain generalization is interesting. Ref [7] also considers measuring the local training through the lens of the domain gap, which can be added for discussion.\n3. The hyper-parameters are not given, which reduces the reproducibility of this work.\n4. The assumption 4.3 require the optimal learning rate, which may require delicate hyper-parameter tuning.\n5. It would be great if the authors add conceptual explanations for the proposed method.\n\n\n[1] Model-contrastive federated learning. In CVPR 2021.\n[2] No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data. In NeurIPS 2021.\n[3] Federated Learning Based on Dynamic Regularization. In ICLR 2021\n[4] Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. In ICML 2022.\n[5] DENSE: Data-Free One-Shot Federated Learning. In NeurIPS 2022.\n[6] FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In ICML 2022.\n[7] FedImpro: Measuring and Improving Client Update in Federated Learning. In ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Our answers are as follows.  \n\n1. We agree with the reviewer that there are other non iid sampling methods that may be more frequently used than ours. We therefore implemented the Dirichlet sampling. Following the reviewer’s suggested references, we set the concentration parameter to 0.5. The results clearly show that our method still outperforms others even when client sampling strategies change. \n  - Results for 10 clients:\n| Method    | MNIST → MNIST-M (LeNet)| MNIST → MNIST-M (ResNet)   | CIFAR10 → SVHN (LeNet) | CIFAR10 → SVHN (ResNet) | Average | \n|--------------|--------------|-------------|--------------|-------------|--------| \n| FedAVG   | 74.1±0.6 | 82.2±0.3 | 65.2±0.4  | 72.8±0.9 | 73.5 | \n| FedSR     | 75.5±0.8 | 82.0±0.2  | 66.1±0.5 | 72.1±0.3 | 73.9 | \n| FedIIR      | 75.8±0.2 | **82.8±0.6** | 66.5±0.9 | 74.6±0.1 | 74.9 | \n| FedGTST| **77.3±0.8** | 82.7±0.4 | **70.8.0±0.7** | **75.2±0.2** |**76.5** |\n\n- Results for 100 clients:\n| Method    | MNIST → MNIST-M (LeNet)| MNIST → MNIST-M (ResNet)   | CIFAR10 → SVHN (LeNet) | CIFAR10 → SVHN (ResNet) | Average | \n|--------------|--------------|-------------|--------------|-------------|--------| \n| FedAVG     | 50.4±0.1    | 63.0±0.3      | 43.3±0.2     | 54.6±0.5       | 52.8          |\n| FedSR       | 52.7±0.2    | 62.9±0.3      | 44.7±0.1      | 56.5±0.3      | 54.2          |\n| FedIIR        | 54.2±0.4    | 64.1±0.1      | 47.4±0.4     | 58.7±0.2       | 56.1         |\n| FedGTST  | **59.5±0.3**  | **69.2±0.2** | **55.1±0.1**  | **65.6±0.2**  | **62.4**  |\n\n\n2. Thank you for your suggestion. We will add a discussion of  [7] into the revision as follows.\nA recent work, FedImpro [7], considers the domain generalization problem over different clients in the FL setting. Specifically, FedImpro assumes that the server can maintain an estimate of the aggregated feature distribution, which is used as an extra term in the client gradient computation to promote lower gradient dissimilarity and thus better generalization ability. Unlike our approach, FedImpro does not use cross-client statistics explicitly during client updates.\n\n3. We would like to point out that the hyperparameters are described in Lines 321-333. If the reviewer would like additional information, please do let us know.\n\n4. The optimal learning rate in assumption 4.3 can be directly computed in practice. An assumption that we made in 4.3 is that we can know the exact value of $\\alpha$, as all other terms can be computed explicitly from the client statistics of round p-1. The assumption is only used for theoretical analysis. In practice we can always estimate $\\alpha$. This type of assumption is also adopted in other papers, such as [Haddadpour, 19].\n\n    [Haddadpour, 19] Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425, 2019. \n\n\n5. We appreciate the reviewer’s suggestion. A detailed conceptual explanation of Theorem 2 is follows:\n\n* **Tuning Cross-Client Statistics Improves Transferability:**\n  - **Large $\\|J_p\\|$ Norm:** A large $\\|J_p\\|$ intuitively enhances transferability. In the early stages of training, it is crucial to prevent the averaged gradient from becoming too small. Small local gradients can cause clients to become trapped in local minima or overfit. As training progresses toward its final stages, the averaged gradient norm will naturally decrease due to convergence.\n  - **Small Cross-Client Gradient Variance:** A small cross-client gradient variance improves transferability by inducing similarity across different domains. This helps avoid local overfitting and promotes better global transferability.\n\n* **Clarification on Gradient Norm Behavior:**\n    It is impossible for the right-hand side (RHS) of the equation to be $ -\\infty$. The averaged gradient norm $\\|J_p\\|$ in term 2 of the RHS cannot approach $\\infty$ because it is bounded by the smoothness parameter $\\alpha$. Specifically, the increment of local gradients is upper-bounded by $\\|J_1 - J_2\\| \\leq \\alpha \\|x_1 - x_2\\|$.\n\n* **Trade-Off Between Enlarging $ \\|J_p\\| $ and Decreasing $ \\sigma $:**\nA larger $\\|J_p\\|$ results in more substantial updates of the local models, which can intuitively lead to an increased variance $\\sigma_{p+1}$ in subsequent rounds. Therefore, Theorem 2 highlights that when implementing the approach, it is essential to balance this trade-off—i.e., increase $\\|J_p\\| $ while keeping $\\sigma$ small.\n\nWe appreciate the reviewer’s feedback, and we are happy to discuss further with the reviewer to clarify any remaining questions or concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sb1KnawWtR", "reviewer": "Reviewer_RowG", "summary": "This work addresses transferability of models pre-trained using Federated Learning (FL). Through a thorough theoretical analysis, the authors identify cross-client averaged Jacobian norm and cross-client Jacobian variance as a key factor influencing transferability. To retain good transferability,  FedGTST controls cross-client averaged Jacobian norm through a guide norm determined by the server while the cross-client Jacobian variance is controlled through local regularisation at the clients based on the guide norm. Different from existing works, FedGTST prevents privacy leakage, and maintains communication efficiency by only communicating additional scalars.", "questions": "1) Does the experimental setup consist of 1 local step or 1 local epoch (similarly in Table 5)?.  The theoretical analysis assumes one local step, however the experiments mention one local epoch. \n2) On the same lines, does the experimental evaluation use GD or SGD? Can the authors show empirical evaluations with SGD and multiple local epochs if not already?\n3) What is the effect of removing the surrogate Jacobian norm evaluation with standard training? Can the authors show an ablation study where there are no clients computing $\\gamma_p^{std, (k)}$ and clients only send $\\gamma_p^{reg, (k)}$ ?", "limitations": "1) Missing important non-IID FL baselines (comment (1) in weaknesses).\n2) Lacking discussion of cross-client average Jacobian norm and its relation to data heterogeneity (comment (2) in weaknesses).\n3) Certain assumptions seem impractical (comment (3) in weaknesses).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Thorough theoretical analysis quantifying loss on target domain in terms of the constituents from FL training.\n- Proposed algorithm is communication efficient, satisfying a crucial requirement of federated settings.\n- Proposed algorithm requires clients to transfer only scalar values in addition to locally updated parameters, preventing any further privacy leakage beyond sharing model parameters.\n- Empirical results cover different model sizes, networks with up to 100 clients and non-IID local domains, simulating diverse FL setups.", "weaknesses": "1) The variance of cross-client Jacobian appears to be a property of the non-IID local domains. For instance, when clients have IID data, the variance would be inherently low and thus naturally promote good transferability. Thus, when considering non-IID local domains, the proposed algorithm should be compared for transferability against advanced FL algorithms that address data heterogeneity and therefore would naturally control cross-client Jacobian variance, e.g. FedAdam, Scaffold and FedNova. FedAvg seems to be a weaker baseline.\n\n2) While the theoretical analysis identifies cross-client Jacobian as a key influencing factor for transferability, intuitive explanations regarding the same are missing. Why does inflating norms help transferability? Does a higher or lower average norm during the initial stages affect the generalisation error of the FL pre-training and consequently transferability? If this is true, then the quest reverts back to learning good FL models under non-IID data instead of transferability. It would be great if authors could provide detailed insights/discussion regarding this.\n\n3) Certain assumptions in the analysis seem too strong and could be impractical – i) one local step (assumption 4.2) and ii) Gradient Descent (GD) instead of Stochastic Gradient Descent (SGD) (assumption 4.2). FL hinges on the idea of executing multiple local steps to achieve communication efficiency. Furthermore, most standard FL/ML algorithms execute SGD due to the well known computationally expensive nature of GD which evaluates a single gradient on the entire local dataset. The authors should clarify the implications of loosening these two assumptions on the theoretical analysis.\n\nMinor comments:\n\n1) The paper has several typos making reading comprehension difficult:\n- Line 136: “does not communicating information that …” \n- Line 319: “We do not report results for … FedGTST …”\n- Line 8 in Algorithm: $\\gamma_p^{unl, (k)}$\n- Line 523 (Incorrect reference): “Convergence results are plotted in Figure C”. \n\n2) Convexity assumption is certainly not easy to meet in practice for today’s ML. Neural network loss functions are non-convex which are the heart of most of the modern ML.\n\nThe reviewer is happy to reconsider the rating once the authors address/clarify the above issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Our answers are below. \n1. **Additional Baseline**. We discuss Scaffold, a baseline suggested by the reviewer (and will cite this and other suggested works). They are as follows:\n\n* 10 clients:\n|   | MNIST → MNIST-M (LeNet)|  MNIST → MNIST-M (ResNet) | CIFAR10 → SVHN (LeNet)| CIFAR10 → SVHN (ResNet) | Average|\n|--------|-------|---------|------|------|------|\n| Scaffold     | 75.6 ± 0.8 | 80.8 ± 0.3| 66.0 ± 0.5 | 71.1 ± 0.4| 73.3      |\n| FedGTST  | **76.2 ± 0.9** | **82.3 ± 0.5**| **70.1 ± 0.8** | **74.5 ± 0.3**| **75.8**      |\n\n* 100 clients:\n|  | MNIST → MNIST-M (LeNet)|  MNIST → MNIST-M (ResNet) | CIFAR10 → SVHN (LeNet)| CIFAR10 → SVHN (ResNet) | Average|\n|--------|-------|---------|------|------|------|\n| Scaffold    | 52.3 ± 0.5 | 63.1 ± 0.3| 45.5 ± 0.1 | 55.5 ± 0.3| 54.1       |\n| FedGTST  | **57.5 ± 0.3** | **67.6 ± 0.2**| **52.4 ± 0.1** | **63.1 ± 0.2**| **60.2**      |\n\n2. **Intuitive explanations for Theorem 2**\n* *Tuning Cross-Client Statistics Improves Transferability*.\n\n    -  Larger $|J_p|$ in initial stages enhances transferability. By preventing small averaged gradients, we prevent clients from becoming trapped in individual's local minima or overfitting locally. As training progresses, the averaged gradient norm naturally decreases due to convergence.\n   - Smaller $\\sigma_p$ induces similarity across different domains. This helps avoid local overfitting and improves global transferability.\n* *Clarification on Gradient Norm Behavior*.\n     \n    The RHS of the equation cannot be $-\\infty$. The averaged gradient norm $\\|J_p\\|$ in term 2 of the RHS is bounded by the smoothness parameter $\\alpha$. Specifically, the increment of local gradients is upper-bounded by $\\|J_1 - J_2\\| \\leq \\alpha \\|x_1 - x_2\\|$.\n\n* *Trade-Off Between Enlarging $ \\|J_p\\| $ and Decreasing $ \\sigma $*\n    \n    A larger $|J_p|$ leads to more substantial local model updates, increasing variance $\\sigma_{p+1}$ in subsequent rounds. Thus, it's essential to balance this trade-off.\n* *Higher initial average Jacobian norms do not necessarily decrease generalization error.*\n\n    Let's begin by comparing the definitions of generalization error and transferability as defined in our manuscript.\n\n    * _Generalization error_ is defined as $L_{D_T}(h_{\\text{pretrained}}) - L_{D_S}(h_{\\text{pretrained}})$.\n\n    * _Transferability measure_ in our paper is $L_{D_T}(h_{\\text{finetuned}})$.\n\n    The key differences between these two concepts are: a) Whether $h$ is finetuned; b) Whether source error is subtracted.\n\n    From Theorem 2, we can see that cross-client statistics help the transferability defined by us by controlling its upper bound, and we provide an intuitive explanation above.\n\n    However, tuning of statistics does not necessarily decrease generalization error. Specifically:\n\n    1. $L_{D_T}(h_{\\text{pretrained}})$ is not necessarily smaller, since even if the pretrained feature extractor $f_{\\text{pretrained}}$ transfers well to the target domain, the pretrained classifier $g_{\\text{pretrained}}$, without any finetuning on the target domain, can result in a high $L_{D_T}(h_{\\text{pretrained}})$.\n\n    2. $L_{D_S}(h_{\\text{pretrained}})$ is not necessarily larger, as tuning the statistics intuitively prevents clients from getting stuck in their individual local minima or overfitting, thus reducing cross-client source loss.\n\n    To summarize, tuning the statistics may lead to a higher $L_{D_T}(h_{\\text{pretrained}})$ and a lower $L_{D_S}(h_{\\text{pretrained}})$, and therefore, does not decrease generalization error.\n\n3. **Assumptions Clarification**\n\n   * The single-step assumption is commonly used in FL theoretical analyses - see FedSplit [2] and FEDAC [3].\n\n       [2] Pathak, Reese, and Martin J. Wainwright. \"FedSplit: An algorithmic framework for fast federated optimization.\" Advances in neural information processing systems 33 (2020): 7057-7066.\n\n       [3] Yuan, Honglin, and Tengyu Ma. \"Federated accelerated stochastic gradient descent.\" Advances in Neural Information Processing Systems 33 (2020): 5332-5344.\n\n   * GD assumption\n\n       The reviewer's point is correct. However, GD can be extended to stochastic learning with batch sampling. The additional randomness in sampling would require incorporating the variance of batch sampling into the generalization bound. This variance term, being independent of the algorithm design, was omitted in our theoretical analysis. \n\n4. **Minor clarifications**\n\n    * We appreciate the reviewer pointing out typo issues, and will make sure to fix them.\n    * Convexity assumptions are solely made for theoretical tractability, and such simplifications were also considered in SCAFFOLD [1], FedSplit [2], etc.\n\n      [1] Karimireddy, Sai Praneeth, et al. \"Scaffold: Stochastic controlled averaging for federated learning.\" International conference on machine learning. PMLR, 2020.\n\n      [2] Pathak, Reese, and Martin J. Wainwright. \"FedSplit: An algorithmic framework for fast federated optimization.\" Advances in neural information processing systems 33 (2020): 7057-7066.\n\n5. **By epoch in experiments, we mean step.** In the theoretical part of the paper, we dealt with steps, while in practical evaluations, we referred to epoch. Sorry for the confusion, we will clarify this in the revision.\n\n6. **We use SGD for experimental evaluation.**\n\n\n7. **Ablation Study: removing the surrogate Jacobian norm**. \n    \n    What the reviewer is asking is part of our additional baseline cross-examination involving Scaffold, which uses Jacobian regularization but does not consider surrogate Jacobian norm. Please see the results presented in the first bullet point.\n\nWe appreciate the reviewer’s feedback, and we are happy to discuss further to clarify any remaining questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bR6Tisf1Z", "reviewer": "Reviewer_SWUE", "summary": "In this paper, the authors proposed a method to address the problem of transfer learning under FL. Specifically, after a thorough theoretical analysis, the authors observe that the cross-client averaged Jacobian norm and its variance control the bound on the target loss. Therefore, they propose FedGTST to properly increase the norm while reduce its variance. The proposed method achieves promising results on several benchmark datasets.", "questions": "Please refer to the Weaknesses section.", "limitations": "The authors have discussed the limitations of the proposed method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The structure and writting of the paper is clear.\n2. The proposed method is effective, and the executed experiments are thorough.\n3. The theoretical analysis is sound.", "weaknesses": "1. The adopted benchmark datasets are a bit easy. Since the authors already used ResNets in the experiments, comparing on more complex datasets, e.g., PACS, DomainNet would make the experimental results more convicing.\n2. From the definition introduced in Section 3, only the classifier will be optimized during the federated communication. Is the proposed method also effective when the feature extractors are also optimized during FL?\n3. Is the proposed method also applicable to settings when there exists mutliple source domains?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Our answers are as follows.  \n\n1. **We additionally conducted experiments on DomainNet, and the results are presented below.** \n\n    Following FedSR, we applied a leave-one-out strategy, where one domain is treated as the target domain and the other five are source domains. \n* The target domains listed in the table are: C (Clipart), I (Infograph), P (Painting), Q (Quickdraw), R (Real), and S (Sketch). \n|Model  | C        | I         | P        | Q        | R        | S        | AVG  |\n|--------|----------|-----------|----------|----------|----------|----------|------|\n| FedAVG   | 59.3±0.7 | 16.5±0.9  | 44.2±0.7 | 10.8±1.8 | 57.2±0.8 | 49.8±0.4 | 39.6 |\n| FedSR     | 61.0±0.6 | 18.6±0.4  | 45.2±0.5 | 13.4±0.6 | 57.6±0.2 | 51.8±0.3 | 41.3 |\n|FedGTST| **63.9±0.5** | **20.7±0.3** | **47.8±0.4** | **15.2±0.5** | **59.5±0.6** | **54.3±0.2** | **43.6** |\n\n2. **The proposed method is effective when the feature extractors are also optimized during FL.** \n    \n    In our setting, the classifier is not the only component optimized during federated communication. In addition to the classifier, the feature extractor is also updated. As stated in Line 152, \"Let $h^*=g^*\\circ f^*$ be an optimal solution for the source objective 2\", which indicates that both the feature extractor f and the classifier g are optimized during federated pretraining.\n\n3. **Our current setting indeed assumes multiple source domains.** \n\n    As mentioned in Lines 148–149, \"...federated pretraining where the source domain is a union of the agents’ local domains,\" we use multiple domains for pretraining. Furthermore, in Lines 302–308 (Experiments), we explained the construction of the agents’ local domains: \"We … to construct non-iid local domains\". In Appendix G, Line 578, we reiterate, \"For the source domains MNIST and CIFAR-10, we only allow each local client to have access to…\". \n\n    Additionally, in our response to the reviewer’s question about adding results on DomainNet, our setting also involves letting clients have access to different domains, enabling multiple source domains.\n\nWe appreciate the reviewer’s feedback, and we are happy to discuss further with the reviewer to address any remaining questions or concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and AC,

We would like to thank all reviewers for their careful reading and insightful suggestions on our paper. We have prepared a detailed, point-by-point response to each concern raised by the reviewers. Below, we summarize the major questions posed by multiple reviewers:

- Theory: We have clarified our problem formulation, assumptions, and the intuition behind our theoretical results. Specific answers to the questions can be found in our detailed response below.

- Simulation: We have clarified our experimental setup (i.e., hyperparameters), conducted new experiments on the DomainNet dataset, compared our methods with the Scaffold baseline, and used Dirichlet sampling to distribute data across different clients. In almost all new settings, our method still achieves state-of-the-art performance. Detailed performance metrics for each method are provided in the tables below.

- Paper Presentation: We have corrected several typos. We appreciate the reviewers' attention to these details.

We hope our responses adequately address the reviewers' concerns. We are open to further discussion if there are any additional questions.

Best Regards,
The Authors

Meta-review: The paper presents FedGTST, a novel federated transfer learning approach that achieves transferability over the global learning domain and accurate assessment of the degree of transferability. The proposed method utilizes a new regularizer that encodes cross-client statistics and adjusts the local training process towards global transferability. This is achieved by subtractions of global norms of Jacobians communicated by the server. The authors also use two FL-specific factors, cross-client Jacobian variance and cross-client Jacobian norm as direct performance indicators. The proposed method only communicates Jacobian norms, which is a scalar, thus achieving reduced communication overhead and ensuring privacy. The authors also provide theoretical analysis and experimental results demonstrating the effectiveness of FedGTST over existing baselines.

FedGTST has the following contributions:

1.	The studied problem is novel and may have practical usage in real-world applications.

2.	It uses a novel regularizer that focuses on the cross-client statistics to enhance the global transferability.

3.	It communicates only the scalar to reduce communication overhead and protect the privacy.

4.	Overall this paper is well written and tecchnically sound. The authors present a detailed literature review of existing work related to Federated Transfer Learning.

Weakness:

1.	Experiments: In the original version, the datasets are too small, the Non-IID setting is not commom, and the baseline is not sufficient. During the rebuttal, authors add experiments on a large dataset DomainNet, and use Dirichlet sampling for Non-IID setting.  When considering non-IID local domains, the proposed algorithm should be compared for transferability against advanced FL algorithms that address data heterogeneity and therefore would naturally control cross-client Jacobian variance, e.g. FedAdam, Scaffold and FedNova. FedAvg seems to be a weaker baseline. Authors add a new experiments for Scaffold.

2.	Certain assumptions in the analysis seem too strong and could be impractical –Gradient Descent (GD) instead of Stochastic Gradient Descent (SGD) (assumption 4.2). Most standard FL/ML algorithms execute SGD due to the well known computationally expensive nature of GD which evaluates a single gradient on the entire local dataset.
Decision: Accept (poster)