Subgraph Federated Learning for Local Generalization
Authors: Sungwon Kim
Yoonho Lee
Yunhak Oh
Namkyeong Lee
Sukwon Yun
Junseok Lee
Sein Kim
Carl Yang
Chanyoung Park
Venue: ICLR 2025 Oral (2025)
Keywords: Graph Neural Networks
Graph Federated Learning
Primary area: learning on graphs and other geometries & topologies

Abstract: Federated Learning (FL) on graphs enables collaborative model training to enhance performance without compromising the privacy of each client. However, existing methods often overlook the mutable nature of graph data, which frequently introduces new nodes and leads to shifts in label distribution. Since they focus solely on performing well on each client's local data, they are prone to overfitting to their local distributions (i.e., local overfitting), which hinders their ability to generalize to unseen data with diverse label distributions. In contrast, our proposed method, FedLoG, effectively tackles this issue by mitigating local overfitting. Our model generates global synthetic data by condensing the reliable information from each class representation and its structural information across clients. Using these synthetic data as a training set, we alleviate the local overfitting problem by adaptively generalizing the absent knowledge within each local dataset. This enhances the generalization capabilities of local models, enabling them to handle unseen data effectively. Our model outperforms baselines in our proposed experimental settings, which are designed to measure generalization power to unseen data in practical scenarios. 
Our code is available at https://github.com/sung-won-kim/FedLoG

Published as a conference paper at ICLR 2025
SUBGRAPH FEDERATED LEARNING FOR LOCAL GENERALIZATION
Sungwon Kim
Yoonho Lee
Yunhak Oh"
25 Namkyeong Lee'
Sukwon Yun 3 Junseok Lee'
Sein Kim
@
Carl Yang [3 & Chanyoung Park [
1 KAIST, UNC Chapel Hill, Emory University
ABSTRACT
Federated Learning (FL) graphs enables collaborative model training enhance performance without compromising the privacy of each client. However, existing methods often overlook the mutable nature of graph data, which frequently intro- duces new nodes and leads to shifts in label distribution. Since they focus solely on performing well on each client's 's local data, they are prone to overfitting to their local distributions (i.e., local overfitting), which hinders their ability to generalize unseen data with diverse label distributions. In contrast, our proposed method, Fed- LoG, effectively tackles this issue by mitigating local overfitting. Our model gener- ates global synthetic data by condensing the reliable information from each class representation and its structural information across clients. Using these synthetic data as a training set, we alleviate the local overfitting problem by adaptively gener- alizing the absent knowledge within each local dataset. This enhances the general- ization capabilities of local models, enabling them to handle unseen data effectively. Our model outperforms baselines in our proposed experimental settings, which are designed to measure generalization power to unseen data in practical scenarios. Our code is available at
com/ sung won xim/FedLoG
INTRODUCTION
In the realm of Graph Neural Networks (GNNs) (Hamilton 2020). most systems designed for
unified, centralized graph. However, real-world applications Zhang et al. (021a) frequently involve individual users or institutions maintaining private graphs, isolated due to privacy concerns. Graph Federated Learning (GFL.) (Liu al. 2024) provides a solution by enabling clients independently train local GNNs on their data. This decentralized training approach allows a central server to aggregate the locally updated weights from multiple clients, creating unified model that respects privacy constraints. In this paper, among the various settings in GFL, we focus on one of the most challenging aspects-distributee distributed subgraphs (subgraph-FL.), where clients manage largely disjoint sets nodes and their edges.
In weal-world scenarios, graph data frequently changes, particularly in social, citation, and e-commerce networks (Sen al., 2008 McAuleyy et al.) 2015) [Shchur al. 2018). While these changes often result in new label distribution patterns that are distinct from the existing local label distribution within each client, existing subgraph-FL. methods (Zhang al. 202 la Wu al.,
202 la;
Yao et al.)
2024] Back et al. 2023) primarily focus on optimizing models based on the current label distribution within each client (i.e.. local optimization). On the other hand, some studies (Zhang et al.
2022 Lee
ct al. 2022 Guo et al. 2024) demonstrate that client models are particularly prone to local overltting after local updates, resulting in significant decrease in the accuracy of minority classes (i.e., tail classes) within the local data. Given these limitations, current approaches face substantial practical challenges, particularly in adapting to new nodes added to the original local graph. This is especially difficult for nodes belonging to tail classes or unseen classes that are missing from from the local graph but exist in other graphs (i.e., missing classes). These nodes, which form new connections with existing nodes, often have structural patterns unfamiliar to local clients, leading to substantial discrepancies in both label and structural distributions of the local graphs.
Existing methods in FL (Li & Zhan 2021 Zhang et al., 2022 Lee et al. 2022 Guo et al. 2024
aim to ensure that local models can make predictions for all classes without bias by mitigating local
Corresponning author (cy-parkee kaai st. .c.krr)

---

Published as a conference paper at ICLR 2025
overfitting caused by the local label distribution. Specifically, they propose regularizing the logits of each class in the local models to align more closely with those of the global model. While these methods effectively address local overfitting and manage tail or missing classes, increasing the logits of local tail data risks amplifying noisy data, which is harmful for the class representation of the global model. Beyond FL. another common approach to mitigating the problem of overfitting on training data involves addressing class imbalance. Techniques such as down-sampling (He & Ma 2013). over-sampling (Chawla et al. 2002 Zhao et al., 2021] He & Bail 2008), calibration (Niculescu & Caruana Zadrozzyy & Elkan 2001), or constructing expert models for tail data (Menon et al., 2020; Yun ct al, 2022) are commonly used. Despite their effectiveness, they require at least one data point to be present for each class, facing challenges when class is missing in a local client while present in others.
In this paper, we propose to address the local overfitting issue of subgraph-FL by introducing reliable global synthetic data that mitigate class imbalance while addressing missing classes. Specifically, we aggregate knowledge from local data across all clients for each class, and integrate it into the global synthetic data. Subsequently, each client adaptively utilizes the global synthetic data as additional training data to ensure effective learning for all classes, including those that are underrepresented or missing in each client. This strategy helps prevent local overfitting even after local updates and enables accurate class representation (i.e., local generalization). However, there exist two crucial challenges that need consideration:
C1. Which data across all clients should be aggregated to ensure reliability? Since clients in FL heavily rely on knowledge from other clients to learn locally absent information, it is crucial for cach client to share the most reliable knowledge within its own local graph. Here, data reliability refers to the accuracy and consistency of information sourced from decentralized nodes.
C2. How can data from other clients be utilized without compromising privacy? While direct sharing of the data between clients prevents local overfitting, raises severe privacy concerns. Furthermore, directly using training data from all clients incurs high communication costs.
Solution to C1. Knowledge from head degree and head class nodes. Our findings indicate that nodes with a high number of connections (i.e., head degree) and those belonging to the majority class (i.e., head class) provide reliable structural and class-representativv information, respectively, which significantly enhances the model's ability to generalize to unseen data. Building on these insights, we aggregate knowledge from clients and filter it based on the "headness" of both their degree and class.
Solution to C2. Data condensation. We propose condense only reliable knowledge into synthetic data to share across the clients. This avoids the direct use of individual client data, mitigating the privacy concerns, while also minimizing the amount of data transferred between the server and local clients, thereby lowering communication costs.
In summary, we propose a subgraph Federated Learning framework for Local Generalization, FedLoG, that generates global synthetic data with a novel reliable knowledge condensation strategy. This approach reduces the risk of noise in class representations, and enables each client to compensate for locally absent knowledge without compromising privacy. By doing so, FedLoG prevents local overfitting and ensures a well-generalizer representation of all classes, enabling successful handling of unseen data in our three proposed evaluation settings; 1) Unseen Node: New nodes with seen classes are added, and they introduce structural changes in local graphs. 2) Missing Class: New nodes with classes previously absent in the client's 's graph are added. 3) New Client: A completely new graph with distinct label distribution and structure is added.
In this paper, we make the following contributions:
We introduce FedLoG, the first work in subgraph-FL that focuses preventing local overfitting, including the issue of missing classes, to address the mutable nature of the graph domain. This approach enhances the performance of the global model and improves the generalization power of local models, allowing them to effectively handle unseen data.
We analyze what constitutes reliable data in graph- federated learning and propose a a method to condense and share this knowledge across clients. This approach not only leverages reliable data effectively but also protects privacy by using condensed synthetic data.
We propose practical and important evaluation settings on unseen data for subgraph-FL (i.e., Unseen Node, Missing Class, and New Client), enabling measurement of the model's generalization future data, assessing robustness in mutable graph domains, and demonstrating consistent outperformance over other baselines.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 SUBGRAPH FEDERATED LEARNING
Recent works (He et al. 2021: Liu et al.) 2024) have introduced FL frameworks that enable col- laborative GNN training without sharing graph data. Subgraph-FL aims to leverage disjoint graphs from each local client to collaboratively train global model for solving downstream tasks. Existing studies (Zhang et al. 202 Yao al., 2024, Wu et al. 202 Liu et al. 2024) have attempted to supplement the local absent knowledge among local graphs that each client currently holds. For in- stance, FedSAGE+ (Zhang et al. 2021a), FedGNN (Wu et al. 2021a), la), and FedGCN (Yao et al. 2024)
request node information from other clients to recover missing neighborhood nodes and compensate for potential edges. FedPUB (Back et al. 2023) and FedStar (Tan et al., 2023) aim to personalize local models by adapting the global model to specialize in the local data of each client. However, due to the mutable properties of graph domains, subgraph-FI. must generalize well not only to the current label distribution but also to new nodes that will emerge in the future. Unlike these approaches (Zhang et al., 202 la; Wu et al. 2021a; Yao et al., 20244 Back et al. 2023) that only focus on finding missing ledge relevant the current state, our model learns representations for all classes and their connection patterns, ensuring better generalization across various future scenarios.
2.2 LOCAL OVERFITTING IN FEDERATED LEARNING
Imbalanced data distribution is common in world scenarios, and significant efforts (Cui et al.
2019|Menon Menon et al.] 2020 Tan et al., 2020 Yun al. 2022 Li et al. [022] Ma et al.. 2023) have been made to address the resulting deterioration in model performance. Federated learning, the task at hand, inherently faces the data imbalance problem as well. Specifically, the involvement of multiple clients means that each client has its own imbalanced dataset, making local models prone overfitting to
their local data (Zhang et al. 2022 Lee et al. [022] Guo et al. 2024). Recent works (Chen et al.)
2023 Lee al. 2022 Zhang et al.
2022 Guo et al. 2024) aim to alleviate local overfitting FL
by regularizing local models to be similar to the global model. FedHKD (Chen et al. 2023), Fedl C
(Zhang al. 2022). and FedEl (Guo al., 2024 introduce logit calibration, which aligns the logits of each class in the local models more closely with those of the global model. While FedED (Guo) ct al. 2024) addresses the missing class problem in FL, it does not consider the noisy properties of tail data (Subramonian et al.) 2024 Liu et al., 2021 Wu et al., 2021h Xiao et al. 2021). Our method, FedLoG addresses local overfitting and ensures reliable representation of all classes by leveraging class-specific knowledge across clients and considering their structural properties. To the best of our knowledge, this is the first work to tackle local overfitting with missing classes in subgraph-FL.
PRELIMINARIES
Notations. We use G = (V,E) to denote a graph with the set of nodes V and the set of edges E C V x
V. The dataset D = (G. Y) includes labels Y for the nodes that belong to one of |Cyl distinct classes, and Xy E R | is the feature matrix with d as the feature dimension, where each node u E V is associated with a feature vector X, E Rdd. In subgraph-FL, server and K clients manage disjoint subgraphs GR- = (V2, &&) for each client k. The global set of nodes is V = UK V2 with V Ø for all * j. The local dataset for client & is Dr. (Gk, Vie,, and the combined local dataset is Diocal UK_ Dr. Additionally, we generate a global synthetic set Delabal = (Galobal, ylgobal),, where gelnbal = (Yoloball ,Ex) consists of isolated nodes Ug E Vglobal with no edges& Eg. Vgllbbal includes s nodes per class, totaling s x |Cyl nodes.
Problem Statement. We aim to develop a distributed learning framework for collaborative train- ing of node classifier. Specifically, the classifier F uses optimized parameters & minimize a predefined task loss. The objective is to find global parameters Φ² that minimizes the aggre- gated local empirical risk R. defined as: o² = arg ming R(F(@)) = I (1)),, where (F)())) == Yo) L. : M. Enen 1(0; (u()),, Vou) + ppuiaar Engerpoal l(Ø; Fg.. Vo,).. To allow each client to gen- eralize across all classes, including missing classes, we generate global synthetic data Delobal and introduce an additional loss term to take into account this data prevent local overfitting.
RR (R((F
(.),)Y)), and the task-specific loss LG is defined as:
3.1 WHICH DATA ARE RELIABLE?
Data reliability refers to the accuracy and consistency of information from decentralized nodes, crucial for training models across varied environments (i.e., clients). Inspired by the robust performance of
3

---

Published as a conference paper at ICLR 2025
GNNs on head class and head degree nodes (Yun et al. 2022 Park al. 2021 Zhao al. 2021 Liu
jt al. 2021), we found that data reliability largely depends on 1) the extent data connections (i.e., degree headness) and 2) the predominance of certain classes (i.e., class headness).
To corroborate our arguments, we measured the target class accuracy of a client (receiver) receiving information (i.e., weights) from other clients (contributors). To check how degree/class headness varied the contributors' training sets, adjusting the
of the contributors impacts the receiver, degree/class headness of the training nodes, while keeping the receiver's ' training set constant. The
global model was constructed by averaging client weights and evaluated on the receiver's local graph.
Figure topp shows receiver accuracy with contributor training sets composed of 1) "Only head degree nodes', 2) *Only tail degree nodes', and 'Balanced (head+tail) degree nodes'. The receiver's performance is better when knowledge is received from head degree nodes, indicating the reliability of knowledge from head degree nodes over that from tail degree nodes. Moreover, as the number of clients increases, the performance gap widens, highlighting the accumulated negative impact of noise within tail degree nodes.
Receiver's Accuaty
Ta
Dag
Deg.
0.6
0.4
Clients
1.0
Figure [bottom) manipulates label distribution within each contrib- utor's local graph, transforming the target class into tail head class by varying the number of training nodes in other classes while keeping the target class constant. When the class headness of the tar- get class within contributors is high (i.e., negative imbalance rates), contributors enhance the receiver's 8 performance. Conversely, with low headness (i.e., positive imbalance rates), the receiver's perfor- mance deteriorates contributors struggle to represent the target class (Yun al. 2022). This negative impact is magnified the number of clients increases.
Neceiver's Acuurscy
dertt
diets
0.9
HEAD
TAIL
Class Imbalance Rate
Figure Analysis
Data Reliability PubMed used).
In summary, data with 'headness' in both degree and class from other clients (i.e., contributors) helps the target client (i.e., receiver) learn reliable representations, while "tailness' data negatively impacts the model training due to insufficient or noisy information. Building our observations, our method, FedLoG, collects knowledge from head degree and head class data across all clients to alleviate locally absent knowledge.
PROPOSED METHODOLOGY: FEDLOG
Our proposed subgraph-FI. framework, FedLoG, works as follows. Figure shows the overall framework of FedLoG.
Step Local Fitting (Section 4.1
The server initializes the local model parameters K clients with the parameters of the global model gselobal, Each local model is then trained using local data Dr. Concurrently, head degree and tail degree knowledge are condensed into synthetic nodes within each client, denoted as Va.head and tatail
Step - Global Aggregation and Global Synthetic Data Generation (Section 4.2) : After local training, the server aggregates the local models to create the global model giglenn and generates global synthetic data Delobal by aggregating Vee for all k; weighted according to the head classes within each client k.
Step - Local Fitting (Section 4.1 & Local Generalization (Section 4.3): ; At the start of each round, local fitting is performed first. After local fitting, local models are generalized using Dglobal which possess both head degree and head class knowledge, adaptively learning the locally absent knowledge.
While the framework starts with Step (r- = 0), it continues to alternate between Steps 2 and 3 until the final round R is reached (r 2 1). In summary, our method extracts head degree knowledge at the client level and head class knowledge at the server level, then condense them into the global synthetic data, which is utilized train the local model during Local Generalization adaptively compensate for the locally absent knowledge within each client. Algorithm | outlines the algorithm of FedLoG.
Please refer to Appendix A.8 8 for detailed deseription of experimental settings.
4

---

Published as a conference paper at ICLR 2025
Server
[oll Local model
[x2] Global lagrogetion
(b1) Local Fitting
Local
Local
CIF. ALU
(b2) Local Gerennbattio
Hoput
NNN
N
Head- degree Branch
000
dhebeing
Tall- diggrne Branch
000
000
880
CIF
609
Figure 2: Overview of FedLoG with Clients and 3 Classes.
4.1 LOCAL FITTING
The local model for each client & consists of one GNN embedder (4%)) and two classifiers (0k, and (x.r), for the head and tail degree branches, respectively, as shown in Figure 2b3). Each branch forms a prototypical network (Snell et al. (217))bbaedd architecture for prediction. While prototypical network is originally designed for few-shot learning, we repurpose it by designating learnable nodes as prototypes, enabling both prediction and the condensation of class information into these prototypes. Specifically, each branch contains |Cy| X learnable nodes, each with features Pr of dimension d, allocating 8 nodes per class. Thus, each client k has learnable node sets Va, and Vicual with features V. E [2(Cy]Xx xxs)xx and Xy, E IR Cvl xx)x))xdd_ respectively. At the client level, we condense knowledge from locally observed nodes into these learnable nodes. Head degree nodes are condensed into Vichead and tail degree nodes into Va,uil, integrating condensation and prediction into a single process. The detailed processes are described as follows (See Figure bbl) and (b3)):
(Initialization) In the initial round (r 0), we initialize the local model weights for each client k. denoted de {pp,, H, rx,r), with the global set of parameters gelobal [spelobal, gelobal . oelobal).
(Embedding) Given client k's local graph Gj (Ve, Ex) where the feature of each node us E Vk is initialized with ( Kny, a shared GraphS (Hamilton et al. 2017 GNN encoder you: is employed to embed each local node uk E V; and learnable nodes Ukc Vkeh and Ukktail E Vi,iall
Heg GNNN,, (Ve,Gk), Hes GNN (Vk.head: Gi.head), huv..u GNN (FK,till ua. (I)
where Nuy. I Meg.eww and hor.au are the representations of nodes Uk. Uk.head- and Uk,till respectively. Gk. = (Vk.head: Eg) and Gr.till = (Va.ait. Eg) denote synthetic graphs constructed by learnable nodes from head and tail degree branches, respectively. It worth noting that the learnable nodes do not adhere to a specific graph structure but share the same GNN encoder with the local graph, allowing us condense structural information into the features of the learnable nodes.
After acquiring node representations, we generate model predictions in each branch using class prototypes, which are the representations of learnable nodes. Since the process of generating model predictions is identical in both branches, we only explain the head branch here. In the head branch, prototypes are defined as Pk,head = (h (1.1) ..... h (1.2) h (eell.))
h (eell.))) }. with proto- types per class. To ensure all class information contributes to the final prediction, the target node representations are further updated based on feature differences with all prototypes assigned to each class as follows: h',, = O2.51 (ho,, (1hn - h (, ).... (hm, - h (cv)--)))) Please refer to
Appendix A.4 for more details on OR. Then, the class probability for target node Uk is given as follows:
hvg ))
p(clk..) = exp(- Myye beal)
(2)
Ed=
where d(.,.) is the squared Fuclidean distance and hy; .ad indicates the average of prototype represen- tations of class c, i.c.. hyg, = E =
(c..) By updating the target node's 's representation based on its relationships with all prototypes and minimizing the distance to its correct class prototype, we aim to guide the synthetic learnable node to effectively learn a representation that reflects both the correct class and the broader relationships among all classes.
To obtain final prediction p, we combine the class probabilities from both branches P Phend and Prail by weighting them based on the degree value the target node U/ (i.c., deg( (vx)) as follows:
p a Phead + (1 - a) Prall:
(3)
5

---

Published as a conference paper at ICLR 2025
where a = 1/(11+ + (dd((())) and is the tail degree threshold outlined in the Appendix A.9 Note that the hyperparameter a balances the influence between the head and tail branches based on the node's degree. Specifically, a acts as weight for the prediction loss contributed by each branch for the target node. For instance, when target node has a high degree, a becomes large, increasing the influence of the head degree branch on the final predictions. Consequently, knowledge from high-degree nodes is condensed into V3, head, the model significantly updates the learnable features X ae of Vead to minimize prediction loss. Conversely, when a node has a low degree, its knowledge is condensed into Xvy,, of Vk.tail instead. Additionally, the sigmoid- based formulation of a prevents nodes with extremely high degrees from dominating the learnable features, ensuring a balanced contribution across all nodes.
Training objective for each client k. The prediction loss for each client k calculated as Ck,els = EehEv, Eoec, eV = c) log(p(e)). Furthermore, to ensure the stability of the con- densation process, we minimize the L2 norm of the learnable features, denoted as Lk.norm = EnEVi..eal...... |axul12- Thus, the total loss for model parameters is Lx (ok; Xye X ( LRRCOs + B Lo,norm: where |3 adjusts the extent of regularization. It is important to note that besides updating the local model weights Φb- = (fp,, H,, 0g.r) for each client k, the learnable features X Verpooll and X Vv,, are also updated.
4.2 GLOBAL AGGREGATION AND GLOBAL SYNTHETIC DATA GENERATION
Global Aggregation. As shown in Figure 2 a 1), after training the K local clients, the server aggre- gates the local model weights for round r using the weighted average datlobal R Ex_ 2
Global Synthetic Data Generation. In addition, as shown in Figure aa),, the server generates global synthetic data Delohal which will be employed during the Local Generalization phase (Section 4.3) to help mitigate the issue of local overfitting. More specifically, we first generate node features in the global synthetic data Delobal by merging he, from all K clients. For example, when merging Ve.d for all K clients regarding class c, the server gives more weight the synthetic data from client for whom class e belongs to the head classes within the local data Dz. This is to take into account the expert knowledge of each client regarding the dominant classes within its local data, which is supported by our empirical analysis Section 3.T More formally, for each class e E Cy the feature vector of the i-th global synthetic node for class C, i.e., (...) E Rdd is generated as follows:
K
x (e.1)
rix (c.0)
(4)
",
where x (e.1) E Xve and r baat
V M
represents the proportion of nodes belonging to class c
in the k-th client's dataset. By giving more weight to clients with expertise in each head class, the server effectively combines the most reliable knowledge from all clients to create the global synthetic nodes. This process results in |Cyl x 8 global synthetic nodes Velobal with features X youu = fxx Us "K,g,,)....,. (g ................................ x vi (icp,.)......................... .................... ((ppl.))) J. The final global synthetic data is represented as Delobal = (gelobal yelobal), where Gelobal = (Yelobal )) with node features X y At the end of each round r, the server distributes the aggregated model weights gilobal and the generated global synthetic data Dglobbll In summary, considering both the degree headness through the learnable
features Xv vost and the class headness through rf allows the global features X Kynne to contain knowledge about both head degree and head class across all clients.
4.2.1 DISCUSSIONS ON THE GRAPH STRUCTURES OF GLOBAL SYNTHETIC NODES
Note that even though each global synthetic node contains only features without explicit graph structures, these features still implicitly capture the original graph structure. This is because these features are learned using a graph embedder you that is shared across both synthetic and original nodes. As result, the structural information of the original graph is condensed into the features of the synthetic nodes.
4.3 LOCAL GENERALIZATION
At the beginning of each round (r 2 1), each client & initializes its local model de with the distributed global model parameters dglobal followed by the local update of de based on its local data Dg (i.e.,
6

---

Published as a conference paper at ICLR 2025
Local Fitting described in Section 4.1 Then, as shown in Figure 2 b2) and (b3), we additionally train the local model with the global synthetic data Delobal, enabling the model to generalize to locally absent knowledge (i.e., Local Generalization), such as tail and missing classes. Since each client has different locally absent knowledge, we first adaptively customize the global synthetic data for the current state of the local model through two strategies, i.c., 1) feature scaling and 2) prompt generation, and then train the local model with the customized data.
Strategy 1) Feature Scaling, When the local model is strongly biased toward the local distribution, tends to assign high logits to the dominant class, making it difficult to predict tail missing classes. To address this issue, we apply strong perturbation to the training data of the dominant class to help balance predictions across all classes, allowing the model to effectively learn absent knowledge within the local data. To achieve this, we use feature scaling for the perturbation on the global synthetic data
X Valiad = {xr (1.00..)... F. ((..))...... F ((cppl)))) ...)) by ((lplaa) } as follows:
8 (. = ggg + [eld (Fypen
x fea)), where Fypand
(5)
"g
Vglobal
Nug E.X yotenal
where Yt E R2Cvl is the class wise adaptive factor that adjusts the strength of the perturbation by moving the global synthetic data of class c to the average of global synthetic data, making it harder to predict. Note that when [/[[]] = 1, the global synthetic data for class e is completely replaced with the average of the global synthetic data, whereas the global synthetic data for class remains unchanged when Yrs 0.001 whenever the local model's s accuracy for class e exceeds the threshold at the end of the round, thereby increasing the perturbation of the corresponding class.
O. During training, we dynamically modify the factor by incrementing by
Strategy 2) Prompt Generation. While we have focused on condensing knowledge into the global synthetic data, we have not yet addressed how to train our GNN encoder on this data. Recall that the GNN encoder (P is shared across both the synthetic and original nodes. However, since global synthetic nodes only contain features without an explicit graph structure while the original nodes involve a graph structure, the parameters of the shared GNN encoder would be differently affected by them, leading to discrepancies in the gradient matrices.
Our main idea to solve this issue is to ensure that synthetic nodes are trained in the same environment as original nodes within the local graph. Specifically, the objective of the prompt generator is to make the synthetic the target node's 's features ppodduce the same gradient matrix as when the target node is predicted using its true h--hop subgraph within the local graph. Details on pretraining the prompt generators are in Appendix [A.7 More formally, we generate a prompt node for each feature-scaled global synthetic node using the class-specifie prompt generator PG corresponding to class e (Figure ((b)) as follows: x =
sasph-hhichh consists of a target node and its corresponding prompt node derived from
(fog), where xe is the feature of the prompt node vP, corresponding to the node Ug- We then construct a synthetic graph 9 Uo, eypia 9 for client k. where each Phin, consists of global synthetic node vg and its prompt node v..-- connected each other.
Training with Customized Data. Using the synthetic graph 9.y, the prediction for the target synthetic nodes within Velobal follows Eqs. 3 using the same local model Φv as for normal nodes Vs. with a set to 0.5. The prediction loss for Dinonal is Lt.g = =E,, ",,,vven EvaNn Enecr -I(Ye, c) log(p(c)). At the end of each round, we adjust the adaptive factor "x[c/v based on the class prediction accuracy. In summary, the final loss of the local model is: Lt Lkketo + Lkg + 3 Lk.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets. We conduct experiments on five real-world graph datasets. Distributed subgraphs are con- structed by dividing each dataset into a certain number of clients using the METIS graph partitioning
algorithm (Karypis& & Kumar 1997). The datasets used are Cora (Sen et al. 2008 CiteSeer (Sen cet al., 2008), PubMed (Sen et al. 2008), Amazon Computer (McAuley et al. 2013). and Amazon
Photo (Shchur et al., 2018). For more details, see Appendix [..12
Baseline Methods. 1) Local: Refers to local training without any weight sharing. 2) FedAvg
(McMahan et al.] 2017): The most widely-used FL baseline. 3) FedSAGE+ (Zhang et al.) 20211). 4)
FedGCN Yao et 2024) and 5) FedPUB (Baek et al. 2023): subgraph-FL. baselines that primarily
7

---

Published as a conference paper at ICLR 2025
Table Model performance on Seen Graph settings. Mean accuracy with std. over runs.
Cora
Cludser
PrnMeed
Amazon Phato
Amaaze Computers
Methoess
SCiii
(0 Ciarr
CSeie Ciors
ClCett
[10C
1.3
|10Clichh
Clech
Local
0.008
00:00
0.4m
64001
Fodtig
03616
08.04
0.7162
1988
0988
0..00
19846
0.20
0.844
0.804
0.000
0.001
Seen Grape
FedS-GGF+
0,500
0,4156
0,5025
0.505
0.5177
0,005
0.898
0.9091
00041
0.939
0.607
0/335
6.800
5/631
0,6556
0,824
0,725
6726
0,7155
07023
0.001
03841
0.738
0.5117
0.503
0.533
9/317
FedtPin
0.876
03662
630M
0.700
0.7935
D1901
0.90
09133
0.339
08333
FoINTD
0,8632
0.553
0,0064
6,5455
0.3336
0.7146
D0349
0.5065
0.961
09376
0.0000
00119
0.402
0860
RL8365
FedEDD
0,850
07911
9.000
03631
FediLais
address missing knowledge within the current local label distribution. To ensure a fair comparison, we also evaluate our method against 6) FedNTD (Lee et al., 2022) and 7) FedED (Guo et al. 2024). which address local overfitting in FL.. For more details, see Appendix |3.13
Evaluation Protocol. We perform FL for 100 rounds. Node classification accuracy is measured on the client side and averaged across all clients over three runs. More details are in Appendix A.14
5.2 EXPERIMENT RESULTS
Q1. How does FedLoG perform in conventional FL settings? Table ! presents the evaluation of models graphs that were used for training. The label distributions of the test nodes match the training label distribution of each client. We refer to this conventional setting as Seen Graph, where models are evaluated on test nodes within the same graph structure as the training nodes (i.e., transductive setting (Kipf & Welling 2016)). The overall performance of FedLoG on the 'Seen Graph' outperforms that of other basclines, demonstrating its strong performance in conventional settings.
Q2. Does FedLoG generalize to unseen data after local updates? In this section, we introduce practical and novel test settings for subgraph-FI. aiming to assess the model's ability to generalize to potential unseen data in real-world scenarios. By proposing these new settings, we emphasize the importance of evaluating models beyond conventional FL settings, and demonstrate that our method consistently achieves superior performance compared to other baselines. We introduce three practical scenarios for unseen data:
1) Unseen Node (Table (l)). Each client has new nodes with seen classes added to its local graph, which introduce structural changes. We perform evaluations on the new nodes to assess how well the FL framework adapts to these structural changes. 2) Missing Class (Table (Table Each client has new nodes with missing classes added to its local graph. We evaluate the performance on new nodes representing missing classes for each client, assessing how effectively the FL framework enables the local model to learn previously absent knowledge. 3) New Client ( (Table [[cc)). A new client that has never participated in the FL framework emerges. This client has distinct label distribution and graph structure. We assess how well the FL framework generalizes to accommodate this new client, ensuring robust performance across diverse scenarios. To do so, we perform evaluations on the unseen graph of the new client using each trained local model without the new client being involved in the training, and then report the mean accuracy over all clients. These approaches help simulate real-world scenarios where clients have incomplete information. Detailed settings are described in Appendix 10 and 16
observe that FedLoG outperforms baselines by preventing local overfitting and effectively addressing unseen data. Specifically, in the Unseen Node and Missing Class settings, FedLoG shows superior performance on the added nodes, even with missing classes. For the Missing Class, which requires extensive knowledge from other clients, Local and personalized FL models like FedPUB (Back et al., (023) fail to predict the missing classes as they optimize for the training label distribution. Although FedSAGE+ (Zhang al. 202 and FedGCN (Yao et al. 2024) attempt to compensate for missing neighbors, they are not al always effective because the missing class is not always within the neighbors. Moreover, FedNTD (Lee et al.) 2022) and FedED (Guo et al.) 2024) address local overfitting and achieve relatively high performance in missing class prediction. However, they regularize the local model logits to match the global model, risking noisy information from tail data and resulting in inconsistent performance across different settings. In contrast, FedLoG alleviates local overfitting by using reliable class representations and structural information across clients, reducing the emphasis on noisy information. Thus, FedLoG successfully addresses unseen data, ensuring robust performance even with unseen graph structures (New Client) due to its generalization ability across all classes and structural features.
In Table
8

---

Published as a conference paper at ICLR 2025
Table 2: Model performance in FL settings on unseen data. Mean accuracy with std. over runs.
Cara
Cheswer
PrbMed
Phate
Amannn Computen
Metlools
10CTest:
1l3
SClleth:
Ci
SCliehh
Ch
Taccll
0,297
5,371
0,002
0,190
Unseen Node
Podltgg
03099
0.10
0.5832
8.00
0,819
038070
03782
200m
02727
900
0,300
0,4177
VedSAGE-
03653
0.265
0.836
0,572
0,923
04154
0.834
0,802
3,925
0478
0,2966
0,907
03352
0,5555
5,500
0.353
0397
035015
03233
0,30
00135
08354
0,002
33833
03353
0,500
63355
0,306
0,400
0.593
04353
00784
6,00
04016
0,8822
0.304
0,073
6414
0.905
0,555
FeiNTD
0.588
0.31
8 33557
6.014
04151
0.8839
0.8857
038815
0,9933
0.5833
6.576
0.9056
0,005
FedED
01236
02226
0.546
C0148
0
00258
0,8834
0,629
06222
0.4355
04552
0.540
Fedloo
6.734
6/403
0,700
667634
0,5055
0/9664
0,9956
BJRMSS
6/7667
0,7077
0.777
6:677
0.116
0,6735
Cora
CHicker
PulMed
Amsaass Photo
Amsasan Compaten
Mithood
Cieees
1.5
Charr
10 Clests
Clcas
Clasto
Clions
Clastaa
S.Clcbb
1C Clicss
5 SCicab
10 Charo
0.000)
bu000o
0,5000
115000
0.030
0,0000
ML000
D45303
100300
0.8893
003335
0.404
(
(b Mosiing Ches
Fedlog
01319
0.0822
113003
0,914
0,3199
R.000
0,0113
R00333
100300
0.8331
CURISS
0.1841
0
Fodt-MGl-
(1500)
0.0087
01633
0.150
B6000
010115
D/031
R1000
038331
01899
076001
01009
FellOC
0
Fedrem
FoINTD
00313
FodED
Fedl.GG
66672
0,4848
0,4837
06643
0,9923
0,585
0,770
0,229
0,4766
0,5505
0,132
0,358
0.3175
0,043
Cara
CHeleer
PaaMMed
Amsznn Photo
Amaaii Compater
Methods
1Clees
5
10CTeen::
Villeess
C
Cllcnts
Cllenn
Cliens
3
SCllean
Lacil
(c) New Client
Fedlog
9,213
0.2234
6,2899
8.00
0,504
0,5125
0.1061
0,409
0.355
04124
6000
0.1088
03884
0,5016
0.575
007883
0.099
0.672
0.718
0.7%%
VealGNN
03840
0,300
0,435
03552
0,458
0.144
03333
0,883
3,6893
03599
0.009
0,603
02066
0.8541
113000
0.3939
6.5315
0.3330
B4433
0.003
0,533
0.755
0.534
0.076
0,5635
0L3304
0.433
6,506
0.3357
B44133
04633
2.0055
0.1529
0,000
60014
0.7146
0,7679
FellDD
6000
04597
0.334
0,330
6,403
013366
0.4422
0,6822
9,999
0,545
0.15
0
0,162
0.7147
Peutlaui
IU984T
8:440
0,507
0,799
8,7646
0.9083
0,5554
6:7886
09964
Scon Graph
[b] Unseon Node
(c) Missing Class
New Client
HT
0.60
Tr
Round
Round
Round
Round
Figure 3: Impact of headness of class/degree for various scenarios (Amazon Clothing 3 Clients).
Q3: Do the headness of degree and class really help other clients? We evaluate the importance of the headness of degree/class under various scenarios, both of which are expected to enhance data reliability. As FedLoG additionally trains the clients using global synthetic data, we measure the impact by varying the knowledge condensed into global synthetic data. Specifically, we compare four different test settings for constructing global synthetic data, i.e., using 1) head class & head degree nodes (HH). 2) head class& & tail degree nodes (HT). 3) tail class & head degree nodes (TH), and 4) tail class & tail degree nodes (TT). Detailed descriptions are provided in Appendix(A..
Figure 3 shows test accuracy curves illustrating the impact of each test setting on performance and stability. Data reliability varies with global synthetic data knowledge; HH knowledge is the most reliable. Class headness significantly affects reliability, evident in the performance gap between head and tail classes. Degree headness impacts stability; tail degree settings show more fluctuations. Thus, using HH knowledge is crucial for maintaining reliability and stable outcomes.
Q4. Does each module effectively address the local overfitting problem? Figure shows the results of ablation studies under the Missing Class setting: w/o LG denotes excluding the Local General- ization (LG) phase. w/o PG denotes without Prompt Generation, and
Accuracy 0.5
0.4
w/o
w/o
w/o ES denotes without Feature Scaling. We validate that the Local Generalization phase is crucial for addressing the absent knowl- edge, while other modules (i.e., PG and FS) impact performance and stability. Refer to Appendix A. details.
0.0
20
60
60
350
Round
Figure 4: Ablation Studies
(CiteSeer- - Clients).
Further experiments, including an assessment of Feature Scaling's adaptive effects, hyperparameter analysis, and evaluation on unseen data in an open set, are provided in Appendix A. Communica-
tion overhead and time complexity analysis are described Appendix A.3 and A.5 respectively.
9

---

Published as a conference paper at ICLR 2025
6 PRIVACY ANALYSIS
Q1. Does utilizing the class distribution of the clients pose privacy problem? The class distribution does not include individual data but merely represents the proportion of each class, which is far less sensitive than the raw data. As the class distribution is general statistical information that indicates the trends within group rather than specific data about individual users, we argue that it is very difficult for an attacker to infer specific data of individual nodes from the class distribution.
However, in case privacy concerns remain, we can add noise to the class rate to make it difficult to determine the exact class distribution. We experimented with two methods of adding noise to the class rate: 1) adding class-wise Gaussian noise with ja as O and a as a x rg. where a is chosen from [0.01, 0.1, 0.5] and r M
| and 2) performing random permutation of the elements in the class
rate vector. To maintain the trend while applying random permutation, we permuted only the elements within the head classes and within the tail classes.
In Table both Gaussian Noise (GN) and Random Permutation (RP) methods, which result in class rates roughly similar to the original, showed no significant difference in performance, except for the GN (a- 0.5) setting that highly deteriorates the trend of the class rate. This indicates that Fed- LoG does not require an exact class distribution as long as the general trend is maintained, allowing us to protect privacy more rigorously.
Table 3: Class rate with noises (Cora 3 clients).
Noo Noise [GN In 0.00)| (s 1 (,1)
GN to 0153]
RP
Seeu Graph
0
0,8530
0.8542
008621
Unseen Node
0,7341
0,7127
0.7217
0,7057
0.7351
Missing Class
0.6472
0.6344
0.6013
0.6277
0.6308
900m
8.3m
2410
New Cliient
0,5109
60300)
Q2. Can synthetic data be specified to match the original data's features? Since the syn- thetic data is generated by condensing the original nodes within each client's s graph, there may be a potential privacy risk. One way to assess this risk is to compare the feature distributions of the orig- inal nodes (i.e., Va) and the synthetic nodes (i.e.,
Class 0
Dans
Class
Figure 5: 2D PCA visualization of feature distri- butions for the same class in the CiteSeer dataset.
Vishead and Va.. If these distributions overlap, it suggests that the original features can be recon- structed from the synthetic data, which entails privacy risk. In Figure 5 we present 2-dimensional PCA visualization of both the original feature matrix (blue) and the synthetic feature matrix (orange) for the same class in the CiteSeer dataset. The clear difference between the two distributions shows that sharing synthetic nodes poses minimal privacy risk.
Moreover, privacy risk further reduced because the synthetic data represents an aggregation of all training nodes, without being tied to any specific node (i.e., condensation). This aggregation also incorporates structural information into features, resulting in a distribution that diverges from the original feature space. Since the synthetic nodes lack an explicit graph structure, their feature space distills structural information differently from the original nodes, leading distinct distribution.
Q3. How does FedLoG provide protection against gradient inversion attacks? FedLoG en- hances protection against gradient inversion attacks (Zhu et al. 2019),, where adversaries attempt to reconstruct the original data from gradients uploaded to the server. This protection is primarily due to each client being trained not only on its local data but also on global synthetic data. The inclusion of this synthetic data introduces noise into the gradients, making it difficult for adversaries to extract information solely from the original data. Additionally, each client applies different levels of feature scaling to the global synthetic data (Section 4.3), and these scaling factors are never shared. This variation further obscures the gradients derived from the synthetic data, making it even harder for adversaries to accurately invert the gradients and reconstruct the original data.
7 CONCLUSION
In this study, we address the challenges of local overfitting and unseen data (i.e., unseen node, missing class, and new client) in subgraph-FL with our proposed method, FedLoG. Our model generates global synthetic data by condensing reliable information from each class representation and its structural information across clients, enabling adaptive generalization of absent knowledge within local datasets without directly using data from other clients. This approach enhances the generalization capabilities of local models, allowing them handle unseen data effectively while also mitigating privacy concerns. Our experimental results demonstrate that Fedl .oG outperforms existing baselines, proving its efficacy in novel practical scenarios for generalizing unseen data.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (ITTP) and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) RKS-2022-11220155 and KSS-022-004466855) and National Research Foundation of Korea (NRF) funded by Ministry of Science and ICT (FE-20022M33JAA1000221)
REFERENCES
Jinheon Back, Wonyong Jeong, Jiongdao Jin, Jachong Yoon, and Sung Ju Hwang. Personalized subgraph federated learning. In International Conference Machine Learning, pp. 1396-1415. PMLR, 2023.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00772 2021.
Huancheng Chen, Haris Vikalo, al. The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation. arXiv preprint arXiv:2301.0800 2023.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IELE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
lan Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Kuangpu Guo, Yuhe Ding, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Not minorities are equal: clmpt--lass--wwr distillation for heterogeneous federated learning. arXiv preprint arXiv:2401.02339 2024.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
William Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
Chaoyang He, Keshav Balasubramanian, Emir Ceyani, Carl Yang. Han Xie, Lichao Sun, Lifang He, Liangwei Yang, Philip S Yu, Yu Rong, et al. Fedgraphnn: A federated learning system and benchmark for graph neural networks. arXiv preprint arXiv:2104.07. 2021.
Haibo He and Yang Bai. Adasyn: Adaptive synthetic sampling approach for imbalanced learn- ing. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 1322 1328, 2008
Haibo He and Yunqian Ma. Imbalanced learning: foundations, algorithms, and applications. 2013.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. arXiv preprint 10.0:: 2021.
Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li. Danqing Zhang, Jiliang Tang, and Bing Yin. Condensing graphs via one-step gradient matching. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 720-730, 2022.
George Karypis and Vipin Kumar. Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. 1997.
Seong-Woe Woong Kim and Dong-Wor Wan Choi. Stable federated learning with dataset condensation. Comput. Sei. Eng., 16(1):52-66) 62, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 442.69980 2014.
11

---

Published as a conference paper at ICLR 2025
Thomas N Kipf and Max Welling. Semi- supervised classification with graph convolutional networks. arXiv preprint (et9::6600000000 2016.
Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bac, and Se- Se-Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. Advances in Neural Information Processing Systems, 35:38461-38474, 2022.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pp- 965-978 IEEE, 2022.
Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 995-1005, 2021.
Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In International Conference on Machine Learning, pp. 19767-19788. PMLR, 2023a.
Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5319-53299 2023b.
Ping Liu, Xin Yu, and Joey Tianyi Zhou. Meta knowledge condensation for federated learning. arXiv preprint arXiv:2209. 14851, 2022.
Rui Liu, Pengwei Xing. Zichao Deng, Anran Li, Cuntai Guan, and Han Yu. Federated graph neural networks: Overview, techniques, and challenges. IEEE Transactions Neural Networks and Learning Systems, 2024.
Zemin Liu, Trung-Kies Kien Nguyen, and Yuan Fang. Tail-gnn: Tail-node graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 1109-1119, 2021.
Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang. Xu Liu, and Lingling Li. Curvature-balancee feature manifold learning for long-tailed classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15824-1588 15835, 2023.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-basee recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-effieee efficcient learning of deep networks from decentralized data. In Artificial intelli- gence and statisties, pp. 1273-1288 PMLR, 2017.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint 07X1v:2007.07314, 2020.
Alexandru Niculescu-Mizzi and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625-632, 2005.
Seungeun Oh, Jihong Park, Eunjeong Jeong, Hyesung Kim, Mehdi Bennis, and Seong-Lyu Kim. Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup. IEEE Communications Letlers, 24(10):2211-23) 1211,, 2020.
Joonhyung Park, Jaeyun Song, and Eunho Yang. Graphens: Neighbor-aware ego network synthesis for class-imbalanced node classification. In International conference on learning representations, 2021.
Mohammad Rasouli, Tao Sun, and Ram Rajagopal. Fedgan: Federated generative adversarial networks for distributed data. arXiv preprint arXiv:2006.0222 2020.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad -ad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
12

---

Published as a conference paper at ICLR 2025
Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprini arXiv: 1.05668, 2018
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning, arXiv preprint arXiv:2006.05 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Arjun Subramonian, Jian Kang, and Yizhou Sun. Theoretical and empirical insights into the origins of degree bias graph neural networks. arXiv preprimt orXiv:2404.03139, 2024.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11662-1 11671, 2020.
Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learning on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9953-9961, 2023.
Yuan Wang, Huazhu Fu, Renuga Kanagavelu, Qingsong Wei. Yong Liu, and Rick Siow Mong Goh. An aggregation-free federated learning for tackling data heterogeneity. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 26233-26244 2024.
Chuhan Wu, Fangzhao Wu, Yang Cao, Yongfeng Huang, and Xing Xie. Fedgnn: Federated graph neural network for privacy-preserving recommendation. arXiv preprint arXiv:2 04925, 2021 la.
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie, Self- supervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 726-735,75 2021b.
Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 1894-1903, 2021.
Yuhang Yao, Weizhao Jin, Srivatsan Ravi, and Carlee Joe- Loe-Wong. Fedgen: Convergence- communication tradeoffs in federated training of graph convolutional networks. Advances in Neural Information Processing Systems, 36, 2024.
Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang. Feddisco: Federated learning with discrepaney-aware collaboration. In International Conference on Machine Learning, pp. 39879-39902. PMLR, 2023.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprint (xXiv:207700022. 2021.
Xianyao You, Caiyun Liu, Jun Li, Yan Sun, and Ximeng Liu. Fedmdo: Privacy- preneerieg federated learning via mixup differential objective. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Sukwon Yun, Kibum Kim, Kanghoon Yoon, and Chanyoung Park. Lte4g: Long-tail experts for graph neural networks. Proceedings of the 31 ACM International Conference on Information & Knowledge Management, pp. 2434-2443, 2022.
Bianca Zadrozny and Charles Elkan. Calibrating probability with undersampling for unbalanced data. In Proceedings of the First SIAM International Conference on Data Mining, 2001.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, pp. 263 1-2222., PMLR, 2022.
Ke Zhang, Carl Yang. Xiaoxiao Li. Lichao Sun, and Siu Ming Yiu. Subgraph federated learning with missing neighbor generation. Advances in Neural Information Processing Systems, 34:6671-6682, 2021a.
13

---

Published as a conference paper at ICLR 2025
Longling Zhang, Bochen Shen, Ahmed Barnawi, Shan Xi, Neeraj Kumar, and Yi Wu. Feddpgan: federated differentially private generative adversarial networks framework for the detection of covid-19 pneumonia. Information Systems Fronliers, 23(6):1403-144 2021b..
Tianxiang Zhao, Xiang Zhang, and Suhang Wang. Graphsmote: Imbalanced node classification on graphs with graph neural networks. Proceedings of the 14th ACM international conference on web search and data mining, pp. 833-841, 2021.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2025
Supplementary Material for Subgraph Federated Learning for Local Generalization
Appendix
16
A.1
Algorithm
16
A.2
Notations
17
Communication Overhead
18
A.4
Detailed Process of the Classifiers 02, h and ON.T
18
A.5
Time Complexities
19
A.6
Related Works
19
A.6.1
Improving Generalization in Federated Learning
19
A.6.2
Synthetic-based Federated Learning
20
A.7
Detailed Process of Pretraining the Prompt Generator
A.8
Detailed Process of Evaluating Data Reliability
22
................................
A.9
Criteria for Threshold Degree Value for Tail- Hel--eeree Nodes
................................
23
A.10
Detailed Process of Evaluating Unseen Data
23
A.10.1 Closed Set
23
................................
A.10.2 Open Set
24
A. Additional Experiments
24
A.11.1 Impact of the Hyperparameters)
24
A. I.2 Assessing the Adaptive Impact of Feature Scaling on Local Clients
26
1..3 Experimental Results on the Open Set
26
A.II.4 Ablation Study
26
A.12 Datasets
27
A.12.1 Dataset Statisties
28
A.13 Baselines
28
A.14 Implementation Details
29
A.15 Detailed Process of Generating HH/HH//TH/1 Global Synthetic Data
29
A. 16 Experimental Dataset Statistics)
30
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 ALGORITHM
Algorithm 1 FedLoG: The Overall Algorithm
Server S
1: if round == 0 then
2:
Initialize global model @
3: else
Aggregate local models: @ftlobal * EE &2
4:
5:
Generate global synthetic data Delohal
6:
Combine Vk based on class proportions
Equution
7:
end if
Local Client
8:
if round == ( then
9:
Initialize local model: de gialobal
10:
Local Fitting: Update Φx, Vk.head, Vesail using D3
else
12:
Update local model: Φu +- obelohal
13:
Local Fitting: Update Øss V&. Vi.lead, Visuil using D3
14:
Local Generalization:
15:
1. Download Delobal PG from server
Feature Scaling: Adapt Delohal features locally
Equation
17:
3. Prompt Generation: Generate prompt nodes of Delobal via PG
Update @k. Vk,head Vk,tail using adapted Dalobal
19:
end if
Pretrain Prompt Generators (PGs)
20: Local Client k: Pretrain PG₁ using Dj
21: Server Aggregate PG + E.rg Σn PPG for each class
P Weighted by rf
1 [.
16

---

Published as a conference paper at ICLR 2025
A.2 NOTATIONS
Table Summary of the notations. For simplicity, describe the notation based on head-branch.
Notation
Description
General Notations
S
Server
K
Number of clients
G
Graph
V
Set of nodes
E
Set of edges
D
Dataset consisting of ② and Y Label set for the nodes
Y
Xy
Feature matrix of set of nodes V
Feature vector of node E V Set of parameters of a global model
Cyl
Number of classes within set of nodes y Current round
h.
Represeniation of node
Weight of prediction between head and tail branches
Local Client Notations
Local graph for client &
Vi
Set nodes within local graph for client
E
Set of edges within local graph for client k
D
Local dataset for client
Diocal
Combined local datasets, Diocal = U Dx
Y
Label set for the nodes within local graph for client & Set of parameters of a local model for client &
db
Parameters of a GNN embedder for client Parameters of a head- branch classifier for client kt Parameters of tail aill-brnch classifier for client Ie
OuT
Vi.
Set of synthetic nodes within a head- head-degree branch of client &
U bend
(c.a)
s-th synthetic node for class e in head-branch for client k
X Vaa
Feature matrix of set of nodes head
(c.s)
Feature of the s-th synthetic node for class in a head-branch for client
head
h t heud
(c.4)
Representation of the s-th synthetic node for class in head-branch for client &
Pk head
Set prototype representations (i.e., representations synthetic nodes) head- branch for client l
hy
Average of prototype representations class e bead-branch for client Proportion of nodes laheled in client A's dataset Dj Pretrained prompt generator for client & (regardless of specific class) Class-wise adaptive factor for client & Synthetic graph consisting of graphs, each containing the global synthetic nodes (which adapted locally) neighboring with their generated prompt nodes.
PG
Yyk
rk
Generated prompt node of node th
Xug
Generated feature generated prompt node of for the input feature x.,, of node eg
Average of features of the Athop neighbors of node vs within Sr.
Global Synthetic Node Notations
Delobal
Global synthetic dataset
Goloban
Global synthetic graph
Set of global synthetic nodes
glabaal
Label
the global synthetic nodes
X Vamma
Feature matrix of the global synthetic nodes yskhuul
X (eua)
Feature the s-th glohal synthetic node Ny for class
PG
Class spes-sc prompt generator class e (by aggregating NG,
class wise manner)
Hyperparameter Notations
8
Hyperparameter for assigning the number of synthetic nodes per class Hyperparameter for adjusting tail degree threshold
A
Hyperparameter for adjusting the extent of regularization of the features of synthetic nodes
17

---

Published as a conference paper at ICLR 2025
A.3 COMMUNICATION OVERHEAD
In this section, we provide comparisons of communication overhead across different baselines. FedLoG uploads and downloads both the synthetic data and the model parameters at the end of each round. For the Cora dataset with a setting parameters to share with the server, resulting in 4 bytesx 1..88..96=4.33 MB (excluding the prompt generator, which only trained once the first round). Additionally, the synthetic data has 182,000 parameters (s x [Cyl x d, where [Cyl denotes the number of classes and d denotes the dimension of the features), amounting 0.72 MB. In summary, our model requires 10.08 MB (4.32 0. 0.72)) for upload and download each round. Below are comparisons of communication overhead between models over 100 rounds:
clients and & = 20, our model has 1,081,926
FedAvg
FedSAGE+
FedGCN
FedPUB
FedNTD
FedED
FedLoG
Cost (MB)
393.11
1543.58
393.11
786.03
393.11
393.11
1011.14
Table 5: Comparison of Communication Cost (MB) Across Different Models
Although FedLoG relatively requires higher communication overhead compared other baselines. shows faster convergence due to its utilization of reliable class representation, leading to a stable training process. Below are comparisons of communication overhead until each model reaches the same accuracy (i.e., 0.8 on the Cora dataset with clients).
Model
FedAvg
FedSAGE+
FedGCN
FedPUB
FedNTD
FedED
FedLoG
Rounds to Reach 0.8
58
100 (Fails to reach)
57
29
19
39
10
Cost (MB)
228.00
1543.58
224.07
227.95
72.79
149.41
101.11
Table 6: Rounds to reach 0.8 accuracy and corresponding communication cost in MB across different models.
Despite FedLoG's s higher communication overhead per round, its faster convergence results in lower overall communication overhead to achieve the same accuracy compared to other baselines. This demonstrates the efficiency and stability of FedLoG's training process, making it an effective approach despite the initially higher communication cost per round.
A.4 DETAILED PROCESS OF CLASSIFIERS 0s,H AND OR,T
In this section, we provide the detailed process of the classifiers 0g and Og, Specifically, we describe Eq. 2
{ (M....
W', = 0x, (over
-h (1.13
h ((.ppple))
which corresponds to the head head-degree classifier in the local model. Since both the head-degree and tail-degree classifiers have the same architecture, we focus on describing the head-degree classifier.
The primary objective of the classifier is to ensure that all prototypes in Px.bead contribute to the final prediction of the target node, allowing the prediction loss to be influenced by all prototypes. To achieve this, we first generate a message m,y from each prototype node vy E P to the target node based on the distance dæy between them. This is computed using MI.P msg as follows:
m,j
MLP mag (hh
n || dRy]),
(6)
where the distance dæj calculated as:
dæj |r 112 Arks Ihv |l [Nval (h, hoy
and denotes concatenation. Here, noy IN Enoercy hr. is the average embedding of the 1- 1-hop neighbors of the target node U/ within the graph Gz.
The neighbor information is used because the embeddings of neighbors from head-degree and tail- degree nodes differ, enabling each branch to leverage degree-specific knowledge. The target node
18

---

Published as a conference paper at ICLR 2025
embedding un is then updated by applying a learned transformation to the representation differences Arkj, followed by aggregation:
[aj = 1h Hnyl
MLP trans (m&z)
(8)
1
(ki
(9)
PPr.henal 3
ck = hive 1/3
(10)
where MLPPirass transforms the message maj into scalar value.
In summary, the classifiers Os. and 0g.7 update the embedding of the target node h3 by reflecting the interactions between all different prototypes in Pk Pk,head ensuring that the final prediction and its loss are influenced by all prototypes (i.e., learnable synthetic nodes).
A.5 TIME COMPLEXITIES
In this section, assess the time complexity of FedLoG and demonstrate its efficiency in com- putational requirements. Notably, each classifier, including the head and tail branches, shares the same time complexity multi-layer perceptron (MLP), specifically (dd), where d represents the feature dimension of its input. This ensures that the branches have minimal computational overhead, even when processing high-dimensional features. We provide detailed time complexity calculations for each module follows:
Classifier. Pairwise Distance between Prototypes (Eq. 7 The naive time complexity is O(PF), where P is the number of prototypes (i.e., |Cul x s) and F IS the dimension of the input (2 x d, where denotes the dimension of its inputs). Since P is small enough to be negligible, the complexity reduces to (dd).
Distance- Based Message Generation (Eq. 6). The naive time complexity O(FF2), where F is the input dimension (i.e., 2 x d 1). With P Peinng negligible, this results in a complexity of (dd)), which is the same that of MLP.
Updating the Target Node's 's Representation (Eqs. operations with subtraction, giving total complexity of O(d)). Eqs. and 10 involve only simple additions and are therefore negligible. Thus, the total time complexity Tor this update (dd).
Eq. includes an MLP and elementwise IOp.0
Prompt Generator. In the pretraining phase, the prompt generator requires O(DVdd complexity for for Eq. [2] Therefore, the total time complexity of the prompt generator is OQE + Mld2), which is the same as the GNN encoder. However, it worth noting that, during inference, it requires only (dd)) which has the same time complexity as the MLP.
Eq. and O d + | [V(=2)
Feature Scaling. For feature scaling in Eq. 5 the time complexity ((d) since the operation involves only simple element- wise additions.
Consequently, the total time complexity of the classifiers, including the Prompt Generator and Feature Scaling, is (dd)). ) This is significantly lighter than the complexity of the GNN encoder, O(EEd (Vld)) where El and [V] denote the number of edges and nodes, respectively.
Graph Encoder. As we utilize GraphSAGE for the GNN encoder, it requires O(EEdd + (Vd?) for both forward and backward passes.
Overall Model Complexity. To sum up, our model requires O((Ed + |Vd2) complexity, which is the same as a GNN encoder. Importantly, each classifier has the same time complexity as an MLP (i.e., ((d2)), which has little influence on the total complexity of our architecture. This highlights the efficiency of FedLoG in handling computational demands, even for large-scale graphs with high-dimensional data.
A.6 RELATED WORKS
A.6.1 IMPROVING GENERALIZATION IN FEDERATED LEARNING
One of the core challenges in Federated Learning (FL. is achieving strong generalization across heterogeneous and biased client datasets. Clients often have non-i.i.d. data distributions class
19

---

Published as a conference paper at ICLR 2025
imbalances, which make it difficult to train robust and generalized models. To address these challenges. various approaches (Chen & Chao 2021: Li et al. 2023b a Ye et al. 2023) have been proposed,
focusing on improving the generalization capabilities of both global and local models.
FedRoD (Chen Chao )021) bridges the gap between generic FL and personalized FL by leveraging a class-balanced loss and empirical risk minimization. While this approach improves generic Fl., it depends on the presence of least one data point for each class within each client. This reliance makes it less effective in scenarios where certain classes are entirely absent in some clients, a common challenge in federated learning (i.e., the Missing Class setting).
FedETF (Li et al., 2023b) addresses classifier biases by enhancing the generalization of the global model and enabling personalized adaptation through local fine-tuning. To improve generalization, FedETF employs a balanced feature loss weighted by the number of samples in each class. However, its generalization phase does not adequately handle the Missing Class scenario, where certain classes have samples at all. Furthermore, its reliance on local fine-tuning exacerbates the local overfitting problem, making local models more prone to overfitting their biased data distributions and struggling to generalize to unseen data, such as missing classes.
FedLAW (Li et al. 2023a) enhances the generalization of global models by introducing a learnable weighted aggregation mechanism, where the LI norm of the aggregation weights is constrained to be less than 1. Additionally, it incorporates the concept of client coherence to identify clients that positively contribute to generalization. Similarly, FedDisco (Ye et al. 2023) proposes a weighted aggregation method based on the discrepancy between local and global category distributions, further improving the performance of the global model. While both FedLAW and FedDisco primarily focus on enhancing the generalization of the global model, our work takes a different approach by addressing the local overfitting problem. Specifically, we aim to improve the generalization of local models, which are prone to overfitting their local data distributions after a few local updates from the global model, even when the global model itself is well generalized.
A.6.2 SYNTHETIC-BASER FEDERATED LEARNING
Generating synthetic data using aggregated knowledge from clients has emerged as promising approach to compensate for the limitations of local training data. This method facilitates data augmentation while addressing challenges such as class imbalance and limited data availability.
Mixued synthetic data generation methods YYon et al., 202 I You et al. 2024 Oh al. 2020 (hin et al. 2020) augment training datasets by mixing data samples with privacy prreycppeesrinngggg techniques. However, these approaches operate in the raw feature space, which poses significant risks of privacy leakage. particularly when the local data size is small.
Alternatively, GAN-based (Goodfellow et al. 2020) methods, such as FedGAN (Rasouli et al., 2020) and FedDPGAN (Zhang et al. [202 leverage generative models to create synthetic data. These methods aim to generalize a global generator to produce synthetic data that can mitigate data imbalance while preserving privacy. However, they incur high computational costs, which limits their practical applicability.
Recently, condensation- based methods (Kim & Choi 2022; Liu et al. 2022 Wang et al. 2024) have been proposed to alleviate the impact of data heterogeneity. FedDC (Kim & Choi 2022) condenses synthetic data based on local data and fine-tunes the global model at the server level to ensure stable convergence. FedMK (Liu et al., 2022) generates synthetic data by condensing private data into meta- knowlegee which is used as an additional training set to accelerate convergence. FedAF (Wang et al. 2024) introduces an aggregation-fie free paradigm, where the server directly trains the global model using condensed synthetic data.
Key distinctions of our approach compared to existing synthetic-based methods are as follows:
No reliance on raw features. MixUp-based (Yoon et al.) 2021 You et al., 2024| Oh et al. 2020
Shin et al.] 2020) and condensation- based methods (Kim & Choi 2022 Liu et al. [2022] Wang et al. 2024) generate synthetic data by augmenting or condensing data at the raw feature level of the input, which can lead to privacy leakage, particularly when the original data is limited.
contrast, our synthetic data has distinct feature distribution from the original data, arising from differences in the embedding approach used for synthetic and original data, particularly due to the presence or absence of the explicit structure. Furthermore, our method leverages not only the original data but also global synthetic data as an additional training set for condensation. This design significantly
20

---

Published as a conference paper at ICLR 2025
reduces privacy risks, especially when the local data size is small. Moreover, only share subset of the synthetic data (i.e., synthetic data within the head-degree branch), which not only excludes complete information about the local data to enhance privacy but is also specifically designed to capture reliable information relevant the graph domain.
Handling the local overfitting problem. Our method effectively addresses the local overfitting problem, which is of the most challenging issues in federated learning. Local overfitting occurs after a few local updates with the distributed global model, causing the local model to severely struggle in predicting unseen data that involves unseen distributions, particularly for missing classes. MixUp--asee approaches YYonn et al.] 2021, You al.) 2024) Oh et al.. 2020] [hhin al. (2020) still depend on local data for augmenting the training set. which limits their ability to generate data for missing classes. In contrast, our method generates global synthetic data even in scenarios where local data for certain classes is completely absent. This is achieved without relying on raw reature-based MixUp, ensuring both privacy and flexibility.
Optimizing training with synthetic data. We extend beyond the generation of synthetic data by investigating how train it effectively. Since our syntheric data has a different feature distribution from the original data but is utilized as training data (i.e., local generalization), explore how to optimize the model training process with global synthetic data. To address this, we propose the Feature Scaling and Prompt Generator phases, as detailed in Section 4.3) to minimize the training-effect gap between original nodes and synthetic nodes.
essential to
A.7 DETAILED PROCESS OF PRETRAINING THE PROMPT GENERATOR
In this section, we explain the process pretraining local prompt generators and how they are aggregated on the server to produce unbiased prompt nodes for each class. Specifically, the primary goal of the prompt generator is
that the synthetic graph-compriisng target node and its corresponding prompt node derived from the target node's s Sratures-produces a similar gradient matrix as when the target node is predicted using its true h-hop subgraph within the local graph.
Local (Client k).
Server.
mean
Festure Matcching
Global PG
Exjon
Class PG
Local graph
9k
gen feat.
Client
Class 2
Trux sutggrast Steee
Client
PG
GNN yand
ganue
PGg
Eegrat
Matching
Class 3
PG
Vaag
Spsthetk paph
Figure 6: Overview of Pretraining the Prompt Generator.
Training Local Prompt Generators. Each client l: trains its own prompt generator PG aiming to generate synthetic prompt node that optimizes the GNN's training effectiveness feature-only data (Figure left)), Let 9 be the generated synthetic graph consisting of the target node Uk and its generated prompt node vg. The objective is to approximate the true h-hop subgraph around within the local graph GA (denoted as GR.ve (V..vg,& Ex, S Gk) in a compact form within g (Jin et al. 202T 2022).
To achieve this, the prompt generator applies feature matching to ensure that the generated prompt node has features similar to those of the true neighbors of the target node. In essence, the synthetic prompt node acts as "compressed summary" of the surrounding structural information derived from the target node's features. The input to the generator PG, is the feature vector of the target node x and it outputs synthetic prompt feature म. PG (x..) E Rdd To align the synthetic prompt node with the average features of real h-hop neighbors within the local graph, minimize the following
21

---

Published as a conference paper at ICLR 2025
feature-matchin loss:
feal
Rxgg &No, 11 where EN.
x--
(11)
Vill
2ic EVE
ve VE Ae
To ensure that the training effect on the synthetic graph Gk.o, resembles that on the true /t-hop graph Gk.uy gradient matching is applied. This approach minimizes the difference between gradients of the GNN when trained on the true h-hop graph versus the synthetic graph, aligning the parameter updates and thus making the learning process similar. The
loss is defined as follows:
Li.grad
( Riny (vR), Mo.) Gond
(( V and
V
Ghie (UR), You I|:
(12)
...,...,44 where N is the number of randomly initialized weights onnd (for 1,2, ...,, N) used to optimize the target-node classification task.
The combined loss for optimizing the local prompt generator is: LPGs = Cx, + Lk.grad.
(13)
Thus, the generator produces a single synthetic prompt node feature & E Rd, which serves two main purposes: (1) it captures essential structural information from the target node's features, and (2) it enhances the GNN's learning effect feature-only data.
We pretrain PG, for & E (l...., K} over P (i.e., 100) epochs using the training sets within each local dataset, resulting in collection PG = (PGI..... PG k. We set N to 20.
A.8 DETAILED PROCESS OF EVALUATING DATA RELIABILITY
In this section, we detail the process of evaluating data reliability as outlined in Section 3.1 We define 'data reliability' as the accuracy and consistency of information from decentralized nodes. Specifically, we assess which data within the local dataset positively or negatively impacts other clients in the FL framework.
Inspired by the robust performance of GNNs for head degree and head degree data Yunn al. 2022
Park et al. [00T]] Zhao et al. 2021] Liu et al. (02I), we design experiments to evaluate data relibilit from two perspectives: 1) headness of degree and 2) headness of class. We use the PubMed dataset for validation.
We set the base settings for both perspectives. In the FL framework, we assign two roles to each client. The 'receiver' is the client who receives information about the target class from other clients. This client is trained using the same training data across all settings for this section, ensuring a fair comparison to validate the impact from other clients. 'Contributors' are the clients who share knowledge from their own data with the 'receiver'. Their training sets (i.e., information shared through the FL framework) vary for each setting, such adjusting the proportion of head/tail degree nodes or class imbalance rate. In a global setting with K clients in FL, we assign one client as the 'receiver' and the others as 'contributors' (i.e., K clients).
To assess how degree or class headness affects data reliability, we measure the target class accuracy of the 'receiver' when varying the training sets of 'contributors". This helps identify whether headness tailness of data positively negatively impacts the 'receiver'. We construct the global model by averaging the weights from each client and then evaluate the global model on the 'receiver' 's local graph following FedAvg [McMahan et al. 2017).
Detailed Process for Headness Degree Perspective We divide head degree and tail degree using the tail degree threshold À set to 3, as justified in Appendix A.9 Nodes with degrees less than or equal to are considered tail degree nodes, while those with degrees greater than 3 are head degree nodes. We only vary the training dataset of the 'contributors'. We create three different training sets for each 'contributor': 1) Head degree nodes only (Head degree). 2) Tail degree nodes only (Tail degree), and 3) Balanced degree nodes (Balanced degree). Each training set contains the same number of nodes, but their headness differs according the setting. The 'Head degree" setting includes only
22

---

Published as a conference paper at ICLR 2025
head degree nodes with the target class, the 'Tail degree' setting includes only tail degree nodes, and 'Balanced degree' setting includes an equal mix of head and tail degree nodes.
We use the FedAvg [McMahan et al. 2017) framework for the federated learning setting with 100 rounds. At the final round. we evaluate the accuracy of the target class within the 'receiver's s' local data using the global model. We average the performances across all classes and report the mean of three seeds results.
Impact of Class Headness on the Data Reliability We define the 'imbalance rate' using the proportion of the number of the target class within each 'contributor's local data. We fix the training nodes of the target class for each 'contributor', and varies the number of training nodes for other classes which are not the target class. Let ne be the number of nodes per class, and let the number of training nodes of the target class be Rt. We then assign ni number of training nodes for each non-target class:
Th = n4 + Timb
x min(n.Ve C)
(14)
10
where min (n, E C") is the minimum number of nodes across all other classes e within the set C, and Timb is the imbalance rate can be defined as:
71 72
Fimh 10
min(n C)
Thus, if the Timb has negative value (i.e., ne ng). it means the target class becomes head class within the local data. Conversely, when the Timb has a positive value, the target class becomes tail class. As the value of Timb increases, the tailness of the target class gets higher. We set Timb in the range from -5 +5, and we average the performances of the 'receiver' the final round across all classes and report the mean of three seed results.
A.9 CRITERIA FOR THRESHOLD DEGREE VALUE FOR TAIL-DEGREE NODES
Recent methods (Yun al. 2022 Liu et al. 2021 ad- dressing the degree long-tail problem consider nodes with degrees less than equal to 5 as tail degree nodes, while those with degrees greater than 5 are considered head degree nodes.
IIII
Th
1500
1250
# Nodes
1000
750
As shown in Figure 7 we illustrate the number of nodes belonging to 1) head class & head degree (HH), 2) head class & tail degree (HT), 3) tail class & head degree (TH), and 4) tail class tail degree (TT) as we vary the threshold value A within the global graph. We use the Cora dataset for validation.
250
1
3
5
Threshold 1
Figure 7: The number of nodes for HН/HT/ТH/TT at threshold À (Cora dataset used).
When the threshold À increases, the number of HH nodes significantly decreases, reducing the amount of knowledge that can be condensed into the global synthetic data. this work, we set X to 3 to utilize a sufficient amount of HH knowledge while filtering out noisy information from tail degree nodes.
A. DETAILED PROCESS OF EVALUATING UNSEEN DATA
In this section, we provide a detailed description of our proposed 'Unseen Data" test settings (i.e., UUnseen Node", 'Missing Class', and "New Client'). To evaluate realistic scenarios, define two different settings for evaluating unseen data: 1) Closed set nodes setting (Closed set) and 2) Open set nodes setting (Open set). The results in Table 2 are evaluated on the closed set nodes setting.
A.10.1 CLOSED SET
23

---

Published as a conference paper at ICLR 2025
Following recent work (Back et al. 2023)
we partition the global graph into several sub- graphs using the Metis graph partitioning al- gorithm (Karypis Kumar 1997). For the "New Client' setting, we generate an addi- tional subgraph, resulting the partitioning of the global graph into k + subgraphs, where & denotes the number of clients. Due to the properties of the Metis algorithm, the extra subgraph has a distinct label distribu- tion, as the algorithm minimizes the number of edges between partitions, leading to the formation of distinct communities.
V1
1/2
1/
1/2
New Client
New Client
V/
V/3
(a) Closed set
(b) Open set
Figure 8: Overview of Unseen Data settings (K 3).
The closed set setting includes unseen data for the 'Unseen Node' and 'Missing Class' settings from other clients. Specifically, in Figure 8)). (). the global set of nodes is V = U Vs. with V; nn, 2 for all + We construct the UUnseen Node' and "Missing Class' nodes for client ls by expanding the h-hop subgraph from the local graph Gk- Since we allocate all nodes within the global node set V to the clients, the nodes within the ht-hop subgraph (i.e., VE inevitably overlap with those of other clients. Although nodes may overlap, no edges are shared between different clients. Unseen nodes from other clients establish new connections with the local data.
For the 'Missing Class' setting, we select the missing classes for each client and then exclude the nodes corresponding to those classes (i.e., Ve within each local graph Gs. To maintain the overall context of the local graph, we select the missing class from tail classes, which have the smallest portion within each local graph. the number of nodes corresponding to the missing classes is insufficient, add additional missing classes for those clients. Excluded nodes Ve included in VE.
When evaluating the 'Missing Class' at test time, we expand the local graph GR to the range of h-hop, and within the evolved graph structure, the local model predicts the labels of nodes in V For "Unseen Node', the local model predicts the labels of nodes in VE Vic
For real-world case for the closed set setting, consider Store-A, which uses model tailored to the purchasing habits of its regular customers. This model may struggle to adapt to the distinct buying patterns of customers from Store-B. These new patterns could create unfamiliar 'also-bought" connections between products within Store-A, especially if they involve new products that Store- A has never sold before. However, these customers can visit Store-A at any time, forming new relationships with existing nodes, reflecting real-world scenario. This complexity increases the difficulty in effectively integrating and addressing new nodes in the model. In addition, we provide the data statisties for each setting in Appendix A.16
A.10.2 OPEN SET
In real-world scenarios, unseen data outside the global nodes V in the FL system can emerge and form new relationships with existing nodes. We define this setting Open Set, where the unseen nodes are V/ n Ø. To create this setting, we randomly crop 20% of the global graph before partitioning it into A ++ subgraphs, denoting the cropped node set as Verop-
Similar to the Closed Set, exclude nodes corresponding to locally assigned missing classes within each local graph. At test time, for the 'Unseen Node and 'Missing Class' settings, we reconstruct the structure between cropped nodes Varop and local nodes Vg. Within the reconstructed graph, we evaluate the nodes Verop that belong to the missing classes for the 'Missing Class' setting and those having locally trained classes for the 'Unseen Node setting.
In Table we provide the experimental results on the open in Appendix A.11.
A.11 ADDITIONAL EXPERIMENTS
A.11.1 IMPACT OF THE HYPERPARAMETERS
In Figure 9 we analyze the impact of hyperparameters such as the number of synthetic data for each class (8) and the tail degree threshold (A).
24

---

Published as a conference paper at ICLR 2025
CD Data
0:3
13-
Seen Graph
Unseen Node
Missing Class
New Client
Seen Graph
Unseen Node
Missing Class
New Client
(a) The number of synthetic data,
(b) Tail-degree Threshold A
Figure 9: Hyperparameter analysis.
The Number of Synthetic Data, s For generating the global synthetic data, sets of learnable nodes Vk.head and VR cail are constructed during the Local Fitting phase within each client. We assign learnable synthetic nodes per class and vary to assess its impact on global synthetic nodes.
As shown in Figure a),, we vary 8 within the range [1, 5, 10, 20,50) and evaluate the model's performance on the same test data using the Cora dataset with 3 clients. Notably, significantly impacts the 'Unseen Data' settings, particularly the 'Missing Class' setting, which relies heavily on global synthetic data. A larger number of synthetic data condenses diverse knowledge expressions. However, too many synthetic data points complicate modeling the interaction between the target node and each synthetic nodes (i.e., prototypes), as all prototypes participate in the final prediction described in Section 4. Consequently, accuracy for 'Unseen Data' incluing 'Unseen Node', "Missing Class', and "New Client' improves with more synthetic data, but an excessive number (e.g., 50) can reduce performance. Conversely, the performance of the 'Seen Graph' settings shows robustness to the number of synthetic data compared to the 'Unseen Data' settings because the dependency on knowledge from other clients is lower for test data with the same distribution as the training data.
Tail-Degree Threshold, A We evaluate the impact of the tail-degree threshold À on performance. Varying À within the range [0, 3, 5, 10, 20), we use the CiteSeer dataset with 3 clients for the evaluation. As shown in Figure((b), the tail-degree threshold À significantly impacts the "Unseen Data' settings as it directly influences the knowledge condensed into the global synthetic data. Increasing filters out more knowledge from tail-degree nodes, condensing primarily head-degree node knowledge. However, as illustrated in Figure in Section Section[.99] the number of HH nodes significantly decreases with a higher X, reducing the amount of knowledge to be condensed into the global synthetic data. Thus, setting À to 3 yields the best performance, effectively filtering out tail-degree knowledge while ensuring a sufficient amount of HH nodes.
Degree-Based Branch Weight for Prediction, The primary objective of each branch is to distill knowledge from the input data into learnable synthetic data. To achieve this, we designed prototypical network-based branch that uses Iearnable synthetic data to represent class-specific knowledge. By adjusting the weight (i.e., alpha) of each branch's s final prediction based on the target node's degree, we guide the gradient flow from head degree nodes primarily towards the head branch. This approach enables head degree knowledge to be distilled within the head branch's 's synthetic data, and similarly, tail-degree knowledge within the tail branch.
As shown in Section 3.] tail-degree knowledge negatively affects the performance of other clients. This effect is illustrated in Figure where we see that synthetic data generated from knowledge (i.e., HH: head class/head degree and TH: tail class/head degree) outperforms that generated from tail-branch knowledge (i.e., HT: head class/tail degree and TT: tail class/tail degree). Specifically, the performance hierarchy (HH > HT and TH > TT) suggests that the head branch holds more reliable knowledge from the input data, particularly head degree knowledge, as discussed in Section 3.I. This demonstrates that each branch effectively captures distinct types of knowledge, successfully separating head and tail degree information from the input data.
25

---

Published as a conference paper at ICLR 2025
Table 7: Impact of degree-based branch weight a on performance (Cora dataset used).
3 Clients
Clients
10 Clients
SG
UN
MIC
NC
5G
UN
ATC
NC
SG
UN
MIC
RC
FedLoG (a 0.5)
0.8613
0.7154
0.3608
0.4852
0.3519
0.7397
0,4837
0.3982
0.8377
0.7160
0,3769
U.3285
19333
20000
20030
19960
FedL.G
0.8001
0.739
0,6472
0,5047
0.8575
0.7413
0,4948
0.4439
U.841
0,7406
0.4037
U.MUSS
mmm
1200
20000
Improvement (%))
0.12
+1.77
+8.04
+1.3
+0.56
40.16
+1.11
+4.37
+0.74
+2.46
+2.08
+7.70
SG: Seen Graph, UN: Unseen Node, MC: Missing Class, NC: New Client
To provide additional clarity, we conducted an ablation study evaluate the effectiveness of weight averaging based on the target node's 's degree. In this ablation, we fixed the a value in Eq. 3 at 0.5, preventing degree-specifee knowledge from being divided across branches. This configuration results in simple ensemble of the two branches without considering the degree. The results of this ablation study are shown in Table 7
The performance on the unseen data settings (i.e., UN, MC, and NC) differs significantly from that on the SG (Seen Graph) setting. Specifically, eliminating degree-baser weight averaging leads to a significant performance decrease unseen data settings. This difference arises because, in unseen data settings, the model relies more heavily on the reliability of global synthetic data. Consequently, weighting predictions from each branch based on the target node's 's degree effectively extracts reliable knowledge into the head branch while preserving tail-specific knowledge within the tail branch.
A.11.2 ASSESSING THE ADAPTIVE IMPACT OF FEATURE SCALING ON LOCAL CLIENTS
FedLoG shares the same global synthetic data with clients at the end of each round, but clients have distinct absent knowledge due to different label distributions. Clients in FedLoG adap- tively utilize the global synthetic data by adjusting its perturba- tion strength in a wlse manner, as described in Section [43] We verify that the adaptive factor optimally adjusts the pertur-
Table 8: 7/c values the final round R (CiteSeer 3 Clients).
Class
5
# Nodes
5
27
129
16
25
0.288 0.289 0.297
0.282
0,282
0.286
bation strength for local clients. Table S shows the value of the adaptive factor for class (i.e., 7[c)) in Client at the last round R. The adaptive factor for the head
class (i.e., class 3) higher than that for tail (i.e., class 1) and missing classes (i.e., class 5), showing that it effectively adjusts the perturbation strength based on each client's s current learning status.
A.11.3 EXPERIMENTAL. RESULTS ON THE OPEN SET
We evaluate the Unseen Node and Missing Class in the Open Set settings validate the model's ability to generalize to nodes never seen at the global level. The results are provided Table Similar to the Closed Set, method, FedLoG, outperforms the baselines across most settings. However, in the Unseen Node setting on the PubMed dataset, some baselines show better performance than our method. We attribute this to the PubMed dataset providing sufficient number of training data for each class, allowing methods to generalize well within each class's local data. Conversely, in the Missing Class setting, the baselines fail to generalize due to the absence of local data for the missing classes. In contrast, our model effectively generalizes to all classes, including missing classes, demonstrating its robustness on various real-world scenarios.
A.1 ABLATION STUDY
Unseen Node
Missing Class
New Client
0.7
0.66
0.5
0.5
0.5
0.4
0.3
0.4
Fed
02
w/o LG
0.3
0.1
w/o PG
0.2
0.0
0.3
20
40
60
100
60
100
20
40
60
100
Round
Round
Round
Figure 10: Ablation studies (CiteSeer 3 Clients).
26

---

Published as a conference paper at ICLR 2025
Table 9: Performance on Unseen Node and Missing Class in the Open Set setting.
Cara
Ciidior
PukAeel
Ansacan Paste
Amsann Compalers
Methhal
10CTRAAN:
1DC
SCClean
Lacil
Lo) Unoeen Node
Fedlog
9:40
K2200
8880
308
828
6.00
0.7102
9.48
92000
0967
9.81
940h
08.M0
4.80
840
600
0800
0.807
8811
6400
0.800
9-40
8450
00.0
9.00
9400
0.658
0.591
PelGCN
8.000
00.94
8100
0.818
0990
8:00
0.113
06554
0.3411
0,617
0.363
0.307
PedPUB
66872
03973
04333
0.887
67222
0,935
0.0644
0.8564
14442
0,500
0,500
04886
0,5008
0,4014
3066
16600
04315
0.443
6,638
0.5054
0.353
0,000
0,5399
0.506
0,6100
0,003
6414)
0,600
0,677
FedlD
0.604
0.5453
0,004
0,5855
0.4148
0.875
0,671
0,8905
06481
0.5672
0,3581
64336
0,300
0,5931
0,8822
009627
607544
0,163
CleSeer
PulMel
Plitte
Amaave Compaters
Mathads
STlless
15C
1C Cliens
3.7keass
SCleann
10 Cilmes
0.000
bu000
0,0000
0,0000
0,030
0,0000
1.000
0,0003
D/3303
1,0300
0.18881
00003
0.404)
0.
(b Missing Class
Folldng
0.000
RA391
0,0000
0.1801
64369
01430
0,371
0,699
B0085
00080
0,0000
00000
60030
0.000
0,0000
01300
02214
008344
0.3243
04133
0,303
00733
B1008
00030
030003
60073
00033
01000
0,0097
FreloON
01540
0.079
0.035
0/049
00000
FearuB
10000
00083
0,855
0,000
6055
00076
0,0002
00000
00000
0.000
00000
60000
0.000
0.000
PoUNTD
FedEED
0333
PM
0.U27
00004
Fedtat
0.4373
0.3849
0,4184
IL3897
8,758
8:5087
B.3333
0,5421
6:668
0,2029
We perform an ablation study on 1) Local Generalization (w/o LG). 2) Prompt Generation (w/o PG), and 3) Feature Scaling (wo FS). As these modules are all directly related addressing unseen data, we depict the test accuracy curves in Unseen Data settings to easily verify the effectiveness of each module.
Local Generalization Local Generalization is an essential phase to prevent local overfitting after the local updates of each client within the FL. framework. The Local Generalization phase enables clients to learn locally absent knowledge from the global synthetic data, allowing them to generalize all classes even if they don't 't have any data for certain classes within their local data (i.e., missing class). As shown in Figure I0, our method without the Local Generalization phase fails to generalize the missing class, which means Local Generalization is crucial for addressing the absent knowledge. Furthermore, for the Unseen Node and New Client settings, the performance deteriorates when we omit the Local Generalization phase.
Prompt Generation We evaluate the effectiveness of the prompt generators PG*V(c). The prompt generators generate the prompt nodes of the global synthetic data Dg, which contain the h-hop neighbor information for the target nodes and also contribute to training by mimicking the true h-hop subgraphs' gradient. We perform the ablation study for the prompt generators by omitting the generation of prompt nodes for the global synthetic data, which means train them without any generated prompts. In Figure 10, without prompt nodes, there is a discrepancy in the training mechanism of the GNN between isolated nodes and nodes within the graph structure, leading to performance decrease for all settings. Furthermore, the learning curves fluctuate when training the global synthetic data without prompt generation, indicating that using only the features of synthetic nodes negatively affects stability.
Feature Scaling Feature Scaling helps each client learn all classes adaptively. Feature Scaling adjusts the strength of the perturbation of the global synthetic data for each client depending on the class prediction ability for all classes at the current round. Thus, Feature Scaling affects the stability of learning for each client, In Figure I0 we can verify the effectiveness of Feature Scaling, as the learning curves are more fluctuating than the original FedLoG method, and the performance is decreased.
A.12 DATASETS
Cora (Sen et al. 2008): The Cora dataset consists of 2,708 scientific publications classified into one of seven classes. The citation network contains 5,429 links. Each publication in the dataset is described by a I, 1.433-dimensional binary vector, indicating the absence/presence of a word from a dictionary.
27

---

Published as a conference paper at ICLR 2025
CiteSeer (Sen et al., 2008): The CiteSeer dataset comprises 3,327 scientific publications classified
into one of six classes. The citation network consists 4,732 links. Each publication described by a 3.7703-dimensional binary vector.
PubMed (Sen et al., 2008): The PubMed dataset includes 19,717 scientific publications from the PubMed database pertaining to diabetes, classified into one of three classes. The citation network comprises 44,338 links. Each publication is described by a we/ghe-weighted word vector from dictionary with size of 500.
Amazon Computers (McAuley et al., 2013): The Amazon Computers dataset is subset of the Amazon co-purchase graph. It consists of 13.752 nodes (products) and 245,861 edges (co-purchase relationships). Each product is described by 767-dimensional feature vector, and the task to classify products into 10 classes.
Amazon Photos (Shchur et al., 2018): The Amazon Photos dataset is another subset of the Amazon co-purchase graph. It consists of 7,650 nodes (products) and 143,663 edges (co-purchase relationships). Each product is described by 745-dimensional feature vector, and the task to classify products into 8 classes.
A.12.1
DATASET STATISTICS
Table 10: Dataset Statistics
Dataset
Nodes
Edges
Features
Classes
Description
Cora
2,708
5,429
1,433
7
Scientific publications
CiteSeer
3,327
4,732
3,703
6
Scientific publications
PubMed
19,717
44.338
500
3
Scientific publications
Amazon Computers
13,752
245,861
767
10
Amazon co- co-purchase
Amazon Photos
7,650
143,663
745
8
Amazon co-purchase
A.13 BASELINES
In this section, provide details for the baselines and the URLs of the official codes, where available.
Local. This is non-FL baseline where each local model is trained independently using the GCN embedder without any weight sharing.
FedAvg. MMMMaan et al., 2017) This FL baseline involves clients sending their local model weights the server, which then averages these weights based on the number of training samples at each client. The aggregated model is then distributed back to the clients. In our implementation, we GCN the graph embedder.
FedSAGE+. (Zhang et al. 2021a) This subgraph-FL. baseline involves clients using GraphS as an embedder and missing neighbor generator, trained using graph mending technique. The neighbor generator creates missing neighbors based on their number and features. With the neighbor generator, local models trained with compensated neighbors and then their weights are aggregated on the server using FedAvg- based FL aggregation.
FedGCN. Yao et al. 2024) This subgraph-FL baseline involves clients who collect hthop averaged neighbor node features from other clients at the beginning of training to address missing information. The server then collects local model weights for FedAvg-based FL. aggregation.
FedPUB, (Back et al., 2023) This subgraph-Fu FL baseline proposes weight aggregation based on the similarty between clients. identifies highly correlated clients with similar community graph structures by using the functional embeddings of local GNNs, which are computed using random graphs as inputs to determine similarities.
28

---

Published as a conference paper at ICLR 2025
FedNTD. (Lee et al., 2022 This FL baseline is designed tackle the challenge of overfitting in local models due mon-IIID data across clients. It performs local-side distillation only for non-true classes to prevent forgetting global knowledge corresponding to regions outside the local distribution. In our implementation, we use GCN as the graph embedder.
FedED. (Guo ef al. 2024) This FL baseline is designed to tackle the challenge of overfitting in local models and addresses the issue of local missing classes. Similar to our task, it addresses the missing class problem in FL by adding loss term that regularizes the logits of missing classes to be similar those of the global model. In implementation, we use GCN as the graph embedder.
Table 11: Baselines and their corresponding code repositories. * We utilized the FedAvg code implemented in the official FedPub code.
Baseline
URL/Nofe
FedAvg*
FedSAGE
FedGCN
FedPub
FedNTD
nttps://githus. httaa:/ggttttttocccmmmmeeeeeaaaa
FedED
Sell-mplemented due to absence of otheral code.
A. 14 IMPLEMENTATION DETAILS
In this section, we provide implementation details of FedLoG.
Model Architecture. In our experiments, implementation (fE) with dropout rate 0.5, hidden dimension of 128, and an output dimension of 64. The model parameters with learnable features X V. bas and X snt are optimized with Adam (Kingma & Ba 2014) using a learning rate of 0.001. The classifiers 0H and Or consist of 2 main learnable functions (i.e., MLP msg and MLP (LLa) as follows:
2-layer GraphSAGE (Hamilton et al. 2017)
Message generating function (MI.P mwg): Two linear layers with SiLU activation (Inputs -> Linear (2 x 64 - 64) > SiLU > Linear (64 + 64) - SiLU - Outputs).
Message embedding function (MLPPuras) Three linear layers with SiLU activation (Inputs > Linear (64 - 64) + SiLU - Linear (64 64) Linear (64 + 1) Outputs).
In all experiments, we utilize 2-layer classifiers.
Training Details. Our method is implemented on Python 3.10, PyTorch 2.0.1, and forch-geometis 2.4.0. All experiments are conducted using four 24GB NVIDIA GeForce RTX 4090 GPUs. For all experiments, set the number of rounds (R) to 100 and the number of local epochs to 1. This setting is applied consistently across all baselines.
the test nodes using the model that achieves the best validation performance across all rounds (R) in the Seen Node setting. This model is then used to evaluate performance in the other unseen data settings: Unseen Node, Missing Class, and New Client, by testing on the corresponding test nodes for each setting.
Evaluation Details. For the evaluation under the Seen Node setting,
We set the number of learnable nodes to 20, the tail-degree threshold to 3. and select the regularization parameter /8 to values in the range of [0.01, 0.1, 1].
Hyperparameters.
A.15 DETAILED PROCESS OF GENERATING HH/HT/TH/TT GLOBAL SYNTHETIC DATA
In this section, we describe the process of generating global synthetic data using 1) head class & head degree nodes (HH). 2) head class tail degree nodes (HT). 3) tail class & head degree nodes (TH), and 4) tail class & tail degree nodes (TT). Fedl_od oG has two branches, each generating Vi.nead and Ve,ii, which contain knowledge from head degree nodes and tail degree nodes, respectively.
29

---

Published as a conference paper at ICLR 2025
As described in Section 4.2 generate HH global synthetic data by merging the head degree condensed nodes Ve,head from all clients, weighted by the proportion of head classes for each client. In Figure d), for each class c C, the feature vector of the i-th global synthetic node for class x (, is defined as:
HH.
x (c.a)
=k=1
K
(c.1)
1 where r = V
represents the proportion of nodes labeled e in the k-th client's dataset.
HT. In generating HT global synthetic data, we substitute Vk, with Vk,tii- Thus, for each class c E C, the feature vector of the i-th global synthetic node for class c, X (e.a), defined as:
K
X (c.5)
(c,s)
TH. For generating TH global synthetic data, we aim to give more weight to the tail classes. To achieve this, adjust the weights inversely proportional rs. ensuring that tail classes (with lower D receive higher weights. The new equation is given by:
K Lj=1
1 Lk
(c.) where o
(15)
r +
k=1
Here, is a very small positive value added to prevent division by zero. In this revised equation, o assigns higher weights classes with smaller r values, thereby giving more importance the tail classes.
TT. Finally, we generate TT global synthetic data using:
K
Σx rj 14 +
(c,5) Ek=l O k=1
of (c,0, kuui
where of
(16)
A.16 EXPERIMENTAL DATASET STATISTICS
In this section, we provide the experimental dataset statistics for all testing settings for three clients, allowing for an easy verification of the data distribution of each client and the New Client. In the 'Global' row, sum up the statistics from all local clients.
30

---

Published as a conference paper at ICLR 2025
Table 12: Cora Dataset Statisties (Closed Set)
Seen Graph
Unseen Node
Missing Class
Dataset
Class
Train
Valid
Test
Test
Test
0
49
37
32
268
86
1
82
45
75
258
0
2
140
110
133
217
220
Global
3
242
206
171
605
0
4
120
82
90
268
5
45
41
32
120
29
53
27
59
0
12
4
121
0
1
7
4
3
124
0
2
3
190
Client 0
3
208
168
131
125
0
4
33
22
22
5
0
0
0
29
0
23
0
0
86
1
7
3
117
0
2
136
108
130
27
0
Client
3
15
14
202
0
4
74
49
60
74
0
5
2
6
0
0
0
36
0
41
25
28
147
0
1
68
69
2
0
0
0
220
Client 2
3
28
23
26
278
0
4
13
8
98
5
42
37
30
76
6
53
27
0
222
12
2
2
New Client
3
107
87
5
166
31

---

Published as a conference paper at ICLR 2025
Table 13: CiteSeer Dataset Statistics (Closed Set)
Seen Graph
Unseen Node
Missing Class
Dataset
Class
Train
Valid
Test
Test
Test
0
41
24
25
72
0
90
185
2
196
163
137
424
93
Global
3
116
85
77
210
0
4
154
96
110
132
121
5
37
25
26
88
53
0
11
1
52
50
58
59
0
2
67
52
32
296
0
Client 0
3
73
52
44
76
0
4
7
7
60
0
5
0
53
0
5
2
18
0
1
27
16
26
79
0
2
129
111
105
128
Client
3
16
15
14
66
0
4
0
121
5
25
13
17
44
0
0
17
14
12
28
0
1
6
47
0
2
0
0
93
Client 2
3
27
18
19
68
0
4
147
95
103
72
5
12
12
9
44
0
0
35
53
2
12
New Client
3
110
4
93
5
211
32

---

Published as a conference paper at ICLR 2025
Table 14: PubMed Dataset Statistics (Closed Set)
Seen Graph
Unseen Node
Missing Class
Dataset
Class
Train
Valid
Test
Test
Test
0
1271
983
949
221
168
Global
1176
897
951
572
1003
2
2972
2171
2141
658
0
0
277
222
209
93
0
Client
346
2
1642
1227
1189
193
0
0
994
761
740
128
0
Client
0
657
2
594
408
414
213
0
0
0
0
0
0
168
Client 2
1176
897
951
572
2
736
536
538
252
0
0
787
New Client
3500
2
591
33

---

Published as a conference paper at ICLR 2025
Table 15: Photos Dataset Statistics (Closed Set)
Seen Graph
Unseen Node
Missing Class
Dataset
Class
Train
Valid
Test
Test
Test
0
150
102
108
61
28
592
502
468
881
2
266
219
197
61
20
358
241
258
329
Global
4
300
258
250
327
0
332
217
247
0
25
214
146
128
907
7
39
17
27
307
0
0
0
0
0
28
71
50
49
345
0
2
261
215
196
3
0
49
26
33
135
Client 0
4
5
6
10
186
0
332
217
247
0
6
9
5
7
96
0
5
59
146
100
107
7
1
394
369
333
226
0
2
O
0
0
0
20
11
8
8
101
0
Client
4
3
0
67
0
O
0
0
6
197
132
113
69
0
3
40
54
127
83
86
310
0
5
4
1
58
298
207
217
93
Client
4
292
251
240
74
0
0
0
0
7
6
8
9
8
742
0
29
12
15
208
0
72
3
3
43
New Client
64
5
1412
7
248
34

---

Published as a conference paper at ICLR 2025
Table 16: Computers Dataset Statistics (Closed Set)
Seen Graph
Unseen Node
Missing Class
Dataset
Class
Train
Valid
Test
Test
Test
0
168
120
115
146
140
297
221
201
1604
2
558
442
410
2
479
91
63
66
765
4
1399
1094
1103
3397
0
Global
5
129
69
98
0
60
6
182
134
164
132
93
7
343
220
232
4
167
748
597
580
1321
9
119
80
79
105
22
0
164
118
114
32
0
98
65
71
398
0
2
558
442
410
2
0
66
42
46
206
4
41
25
30
813
0
Client 0
0
0
0
0
46
6
0
0
0
0
93
343
220
232
4
0
8
12
10
8
447
0
9
108
59
61
25
0
0
4
2
1
114
0
125
84
80
573
2
0
0
0
0
262
7
5
235
4
139
123
117
1504
0
Client
129
69
98
0
6
181
133
164
4
0
0
0
0
0
143
8
707
561
552
178
0
9
11
21
18
80
0
0
0
0
0
0
140
1
74
72
50
633
2
0
0
0
0
217
18
15
324
4
1219
946
956
1080
0
Client 2
0
0
0
14
6
1
0
128
0
7
0
0
24
8
29
26
20
696
0
9
O
O
0
0
22
0
30
1374
2
2
302
4
1360
New Client
6
8
167
9
5
35


Reviews:
[{"review_id": "NQjuIQH4S3", "reviewer": "Reviewer_SvTi", "summary": "Federated Learning (FL) on graphs can struggle with local overfitting due to changes in graph data and shifts in label distributions. Existing methods only focus on local data, which limits their ability to generalize to unseen data. The proposed FedLoG method addresses this by generating global synthetic data from reliable class representations and structural information across clients. This synthetic data helps prevent local overfitting, improving the generalization of local models to unseen data. Experimental results show that FedLoG outperforms existing approaches, and the code is publicly available.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well-organized and has reasonable motivation.\n- The proposed method, FedLoG, is interesting in improving local generalization from overfitting.\n- The experiments are solid, and extensive results are shown to support the proposed method.\n- The experimental results show that the proposed method can achieve the sota performances across different federated graph datasets against strong baselines.", "weaknesses": "- Since the proposed method relies on synthetic data, it is of particular interest whether FedLoG can protect data privacy. Can differential privacy techniques be compatible with FedLoG to further guarantee privacy issues?\n- Some important literature about generalization in vanilla federated learning (not limited to federated graph learning) should be cited and discussed. For instance, FedRoD [1] bridges the gap between global generalization and local personalization. Recently, FedLAW [2] and FedDisco [3] target global generalization from aggregation perspectives, and FedETF [4], which is inspired by neural collapse, shows a potential way of achieving personalization and generalization at the same time.\n\n---\n[1] Chen, H. Y., & Chao, W. L. On Bridging Generic and Personalized Federated Learning for Image Classification. In International Conference on Learning Representations 2022.\n\n[2] Li, Z., Lin, T., Shang, X., & Wu, C. (2023, July). Revisiting weighted aggregation in federated learning with neural networks. In International Conference on Machine Learning (pp. 19767-19788). PMLR.\n\n[3] Ye, R., Xu, M., Wang, J., Xu, C., Chen, S., & Wang, Y. (2023, July). Feddisco: Federated learning with discrepancy-aware collaboration. In International Conference on Machine Learning (pp. 39879-39902). PMLR.\n\n[4] Li, Z., Shang, X., He, R., Lin, T., & Wu, C. (2023). No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5319-5329).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QW4a7Vt8jq", "reviewer": "Reviewer_sNpJ", "summary": "In this article, the proposed method, FedLoG, addresses this issue by mitigating local overfitting through global synthetic data generation, condensing reliable structural information and class representations from clients. By training on these synthetic data, FedLoG adapts to the knowledge gaps in each local dataset, enhancing local models' generalization capabilities to handle diverse, unseen data. Experimental results confirm FedLoG’s superiority in generalization over existing baselines in realistic settings.", "questions": "1. How can it be handeling the privacy concerns?\n2. What complexity it possess in terms of time with respect to the other baselines? Will it also be significant or just comparable?\n3. Comment 4 on weekness. How can the authors see that issue?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "As the paper deals with Federated learning, the major focus should be on what and which kind of data and knowledge is aggregated to train the global model. Based on this, there are following strengths observed:\n1. The paper posses a good motivation and a limited but fair novelty.\n2. The baselines considered are fair enough and the gain is significant.\n3. The organization of the paper is quiet good.\n4. the methodology containing local and global fitting with aggregations is also fairly novel.", "weaknesses": "1. The novelty is fair but also limited. As it looks like an ensemble of multiple techniques used in earlier literature.\n2. The process contains mutliple steps which may result in overhead in terms of computation and storage.\n3. There may be privacy issues while doing such proposed aggregation.\n4. If any client will be having data bias, it may propagate to the global from local model as it is taking average.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "djetIbwFQS", "reviewer": "Reviewer_1kKw", "summary": "This paper studies the subgraph federated learning problem, where the author hopes to generate a global dataset that includes all categories, with s nodes for each category. The goal is to train a node classifier, and the optimization objective is to minimize the total empirical risk of all clients. Each client k's loss function includes both local data loss and global synthetic data loss. This approach allows each client to generalize across all categories, even if those categories are not present in their current client base. The author first discusses criteria for reliable data judgment and then proposes FedLog based on this criterion. FedLog consists of three steps: local training by clients, global aggregation by servers and data generation, followed by further server-side training. Through these three steps, the author solves the local overfitting problem in subgraph federated learning and conducts comprehensive experiments in three scenarios: Unseen Node, Missing Class and New Client.", "questions": "The loss function of each client includes the loss of local data and the loss of global synthetic data. Does this design exist in previous methods based on data synthesis? Is it a standard practice?\n\nRegarding the selection of reliability data, are there any other measurement standards besides degree and class frequency? Are there any other works with high recognition in this area?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In terms of methodology, the design of Feature scaling and prompt generator is very creative. Especially the introduction of the prompt generator can provide a graph structure for the generated nodes, which is a valuable idea.\n\nIn terms of writing, the content of the paper is introduced in detail, and both writing and presentation are professional.\n\nIn terms of experiments, the experimental design is comprehensive and solid, including multiple verification scenarios, and provides sufficient details to ensure reproducibility.", "weaknesses": "The discussion about 4.2.1 is not entirely convincing to me, even though it implicitly contains structural information, is this enough to express complex graph structures? Are there other methods of synthetic data in the Graph Learning field currently, such as graph compression, graph distillation techniques?\n\nConsidering that the current method is essentially an aggregation of features and has little to do with the graph structure, this feature aggregation approach is also common in federated learning for general classification tasks. Therefore, the paper should supplement discussions on data synthesis-based methods in the federated learning field in the related work section.\n\nThis paper chooses nodes with high degree centrality and class centrality for global data synthesis. How can we ensure that each category can generate $s$ nodes? If a certain category is a tail class or missing class among all clients, how can we guarantee generating high-quality s nodes for this category?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9mU2pogaer", "reviewer": "Reviewer_sNMQ", "summary": "This paper proposes a novel personalization method to solve the local overfitting problem of the subgraph federated learning. Specifically, they propose a two-branch prototypical network for handling high-degree and low-degree knowledge of each node,  and a global data generation strategy for tackling the limitation of tail nodes in each client.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tSufficient comparison of experimental results demonstrates the performance advantages of the proposed method. \n\n2.\tThe figures and language are clear.", "weaknesses": "1. The local fitting process comprises two branches of modules and prototypical inference networks, which may bring expensive inference computational costs.\n\n2. The ablation study about the effectiveness of dividing the inference process into two head and tail branches is required. In my opinion, this design works appearing more using the ensemble of two branches, instead of considering the head or tail degree of nodes. In fact, the effect of degree is only considered by using the weighted average of predictions, $\\alpha$. The design of the two branches themselves are not involved to the degree at all. \n\n3.\tThe construction of global synthetic data requires the uploading of local graph nodes, which not only brings extra communication costs but also causes privacy leakage. \n\n4. This technique seems to apply a mixup of data in FL, which has been used in many previous works, e.g., [1,2]. The novelty may be limited.\n\n[1] FedMix Approximation of Mixup under Mean Augmented Federated Learning NIPS 2022\n\n[2] FedMDO: Privacy-Preserving Federated Learning via Mixup Differential Objective TCSVT 2024\n\n5. Some key steps should be included in the maintext. For example, how to generate the edges for the synthetic data is required to be specified in the main text instead of the appendix.\n\n6. In fact, this paper trains personalized models for each client since the learnable nodes and prototypes are customized for each client. Therefore, the personalization baselines should also be included [3].\n\n[3] Federated Learning on Non-IID Graphs via Structural Knowledge Sharing. AAAI 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8SU1Isg2g", "reviewer": "Reviewer_T9gM", "summary": "This paper introduces FedLoG, a novel framework addressing local overfitting in subgraph federated learning. The key innovation lies in its approach to generating global synthetic data by condensing reliable information from head degree and head class nodes across clients. The framework employs a dual-branch architecture to process head and tail degree nodes differently, with adaptive weighting based on node degrees. Through feature scaling and prompt generation mechanisms, FedLoG enables each client to adaptively learn locally absent knowledge. The method is evaluated under three practical scenarios: unseen nodes, missing classes, and new clients. Experimental results demonstrate FedLoG's effectiveness in preventing local overfitting while maintaining privacy and improving generalization capabilities across various graph datasets.", "questions": "The datasets used in this paper are all publicly available (i.e. they are not task-oriented) and have been specifically processed to fit the described scenarios. While this approach provides a controlled environment for testing, is there potential for future work to incorporate specialized datasets that more naturally exhibit the challenges addressed by FedLoG? Such datasets could offer deeper insights into the framework's performance and applicability in real-world contexts where these issues are more prevalent.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality:\nThe paper introduces FedLoG, a framework that innovatively addresses the problem of local overfitting in subgraph federated learning by focusing on the adaptability and generalization of models across clients. The originality lies in its method of using global synthetic data generated from the most reliable node types—head degree and head class nodes—thus tackling the challenge of missing classes and mutable graph structures in a novel way.\n\nQuality:\nThe research is characterized by its high quality, demonstrated through a comprehensive experimental setup that includes diverse real-world datasets and practical scenarios like unseen nodes, missing classes, and new clients. The thoroughness of the ablation studies and the detailed analysis of the results contribute to a robust validation of the proposed method, ensuring that the findings are both reliable and applicable.\n\nClarity:\nThe paper is clearly written, with well-organized sections that effectively convey complex ideas. The authors provide detailed explanations of their methodology, supported by illustrative diagrams and thorough descriptions of experimental setups. This clarity facilitates a deeper understanding of the framework's mechanisms and its impact on federated learning.\n\nSignificance:\nThe significance of this work is underscored by its potential to enhance federated learning applications where privacy and adaptability are crucial. By improving the generalization capabilities of local models and addressing the mutable nature of graph data, the paper contributes significantly to advancing federated learning technologies. Its approach to mitigating local overfitting while respecting privacy constraints is particularly relevant in today's data-sensitive environment, offering practical benefits and paving the way for future research in this area.", "weaknesses": "1） The paper seems to lack a thorough discussion on the complexity of the proposed FedLoG framework. Given the dual-branch architecture that separately handles head and tail degree nodes, alongside the mechanisms for feature scaling and prompt generation, a detailed analysis of computational and implementation complexities would be beneficial. Understanding these complexities is crucial for assessing the feasibility and scalability of the method in practical, large-scale applications.\n\n2）Another potential limitation is the assumption that each client has distinct strengths in terms of node types or class distributions. While this assumption aids in leveraging reliable data from head nodes, it may not hold in scenarios where client data distributions are highly homogenous or overlapping. This could limit the framework's effectiveness and generalization in such environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)