Federated Spectral Graph Transformers Meet Neural Ordinary Differential Equations for Non-IID Graphs
Authors: Kishan Gurumurthy
Himanshu Pal
Charu Sharma
Venue: Accepted by TMLR ()

Abstract: Graph Neural Network (GNN) research is rapidly advancing due to GNNs’ capacity to learn distributed representations from graph-structured data. However, centralizing large volumes of real-world graph data for GNN training is often impractical due to privacy concerns, regulatory restrictions, and commercial competition. Federated learning (FL), a distributed learning paradigm, offers a solution by preserving data privacy with collaborative model training. Despite progress in training huge vision and language models, federated learning for GNNs remains underexplored. To address this challenge, we present a novel method for federated learning on GNNs based on spectral GNNs equipped with neural ordinary differential equations (ODE) for better information capture, showing promising results across both homophilic and heterophilic graphs. Our approach effectively handles non-Independent and Identically Distributed (non-IID) data, while also achieving performance comparable to existing methods that only operate on IID data. It is designed to be privacy-preserving and bandwidth-optimized, making it suitable for real-world applications such as social network analysis, recommendation systems, and fraud detection, which often involve complex, non-IID, and heterophilic graph structures. Our results in the area of federated learning on non-IID heterophilic graphs demonstrate significant improvements, while also achieving better performance on homophilic graphs. This work highlights the potential of federated learning in diverse and challenging graph settings.

Published in Transactions on Machine Learning Research (04/2025)
Federated Spectral Graph Transformers Meet Neural Ordinary Differential Equations for Non-IID Graphs
Kishan Gurumurthy"
kishanurramarhy@@@mall.oom
Deportment of Computer Science Indian Institute of Information Technology Koltayom (IITK)
Himanshu Pal
Machine Learning Lab
International Institute of Information Technology Hyderabad
Charu Sharma
charu.skarma@hitl.com
Machine Learning Lab
International Institute of Information Technology Hyderabad (IIIHH)
Reviewed on OpenReview: https: opwrrevee...e.eee.o net/ forum? id- TR61UG8162
Abstract
Graph Neural Network (GNN) research rapidly advancing due GNNss capacity to learn distributed representations from graph-struchured data. However, centralizing large volumes of real-world graph data for GNN training often impractical due privacy concerns, reg- ulatory restrictions, and commercial competition. Federated learning (FL), distributed learning paradigm, offers solution by preserving data privacy with collaborative model training. Despite progress in training huge vision and language models, federated learning for GNNs remains underexplored. To address this challenge, we present a novel method for federated learning on GNNs based on spectral GNNs equipped with neural ordinary dif- ferential equations (ODE) for better information capture, showing promising results across both homophilic and heterophilic graphs. Our approach effectivcly handles non-Independent and Icallically Distributed (noн-IID) data, while also achieving performance comparable to existing methods that only operate on IID data. It is designed to be privacy-preserving and bandwidtt-opiimmzze,, making it suitable for real-world applications such as social net- work analysis, recommendation systems, and fraud detection, which often involve complex, non-IID, and heterophilic graph structures. Our results in the area of federated learning on non-llD heterophilic graphs demonstrate significant improvements, while also achieving better performance on homophilic graphs. This work highlights the potential of federated learning in diverse and challenging graph settings. Open-source code available GitHubl
1 Introduction
Graph Neural Networks (GNNs) (Sarselli et al. 2008) have emerged as powerful tools for learning rep- resentations from graph-structured data, which prevalent in various domains such as social networks, molecular biology, and traffic systems (He et al.) 2021 Liu et al. 2024a). In these applications, data. is often decentralized and subject to privacy concerns, making traditional centralized approaches to training GNNs less viable due to data privacy and regulatory restrictions (He et al. 2021 Ju et al. 2024). Federated Learning (FL) offers a promising solution by enabling decentralized learning across multiple clients without sharing raw data, thereby preserving privacy (Zhang et al.) 2023 Wu et al. 2022 However, applying FL to graph data introduces unique challenges due to the non-Independent and Identically Distributed (non-IID)
* "These authors contributed equally to this work. https://github.. www.Sppringizz////eeaeoooo GNDDEFormer

---

Published in Transactions on Machine Learning Research (04/2025)
nature of graph data and the heterophilic relationships often present in real-world graphs. This has led to the development of Federated Graph Neural Networks (FedGNNs), a. novel area of research that seeks to leverage the strengths of FL to address these challenges Liu et al. 2024a) in GNNs.
Federated Learning (FL) was introduced by McMahan et al. 2017 as a method for training machine learning models across decentralized data sources while maintaining data privacy (Zhang al. 2023). FL. has been successfully applied in domains such as image and language processing, where data is typically IID. However, its application to graph-structured data presents unique challenges because graph data is inherently non-IID and can involve complex dependencies and relationships (Zhang et al. 2023). GNNs are well-suited for modeling such relationships in graph data and have been extensively used for tasks like node classification, link prediction, graph clustering and classifications (Fu et al.) (002).. The integration of FL with GNNs, therefore, aims to leverage the advantages of both, enabling
decentralized learning on graph data while addressing the complexities associated with non-IID and heterophilic graphs (Liu et al., 2024a). Despite its potential, the development of FedGNNs that effectively handle these challenges remains relatively under-explored.
The non-llI nature of graph data complicates the training process in federated setting, traditional FL algorithms are designed for IID data and may not perform optimally with graph data (He et al.) 2021 Zhang
et al. 2023). Real-world applications often involve complex graph structures where data points are non-IID, and relationships are heterophilic. Traditional FL methods may struggle with convergence, communication overhead, and privacy risks in these settings. Therefore, there is a need for efficient solutions that can handle such complexities effectively while preserving privacy and optimizing bandwidth usage.
This work addresses the challenges of federated learning on graph-structured data, particularly focusing non-lID scenarios involving both homophilic and heterophilic graphs. We propose method that leverages spectral GNNs and neural ODE (Xhonneux et al.) 2020 Yin et al. [0244 Rusch et al. (022) to enhance the capture of complex graph relationships and dynamics. Our approach demonstrates superior performance in non-IID settings for both graph types, while also maintaining competitive results on IID data compared to state-of-the-art hoth Tt focuses on optimizing communication costs and preserving data privacy, making well-suited for real-world federated learning scenarios. The design ensures privacy preservation and bandwidth efficiency, which are important for practical applications where complex, non-IID graph structures are common. Our initial motivation slemined from these challenges, which led us to explore how GNODEFormer could be adapted to work effectively in federated settings. Notably, while our approach is tailored for federated learning, its design is versatile enough to perform well even in centralized settings, showcasing its broad applicability to various graph learning scenarios. Key contributions of our work include:
We propose novel GNN architecture for centralized settings, called GNODEFormer..
We extend this architecture to a FL setting, introducing Fedl-GNODEForner, designed to handle both homophilic and heterophilic graphs under non-lID data conditions.
Our empirical results demonstrate enhanced performance non-IID data and competitive results on ITD data when compared to state-of-the-art methods.
We develop prreeccirreerving and Bandwidth-efficien strategies to enable federated learning for graph-structured data across various types and data distributions (see A
The remainder of this work is structured as follows: Section 2 reviews related work on federated learning and GNNs, identifying gaps that our approach addresses. Section 3 details our proposed method, explaining the use of spectral GNNs and neural ordinary differential equations. Section g outlines the experimental setup, detailing the datasets and evaluation metrics, presents the results by comparing our mothod's performance with existing approaches, and discusses the implications of our findings along with the limitations of the study. Finally, Section concludes the paper and suggests potential ial directions for future research.
2

---

Published in Transactions on Machine Learning Research (04/2025)
2
Related Work
Data privacy and control have become significant concerns in modern machine learning, particularly due to the vast amounts of data required for deep learning (Shokri & Shmatikov 2015 Albrecht 2016). Recent works (Ju et al.) 2024) highlights key limitations in deploying GNNs under practical conditions, including class imbalance, noisy labels, privacy constraints, and out-of-distribution (OOD) generalization. Federated learning (FL) has emerged as an innovative solution, allowing decentralized model training across multi- ple devices while preserving data privacy. Simultaneously, Graph Neural Networks (GNNs) have gained prominence for their ability to model complex relational data. The combination of these fields, Federated Learning in Graph Neural Networks (FI-GNN), offers both new opportunities and challenges in addressing data privacy concerns while effectively modeling heterogeneous graph data.
GNNs are categorized into message-passing neural networks (MPNNs) and spectral-based GNNs (Otega et al. 2018: Dong et al.] 2020: Wu et al. 2019 Zhu et al. 2021). MPNNs, like Graph Convolutional Networks (GCNs) (Kipf & Welling 2017) and GraphSAGE (Hamilton et al.) 2017a), propagate information between neighboring nodes. Graph Attention Networks (GAT) (Veličković et al. 2018) extended this by using attention mechanisms to dynamically weight neighboring nodes, further refining the learning of node representations. In contrast, Spectral GNNs use the graph Laplacian's spectral properties in the frequency domain (Singh & Chen 2023). GCNs are also classified as spectral methods, employing a simplified spectral convolution. ChebNet (Defferrard et al. 2016) uses Chebyshev polynomials to approximate the graph Fourier transform, which reduces computational demands.
Recent advancements like Specformer (Bo et al. 2023 have integrated spectral GNNs with transformer architectures to effectively capture both local and global graph characteristics. Specformer leverages the standard transformer encoder design, which, while effective, may fall short in modelling the continuous feature transformation (information propagation) across graph structures. Our approach addresses this through higher-order Ordinary Differential Equations (ODEs). In contrast to conventional transformers that model information flow using distinct layers, Neural ODEs offer a more fluid and mathematically solid depiction of continuous processes, as recent interpretations suggest that transformers can be viewed as discrete Neural ODEs (Hashimoto et al. 2024 Building on this insight, our approach incorporates higher- order Ordinary Differential Equations (ODEs), such as second- and fourth-order Runge- Kutta methods, to capture richer dynamics in feature propagation. By utilizing Neural ODEs, our model demonstrates improved adaptability to heterophilic and extreme non-III settings, where traditional transformers often struggle.
Recent advancements have explored the integration of continuous-time dynamics into graph learning. Zhuang et al. (2020) models node evolution using neural ordinary differential equations, while PDE- GCN (Eliasof et al. 2021) draws inspiration from partial diffcrential equations to improve feature propagation across graphs. These models demonstrate the power of continuous formulations in enhancing representa- tion learning but are developed for centralized settings and do not consider distributional heterogeneity. Complementing this, Ju et al. (2024) provide a comprehensive survey of practical challenges in GNN de- ployment, such as class imbalance, noise, privacy, and out-of-distribution generalization. Our work builds these directions by integrat ing a neural ODE mechanism within a federated GNN architecture eexpiicc explicitly addressing the challenges of non-lll) data and privacy in real-world settings, which remain underexplored in prior inlffrrentaa--qquutioo----ssssss GNNs.
A first-order ordinary differential equation (ODE) problem is defined by an equation involving a first-order derivative and an initial condition. These equations often lack closod-form solutions and are typically solved using numerical methods. Euler's method, the simplest numerical ODE solver, discretizes the time derivative via a first-order approximation (Ascher & Petzold 1998). This temporal discretization forms the basis of residual connections/nettene /ntwwrrss as first proposed by (Weinan 2017). Subsequent works, such as (Lu et al. 2018
Chen et al. 2018 Zhu et al.] 2022 Eliasof et al.) 2021), demonstrated that any parametric ODE solver can be interpreted as a deep residual network with infinite layers, optimized through backpropagation
Rumelhart et al.
1986).
Our work aligns closely with (i et al. 2021). which explores connections between transformers and ODEs. They propose that a Transformer's residual block can be viewed as a higher-order ODE. solver, leading
3

---

Published in Transactions on Machine Learning Research (04/2025)
to the ODE Transformer architecture inspired by the Runge- Kutta method. Similarly; (Lu et al. 2019 also interprets transformers as numerical ODE solvers for convection-diffusion equations in multi-partich particle systems, leading to architectural modifications such as relocating self-attention layers. However, (Li et al. 2021] highlights performance degradation due to error accumulation in stacked first-order ODE blocks and addresses this by introducing high-order blocks, resulting in notable BLEU score improvements.
FL has progressed significantly in recent times. McMahan et al's work on Federated Averaging (FedArg) (McMahan et al. 2017) established the field's foundation, addressing privacy issues and reducing centralized data storage needs. Recent research has tackled distributed data privacy through methods like DARLS for learning causal graphs from distributed data (Ye et al.) 2024). FL has emerged as privacy-preserving solution for on-device learning, balancing communication costs and model accuracy (Pfeiffer et al.) 2023). Researchers have also developed FL algorithms for data streams, improving performance on various machine learning tasks (Marfoq et al., 2023). For heterogeneous edge computing, efficient mechanisms like FedCH (Wang et al., 2023) have been introducod, reducing completion time and network traffic. These studies [Liu et al.] 2024a) collectively advance FL's applicability in privacy-prese preservig collaborative learning and distributed model training.
Combining FL with GNNs (FL-GNN) creates a novel paradigm that uses the strengths of both fields. GNNs are good at handling graph data, common in areas like social networks and biology (Hamilton et al.] 2017b) However, centralized training of GNNs raises concerns about data privacy (Albrecht) 2016) Cui et al. 2024 regulatory compliance, and logistical issues, particularly when handling sensitive information (Luu et al. 2024cp. Federated learning helps by allowing decentralized training while keeping data local. GeaphNNNNN (He et al.] 2021 is FL-GGNN method that helps train GNNs in federated way and provides way to test different GNN models and datasets. It shows the challenges of federated training with non-llD data, where data differs across clients. Standard FL methods like FedAvg may not work well for GNNs, suggesting a need for better techniques. Federated Graph Convolutional Networks (FedGCNs) (Yao et al., 2023) apply Graph Convolutional Networks to federated learning. They handle graph data challenges through local training followed by global updates (Liu et al. 2024b). This improves privacy and works well with large datasets. However, FedGCNs often perform poorly with extreme non-lID data and struggle to combine updates from different graph structures. Thus, our aim is to build method that addresses issues of both heterophilic graphs and non-IID data. Our work aligns with this direction by explicitly addressing two of these challenges: non-IID data distributions and privacy preservation in federated settings. Through the design of GNODEFormer and its federated extension, we contribute towards building GNN architectures that are both robust to distributional heterogeneity and mindful of privacy constraints in real-world deployments.
3 Our Method
In this section, we introduce our proposed model, a novel neural network architecture that integrates a transformer-basse framework (Vaswani) 2017) with Ordinary Differential Equations (ODEs) (Runge 1895] Kutta 1901 Butcher 1996 Ascher & Petzold 1998) for node classification both homophilic and het- erophilic graphs. We begin by presenting preliminary information on semi-supervised node classification in centralized and federated settings, emphasizing the importance of capturing dependencies to generate meaningful representations.
Let 9 = E)) represent a global graph dataset, where V is the set of nodes with |V] = n, and & is the set of edges. For this work, we assume that G is undirected. The graph associated with an adjacency matrix A 0, 1=xn, where Aij = if an edge exists between node and node j, and Aij = 0 otherwsse. In semi- supervised node classification, subset of nodes, l C V, is labeled, while the remaining nodes lack labels. The objective is to predict the labels of these unlabeled nodes using a trained model. In our case, this model is GNODEFormer (Graph-Neural-Ordinane
as described in Section 3.1
To effectively model and analyze graph-structured data, the graph Laplacian plays a crucial role. In Graph Signal Processing (GSP), the graph Laplacian defined as L D A, where D is the diagonal degree matrix with entries Dii = jjj for all i V, and Dij = 0 for i + j. Similarly, the normalized graph Laplacian is expressed as L = I, D- AD 2, where I, denotes the n identity matrix.
4

---

Published in Transactions on Machine Learning Research (04/2025)
extend GNODEFormer to the Federated Learning scenario. Specifically, begin by partitioning the global graph dataset, 9, into m disjoint subgraphs, Gi = (Vi,&i). To achieve diverse and representative distribution of nodes and edges across the m subgraphs, we employ a Dirichlet distribution during the partitioning process. Each subgraph is then assigned to a unique client, ensuring that the partitions remain disjoint.
In Section 3.2
3.1 GNODEFormer
In this section, we introduce our proposed model, (GNODEFormer)) and discuss its components and functioning. GNODEFormer centralized architecture designed capture continuous data dynamics and retain complex dependencics graph- learning (asks. The model leverages ODE-inspired mechanisms, incorporating second-order and fourth-order Runge- Kutta methods (Li et al.) 2022 to refine the outputs of transformer layers. These mechanisms enable GNODEFormer to model the temporal and structural evolution of graph data effectively.
To further enhance learning, we utilize the Residual Layer History mechanism, which accumulates and normalizes intermediate layer outputs. This mechanism allows the model retain crucial information across layers, improv- ing its ability to capture intricate dependencies within the graph structure.
Elgen Decomposition
Residual Layer History
Elgen Encoding
Finally, the decoder reconstructs the graph Laplacian ma- trix from the refined representations. This enables the model to perform spectral graph convolution, ensuring that the learned embeddings align with the graph's 's spec- tral properties and effectively encode its structure.
Decoder
Residual Layer History
Graph convolution
By keeping its essential elements Eigen Decomposition and Figenencoding for spectral analysis- GNODEFormer expands upon Specformer. In order to better handle comn- plex graph relationships and heterophilic//mme (oniii/nonnIIDD data, it introduces Neural ODE layers for continuous transforma- tions, a Residual Layer History mechanism for normal- izing outputs, and enhanced convolution that combines Neural ODEs with spectral methods.
Linear
Figure 1: GNODEFormer architecture: Integrat- ing ODE-inspired Runge- Kutta methods, Resid- ual Layer History, and spectral graph convolution for enhanced graph representation learning.
3.1.1 Eigen Decomposition
Eigen decomposition plays a crucial role in analyzing the graph Laplacian spectra, which provides valuable frequency-related insights into the structure of the graph. Specifically, the normalized graph Laplacian C is real symmetric matrix, and its eigen decomposition be expressed
C L=UM
Here, UI contains the eigenvectors of C, and V diag(In; 72. 73 ... 7w]) I. is the diagonal matrix of correspond- ing eigenvalues, which lie in the range [0, 2]. The eigenvectors in U = [ui, u2, u,,..., [nd] um]] are orthonormal, meaning that each eigenvector uy has a unit norm (I|u,]1 = 1), and the dot product u² Uy = 6,y, where d, is the Kronecker delta function, equal to if i j and 0 otherwise. This orthonormality essential for preserving the graph's structural information during spectral transformations.
3.1.2 Eigenvalue Encoding
We leverage the transformer mechanism to design a powerful set-to-set spectral filter that utilizes both the magnitudes and relative differences of the eigenvalues. To overcome the limitations of using scalar eigenvalues directly, which would constrain the expressiveness of self-attention, we use the eigenvalue encoding functions and to transform each scalar eigenvalue into rich vector representation, F(()) R!. - Rd.
5

---

Published in Transactions on Machine Learning Research (04/2025)
The eigenvalue encoding functions are defined as:
XXX 1000022/8
E(y, 21) = =sin
(1)
F(7, 24 +
cos
X
(2)
100002:/d
In the above equations, i is the dimension of the vector representation and e is hyperparameter. Although the eigenvalue encoding (EE) is similar to the positional encoding (PE) of Transformer, they act quite dif- ferently. PE describes the information of discrete positions in the spatial domain. This motivates us to use the Ordinary Differential Equation to construct the further experiments. Eigenvalue encoding identifies the changes in frequency between eigenvalues and generates vector and multi-scale representations. These repre- sentations provide important insights into the eigenvalues within the spectral domain. Then, the eigenvalues are concatenated with their eigen encoding, 2 = I> (elll)(()))))))))))))))) 7a ralle((2))..., In [|(()))) Rnx(d+1), The result is then passed onto the ODE Transformer block which learns the dependency between the eigenvalues.
3.1.3 Ordinary Differential Equations and Layer History Mechanism
Our method incorporates ODE-inspired layers within its architecture, specifically utilizing Rungo-Kutta integration methods such RK2 and RK4 (Li et al., 2022 These methods iteratively refine the outputs of the network's layers, enabling continuous refinement of the model. This process enhances the model's ability to capture the dynamics of data flow, leading to more accurate and stable representations of complex data patterns.
The ODE integration process within each transformer layer can be described by
2 +
Σ wife In
Ligki
(3)
where 20) is the input to the layer, (0+1) is the output of the layer, p is the order of the Runge-Kutta method (2, 3, or 4), fo represents the neural network layer function with parameters 0, w; are the weights for combining each k; term, ai are the coefficients for the intermediate steps, K; are intermediate values representing estimates of the change in x over a step, and kij are the previously calculated k values used in computing the current kg.
This process refines the layer outputs to capture more subtle patterns. Following this, the model introduces the Residual Layer History mechanism, which deviates from conventional transformer models. Instead of passing the outputs directly between layers, the model accumulates and normalizes the outputs across layers, preserving history of previous states. This accumulated history is then used to iteratively compute the model's next state, allowing for more muanced feature representations. The residual update is given by:-
In xn-11 + yn-1
(4)
where Fn represents the output at the current layer n, xn-1 the output from the previous layer n 1, and Un-1 the residual term added at layer n
The use of Neural ODEs in FodDErroe is motivated by their ability to model continnous feature propagation) across graph structures. Unlike transformers, which approximate this process in discrete layers, Neural ODEs offer principled framework for continuous modelling, sup- ported by recent. interpretations of transformers as discrete Neural ODEs (Hashimolo et al.) 2024). This continuous framework enables enhanced representation learning, particularly for non-lID and heterophilic graphs, by scamlessly integrating with spectral GNNs. Additionally, Neural ODEs' adaptive solver strate- gies improve computational efficiency, addressing the memory and processing challenges typical of federated graph learning frameworks.
transfornationniffomaate

---

Published in Transactions on Machine Learning Research (04/2025)
3.1.4 Decoding and Graph Convolution
The representations and the continuous dynamics of the spectra, learned in the ODE stage, are passed to the decoder to predict new eigenvalues, denoted as "n Decoder(22) ()). These eigenvalues, generated by the TDE-sTrasformer block, are refined through a spectral filtering mechanism. Using these refined eigenvalues, the individual bascs of the graph Laplacian reconstructed Luew Udiag(")))44
The reconstructed Lnew is then processed through fully connected layers equipped with residual connections. These layers, along with self-attention and ferd-forward sub-layers, produce a new learnable base, L. This base is used to assign a separate graph Laplacian matrix to each feature dimension, enabling the model to perform spectral graph convolution effectively.
3.2
FordEeeDDFrrmer
conv
Graph
Decoder
Decoder
Graph conv
Decompose
Eigen Decompose
Linear
Eigen
Parameters
Parameters
RHODE Block
Elgen Encode
Eigen Encode
RHODE: Block
A
ODE Blook
Parametera
Timee mmm min
ODE B0E-Block
Residual History
Residual History
Rosidual History
Residual History
88
Send Paramatera
Send Parameters
2
Block
RHODE
Eigen Encode
Eigen Encode
RHODE Block
6
Eigen Decompose
Decomposs
Linear
Eigen
3
Receive
Graph
Decoder
Paramotors
Decoder
Graph
conv
Figure 2: Foo-Frooerrr:er: GNODEFormer in a Federated Learning setup. The diagram illustrates multiple clients, each with a local model, interacting with a central server. The numbered arrows represent the standard FL workflow: (1) the server sends the global parameters, (2) clients perform local updates, (3) clients send the updated parameters back, and (4) the server aggregates them. Arrows indicate the flow of parameters and updates for explanation purposes and do not imply a directed graph structure.
In this section, we introduce our Federated Graph Neural Ordinary Differential Equation transFormer,, abbreviated as FeO-FrooDerrreer, and discuss the performance of GNODEFormer in decentralized machine learning scenario.
The global graph G is partitioned into disjoint sub-graphs, Gi (V,,&), which are then allocated to set of clients, C {c1, C22 cs...., Cx). The goal of Federated Learning (FL) is to minimize the global loss function across all clients while keeping the graph data decentralized. Each client a; E C trains GNODEFormer locally on its assigned sub-graph, G.-
A significant challenge in FL arises from the non-iid nature the data, as the graph partitions across clients are often imbalanced or skewed. This heterogencity increases the complexity of optimizing the global loss function compared to centralized learning. Due to this imbalance, clients may converge to local minima, where the model performs well the client's 's local data but poorly the global task. These local minima

---

Published in Transactions on Machine Learning Research (04/2025)
can vary significantly between clients, particularly when the data distribution exhibits high degree of non-iid-ness.
The global optimization task in FL is to aggregate these locally trained models effectively to minimize the global loss function. GNODEFormer demonstrates robust performance in such heterogeneous scenarios, excelling even when the data concentration parameter (a) of the Dirichlet Distribution as low as 0.01, indicating a high degree of non-iid-ness
3.2.1 Federated Training
Subgraph sampling for federated training involves partitioning the global graph G into m subgraphs, G; E,), which are then distributed among a set of clients C. This partitioning uses a Dirichlet distribution ensure a diverse and representative distribution of nodes and edges, effectively simulating non-IID data conditions. After partitioning of the graph into various subgraphs m and assigning them to the client set C. For each client C E C, we now proceed to perform the eigen decomposition on the adjacency Ae, of the local subgraph. This step plays a crucial role of capturing information in the high non-iid local graph m of the client c₁. Post the eigen decomposition step, each client will have its own eigenvalue set, 7/m and eigenvectors Um, This can be analyzed as:
Ley = Umm sb Umn where vb diag(( 7omg Ymag Yims D.
(5)
Next, we perform the eigen encoding using (1) and (2) to capture the relative frequency shifts of eigenvalues and provide high- tign-dimensional vector representations.
eYm
E(Tim: 26) sin
100002374
(6)
etim
E( 2i + 1) cos
1000020/4
After this step, the eigen values are concatenated with their eigen encoding Zm (tim [Ip(Tim); Tmg (lgl(((m))) (.,.),... Ym. (lll(lmmm)) ),²= where Ym; represents the ith clients (c: local eigenvalue belonging to the local eigen set Tm- The result is then passed on to the ODE inspired Transformer model to learn the dependencies between the eigen values. Here, we use the High-Order ODE solvers like Runge-Kutta methods to capture the continuous dynamics in the spectral domain. We taken second-order Rungo-Kutta and fourth-order Runge-Kutta mothods into consideration. In addition to this, we are also maintaining the residual layer history, as in, we preserve the history of the previous states. This result is then used to iteratively compute the model's next state. Then the result is passed onto the decoder block to filter the new eigenvalues, Tim; and learn them. This new eigen are then used to further enhance the model's ability to capture the spectral characteristic of the local graph. Then we go about reconstructing Finally, we perform spectral graph convolution. The spectral graph convolution is written as f Rg x = Um.go (diag((mm)))
the individual bases Lue; = Um, diagg/nnmm)aaee
This result is then passed onto the fully connected layers equipped with local residual connections to construct new learnable base L.. This enables the local model to effectively learn the node representations.
The server is first. assigned with Wglobat and transmitted to all the participating clients. This feature of controlling the participating clients is controlled by a parameter called fraction fit. This can be viewed as S C.K. where C the client set and K. is the fraction fit.. The participating client divides the sub-graph m. into the respective training set and testing set. It can be analysed as frain (vtrain efrain) and moost= (v/est, ( The objective is to generate the labels to the unlabeled vertices. The clients undergo their own local model training process using the GNODEFormer on the local sub- mub-graph m, to minimize the local loss. Once the local model training process is completed, the weights from the participating clients S is sent back to the model to perform the server aggregation process. For server aggregation we choose to go with the standard FederatedAveragingg algorithm, where all the data from the participating client set S undergoes weighted averaging. Here, the weights are proportional has, ensuring the greater influence of the client with more data on the global model. This process is repeated for T times until the global model's loss function is minimal.
the number of data points each client
8

---

Published in Transactions on Machine Learning Research (04/2025)
3.2.2 Algorithm
Algorithm Fol-GNODDFoomer Algorithm
Input: T, t, C, 0
Initialise 88
do 6: 0 + U-pllleCClienl(R
for each round R = 0,,1,,33 Sample Devices S C
5: for cach client 8; S in parallel do
7: end for
8: 0 + - 00
end for
10:
11: procedure UPDATECLIENT(RR, 0²):
12:
Cup + --NernaisssCCmppllapppiiiiiiiii. A, D)
13: Vo. U2 + EigenDeccomposiiion C..)
14: for each round r= 0, 2, ...,, 3, do
15:
Da new + GNODEFormer(2,, Us, Ba,)
7.. 8%, + arg min (ū,, [train,,] labels,, ,))
16:
00
17:
end for
18: Return R
19:
end procedure
4 Experiments
In this section, we conduct wide variety of experiments on various heterophilic and homophilic graph datasets to verify the effectiveness of our approach. We compare our model against several state-of-the-art models including FedGON (0 Hop, 1 Hop, 2 Hop) and FedSAGE, to demonstrate the improvements model offers in different heterophilic and homophilic graph settings.
4.1 Dataset
For our experiments, we selected diverse set of standard heterophilic and homophilic graph datasets evaluate the robustness of our approach across different scenarios. The Cora and Citesoer datascts represent citation networks, and the Photo dataset an Amazon co-purehase graph, all exhibiting homophilic properties. In contrast, the Chameleon and Squirrel datasets are heterophilic Wikipedia networks, while the Actor dataset corresponds to film-related network. Each dataset includes nodes (representing distinct entities), edges (denoting relationships). node features, and class labels specific to their respective domains. Detailed information and summaries of these datasets are provided in the appendix.
4.2 Non independent and identically distributed Data
In FL-GNNs, non-IID data poses significant challenge due varying distributions across clients. We simulate these conditions using label Dirichlet partitioning, which controls distribution heterogeneity. The concentration parameter (a) of the Dirichlet distribution determines data skewness: higher values (>= 1) create near-IID settings, while lower values (< 1) result. in highly skewed distributions, mimicking real-world non-IID scenarios.
In highly non-lID settings (low a), local model updates can diverge significantly, leading to slower conver- gence, higher communication overhcad, and potential model biases due imbalance. A moderate balances non-IID characteristics with learning efficiency, crucial for developing robust FL algorithms under realistic data heterogeneity.

---

Published in Transactions on Machine Learning Research (04/2025)
Our proposed architecture is specifically designed to address these non-IID conditions in FL settings effectively than existing methods, achieving better convergence and generalization performance.
4.3 Experimental setup
We conduct our experiments using setup with 5 clients and a centralized method to evaluate our model's performance in both federated and centralized settings. To distribute data among the clients, we use a Dirichlet label distribution with parameter a for all datasets. This approach allows us to assess the model's robustness and effectiveness under different data distribution scenarios, reflecting realistic federated learning environments.
In this setup, we compare the performance of the State-of-the-ar Spectral Graph Transformer integrated with Neural Ordinary Differential Equations (GNODEFormer), We experiment with different ODE methods, specifically Runge-Kutta-2 (RK-2) and Runge-Kutlan (RK-4), across various datasets. We then compare our model's accuracy against existing state-of-the-ar federated learning models, such as FedGON with O-Hop, 1-Hop, and 2-Hop configurations, and FodSAGE+ (Zhang et al. 2021) on both homophilic and heterophilic
graph datasels.
4.4 Results
We present the performance evaluation of our proposed architecture, GNODEFormer, in both centralized and federated settings.
Table Accuracy comparison of different models on heterophilic and homophilic datasets.
Model
Heterophilic Squirrel
Homophilic
Chameleon
Actor
Cora
Citeseer
Photo
GCN
59.6142.21
16.78-40.87
33.2311.16
87.14+1.01
79.86:10.67
88.26+0.73
GAT
63.13++1.93
14.9--0.88
33.93+-2.17
88.03:40.79
80.52:10.71
90.9114.68
JacobiConv
74.20=#1.03
57.3841.25
41.17=0.64
88.98:10.48
80.7810.79
95.4311.22
Graphormer
54.49=t3111
36.96+1.75
18.45=1.38
67.71+0.78
73.30#1.21
85.20=4.12
Specformer
74.72:11.29
64.64+0.81
41.93+1.04
88.57+1.01
81.49=0.94
95.48+0.32
GNODEFormer (RK-2)
79.04=00229
66.26+0.266
45.15=0.110
88.30+00.05
83.85=00115
95.94+0.02
GNODEFormer (RK-4)
79.41=00.02
66.65=0.03
44.56 + 0.39
89.40=0.228
04.10::0.12
95.65 + 0.02
Table 2: Accuracy results for Homophilic non-III for 5 clients. Bold is the highest accuracy while underlined is the second highest.
Homophilic Graphs Citeseer
Dataset
Cora
Photo
0.01
0.1
0.01
0.1
0.01
0.1
FaIGON
87.73%
$1.87%
-1.845
0.49%
70.03%
2.42%
16.78%
10.43%
34.53%
4.39%
FaIGON
82.43%
0.54%
61.67%
+ 0.00%%
70.60%
+ 0.92%
70.10%
0.57%
17.12%
15.96%
36.08%
3.33%
FalCCN
81.17%
0.33%
61.03%
0.00%
68.83%
+ 1.23%
69.47%
0..6%
18.29%
14.28%
12.80%
18.08%
FeISAGE+
87.83%
0.61%
$1.80%
1.61%
74.73%
0.82%
69.63%
1.89%
16.19%
16.36%
23.46%
16.88%
Fal.CNNODEFomee(RR...
80.28% + 1.97%
$4.32%
0.23%
74.54%
4 1.41%
74.18%
1.91%
53.50%
4 0.45%
89.4676
4 10.27%
Rol-cN00DEFume((K-))
89.64% 0.93%
85.16%
+ 0.66%
75.18%
1.02%
74.22%
1.24%
96.76%
0.72%
94.94%
2.17%
Table Accuracy results for Heterophilic non-ITD for clients. Bold the highest accuracy while underlined is the second highest.
Ulient=5
Heterophilic Grapis Squirrel
Dilaset
Chmmeloon
Actor
0.01
0.1
0.01
0.01
FelGON 0
$2.07%
10.36%
27.70%
8.422
222.50% + 4.11%
22.50%
+ 4.11%
27.61%
+ 2.03%
27.33%
6.05%
FedGON
26.31%
7.25%
2.98%
1.89%
20.81%
1.89%
22.30%
3.33%
2.72%
FedGUN 2
27.11% 1 7.96%
22.06%
4.54%
21.28%
4: 2.12%
21.28%
4: 2.12%
21.75%
4: 3.77%
23.47%
4: 2.51%
FedSAGE]
32.76%
8.66%
28. 19%
8.70%
22.50%
4.11%
22.30%
4.11%
27.61%
2.00%
27.33%
6.09%
65%,
0.76%
60. 78%
0.32%
53.12%
1.31%
48.74%
1.60%
11.51%
28%
50.63%
1: 23.36%
70.59%
4 1.90%
64.22%
1. 3.21%
53.72%
J 5.53%
45.11%
4 0.41%
44.42%
20.91%
41.63%
18.01%
10

---

Published in Transactions on Machine Learning Research (04/2025)
On heterophilic datasets(Table 1. GNODEFormer achieves state-of-the-art results, outperforming existing models by notable margins on Chameleon (79.41%) and Squirrel (66.65%) datasets. For homophilic graphs, it shows competitive performance, achieving the highest accuracy on the Photo dataset (95.94%). In feder- ated learning with extreme non-IID conditions (a = 0.01), GNODEFormer consistently outperforms other approaches across both homophilic and heterophilic graphs. Its strength is further demonstrated by main- taining competitive performance across varying levels of data heterogeneity, as shown in 4.5 The model's success can be attributed to its innovative architecture combining neural ODEs with transformer Vaswani 2017) enabling more flexible modeling of graph dynamics. This design proves particularly effective in het- erophilic and non-IID scenarios, with potential applications in social network analysis, recommendation systems, and fraud detection. This robustness in extreme non-lID settings is attributed to two core mech- anisms in Fed-GNNDDFormer'ss architecture: spectral graph methods and Neural ODEs. Spectral GNNs leverage the graph spectrum, enabling each client to learn global features rather than being restricted to local neighborhoods(Bo et al., 2023). This mitigates the impact of heterogencous data distributions by learn- ing patterns that generalize across clients. Neural ODEs further enhance this adaptability by dynamically each client based its data complexity, allowing personalized yet aligned
adjusting the computation updates to the global model(Chen et al. 2018). Together, these mechanisms ensure Red-GGNODEFormet achieves better accuracy and convergence across clients, even under extreme non-IID conditions. However, GNODEFormer faces challenges in convergence speed and computational requirements, especially for larger heterophilic graphs, See Appendix D for scalability results. Future work should focus on improving effi- ciency and scalability while maintaining the model's strong performance across diverse graph types and data distributions.
4.5 Ablation
Table 3a shows the performance of our architecture across different values. The model achieves its highest accuracy at 1.0 (66.05%) and its lowest a 0.5 (62.87%), with low standard deviation across all scttings. These results demonstrate the robustness of our approach, maintaining stable performance even in non-IID scenarios.
To further understand the contribution of individual components to robustness, we conducted an ablation study (Table ! by evaluating two variants: (i) Fed-GNODDEormer without the Neural-ODE Block, high- lighting its role in adapting to non-IID data, and (ii) without the Residual Layer History, showing its impact on normalizing intermediate representations.
The results from these ablations underline the synergistic role of Neural-ODEs and Residual Layer History in achieving superior performance, particularly in challenging non-llI scenarios.
Avg (%)
STD (%)
0.1
64.71
0.53
500.009
0.2
64.32
1.25
0.3
63.74
0.39
75,9%%
0.4
64.67
0.35
0.5
62.87
0.20
90.955
0.6
64.58
0.28
0.7
64.00
0.77
0.8
64.23
0.22
0.9
65.21
0.54
1.0
66.05
0.20
(a) Model Performance Across Different & Values.
(b) Model Performance on varying IID values.
Figure 3: Comparison of Model Performance on chameleon. The table (left.) shows the average accuracy and standard deviation the model across different a values for rk-2 model. The adjacent graph (right) provides visual representation that. further illustrates the trend in model performance, offering a clearer understanding of how the model adapts varying data distributions.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Component-wii wise ablation of Fed--erDD We have compared the (1) and (2) with the original results of the FoooFrroeerrmer for the varying non-iid a values.
Methods
Homophilic Citeseer
Heterophilic
Dataset
Squirrel
0..11
0.05
0.01
0.05
Foor Without the Neural ODE block
73.56%
73.1%
71.2%
43.22%
42.26%
40.44%
Foo-Goooeerrrrr (RK-2) without Residual Layer
73.98%
74.01%
73.54%
51.26%
43.94%
42.18%
GooDEFoeeerrerr (RK-4) without Residual Layer
74.85%
74.36%
74.18%
52.13%
44.37%
43.67%
Ted--GrODEFormer (RK-2)
74.54%%
74.58%
74.18%
53.32%
46.78%
43.74%
Ted--GroDEFormer (RK-4)
75.18%
74.44%
74.22%
53.72%
46.22%
45.11%
The component-wise ablation results of Fed-GGNODEFormer on CiteSeer (homophilic) and Squirrel (het- erophilic) under different non-iid levels (a) are shown in Table Performance is severely reduced in all settings when the Neural ODE block is removed, underscoring its significance, especially on heterophilic graphs like Squirrel. The residual layer's function in stabilizing learning is suggested by the smaller but consistent negative impact when it absent. RK-2 performs marginally better on heterophilic datascts, but RK-4 typically beats RK-2 homophilic ones. Performance deteriorates increases, particularly for heterophilic graphs, highlighting the difficulties caused by high non-iidness.
5 Conclusion
In this work, we present GNODEformer, novel federated learning method for Graph Neural Networks (GNNs) that effectively addresses the challenges of training non-lID data in both homophilic and het- erophilic graph settings. Our approach combines spectral GNNs with neural ordinary differential equations (ODEs) to enhance information capture and model adaptability across diverse graph types. The results show that GNODEformer significantly improves performance on non-IID heterophilic graphs and achieves competitive accuracy on homophilic graphs and IID data, oulperforming several stale-of-the-arl methods. For future work, there are two promising directions. The first is developing a unified model that can seam- lessly handle both IID and non-lll data, enhancing the flexibility and applicability of GNODEformer. The second is improving the runtime efficiency of federated learning setups, especially in large-scale and resource- constrained environments, to further optimize our approach.
12

---

Published in Transactions on Machine Learning Research (04/2025)
References
Jan Philipp Albrecht. How the gdpr will change the world. Eur. Data Prot. L. Rev., 2:287, 2016.
Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and differential- algebraic equations. SIAM, 1998.
Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural nelworks meet transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview.oo http:///.oooomrmoooot0033o/o3//
J.C. Butcher. A history of runge-kutta methods. Applied Numerical Mathematics, 20(3):247 260, 1996. ISSN
0168-9274. doi: https://ddoi.org/10... /t..oor///10/100000 1016000..11...0100100000 11110001161011-1111 /01///01119919// (1168-9274(95)0000-55
com/
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential oquations. Advances in neural information processing systems, 31, 2018.
Nan Cui, Xiuling Wang, Wendy Hui Wang, Violet Chen, and Yue Ning. Equipping federated graph neural networks with structure-aad aware group fairness, 2024. URL https://arxix: otgg/bs/////// 12350
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. In D. Lee, M. Sugiyama, U. Luxburg, 1. Guyon, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: httpas://poceeedngege wwwpeeedinnn.uuriss.o.ooo neurips. cewp.ps.ccppppperrrrr http:p/s////pooroooo/////////// ///es///ppe//11///ii 1--65- Papp-pparr.ee
Xiaowen Dong, Dorina Thanou, Laura Toni, Michael Bronstein, and Pascal Frossard. Graph signal processing for machine learning: A review and new perspectives. IEEE Signal processing magazine, (776):1177177, 2020.
Moshe Eliasof, Eldad Haber, and Eran Treister. Pde-gen: Novel architectures for graph neural networks motivated by partial differential equations. Advances in neural information processing systems, 34:3836- 3849, 2021.
Xingbo Fu, Binchi Zhang, Yushun Dong, Chen Chen, and Jundong Li. Federated graph machine learning: A survey of concepts, techniques, and applications. ACM SIGKDD Explovations Newsletter, 24(2):32-47 47, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL https:// http///ooeeinnn.eeeeee neurips. wewrips.cc/paper files//papptt2/1//fill 5dFd5503aas0fbbbaaccrrrreoeoeea Pappr..
William Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint (533::17990058/// 2017b.
Koji Hashimoto, Yuji Hirono, and Akiyoshi Sannai. Unification of symmetries inside neural networks: Trans- former, feedforward and neural ode. Machine Learning: Science and Technology, 2024.
Chaoyang He, Keshav Balasubramanian Emir Ceyani, Carl Yang, Han Xie, Lichao Sun, Lifang He, Liangwei Yang, Philip S. Yu, Yu Rong, Peilin Zhao, Junzhou Huang, Murali Annavaram, and Salman Avestimehr. Fedgraphnn: A federated learning system and benchmark for graph neural networks, 2021.
Wei Ju, Siyu Yi, Yifan Wang, Zhiping Xiao, Zhengyang Mao, Hourun Li, Yiyang Gu, Yifang Qin, Nan Yin, Senzhang Wang, et al. A survey of graph neural networks in real world: Imbalance, noise, privacy and ood challenges. Xix preprint 0xXi:2403.04466 2024.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Represenlations, 2017. URL SJU4
Wilhelm Kutta. Contribution to the approzimate integration of total differential equations. Teubner, 1901.
Bei Li, Quan Du, Tao Zhou, Shuhan Zhou, Xin Zeng, Tong Xiao, and Jingbo Zhu. Ode transformer: An ordi- nary differential gquation-inspires model for neural machine translation. arXiv preprint X10:2104.023088, 2021.
Bei Li, Quan Du, Tao Zhou, Yi Jing, Shuhan Zhou, Xin Zeng, Tong Xiao, JingBo Zhu, Xuebo Liu, and Min Zhang. ODE transformer: An ordinary differential equation-inspired model for sequence generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8335- 8351, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (0.18653/v1, /ttp//11122222211///// aonno222211-11.551111 URL
org hwtgg//22..aarr aal-2...cllooon acl.longllllnnnn1 571
Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods.
volume 34, pp. 20887 20902, 2021.
Bingyan Liu, Nuoyan Lv, Yuanchun Guo, and Yawen Li. Recent advances on federated learning: A systematic survey. Neurocomputing, 597:128019, 2024a. ISSN 0922-2312. doi: https://doi. wttpg////1....0000 128019. URL /tstps::/www.ce http://wwwww.cccceeeeetccccc htt.://ccwc.cccccoooooooooooaota
Rui Liu, Pengwei Xing, Zichao Deng, Anran Li, Cuntai Guan, and Han Yu. Federated graph neural networks: Overview, techniques, and challenges. IEEE Transactions on Newral Networks and Learning Systems, pp. 1-17, 2024b, doi:
Yanjun Liu, Hongwei Lı, and Meng Hao. Personalized and privacy-preserving federated graph neural network. Frontiers in Physics, 12: 12:1383276, 2024c.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276 3285. PMLR, 2018.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie Yan Liu. Under- standing and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.Jm 027622 2019.
Othmane Marfoq, Giovanni Neglia, Laetitia Kameni, and Richard Vidal. Federated learning for data streams. In Francisco Ruiz, Jennifer Dy, and Jan-Willen van de Meent (ods.), Proceedings of The 26th Interna- tional Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learn- ing Research, pp. 8889 8924. PMLR, 25-27 27 Apr 2023. URI. marfoq23a.hta html
mlr. ptt..prress/v///////
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Imago-based recommen- dations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pp. 43 13-52, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336215. doi: 20.1145/276644222767773. URL https://doi.org. 10.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas Communication Eiffiient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (ods.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 20-22 Apr 2017. URL
https://proceeddnng: mlr. http:////g..........eee/////aaaa wttppees//.///maaaaaaaaa......
14

---

Published in Transactions on Machine Learning Research (04/2025)
Antonio Ortega, Pascal Frossard, Jelena Kovačević, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106(5):808 828, 2018.
Kilian Pfeiffer, Martin Rapp, Ramin Khalili, and Jörg Henkel. Federated learning for computationally constrained heterogeneous devices: A survey. ACM Comput. Sure., 35(145):33:113334) 2023. URL
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, (2)cnnab014, 2021.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, (23(608)):5333-555 536, 1986.
Carl Runge. Un the numerical solution of differential equations. Mathematical Annals, 46(2):167-178, 1895.
T. Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18888-18909, PMLR, 17-23 Jul 2022. URL attps://proceedings mlr. wwtpse/vv1//rusch2eeeeee html
T. Konstantin Rusch, Michael M. Bronstein, and Siddhartha Mishra. survey on oversmoothing in graph neural networks, 2023. URL httpp://aarrvv.orr//b//220...
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural nelworks, 20(1):61-80) 80, 2008.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference computer and communications security, pp. 1310-1321, 2015.
Rahul Singh and Yongxin Chen. Signed graph neural networks: A frequency perspective. Transactions on Machine Learning Research, 2023.
Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. Social influence analysis in large-scale networks. In Proceed- ings of the 15th ACM SIGKDD International Conference Knowledge Discovery and Data Möning, KDD '09 pp. 807-816, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584959, doi: 10. 415544001559911111
URL lhttps:///ol http:///ooooooo/./// 1ee..11141115700 0tt.///5557191..511188888
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URL https:
//openreview. wtw.neeeoooommmddrrXXpppppppp
Zhiyuan Wang, Hongli Xu, Jianchun Liu, Yang Xu, He Huang, and Yangming Zhao. Accelerating federated learning with cluster construction and hicrarchical aggregation. IEEE Transactions on Mobile Computing. 22(7):3805-388 3822, 2023. doi: 20.1009/TMC.2022.333 3177792..
Ee Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 155):1-11, 2017.
Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Tao Qi, Yongfeng Huang, and Xing Xie. federated graph neural network framework for privacy-preservigg personalization. Nature Commanications, 13(1))3091, 2022.
Felix WU, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861-6871. PMLR, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Louis-Pascal Xhomneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10432 10441. PMLR, 13 18 Jul 2020. URL https: //proceedings. mlr wwt.s//rsss/v/veeeeeee xhonneux20a. htmm
Yuhang Yao, Weizhao Jin, Srivatsan Ravi, and Carlee Noe-Wong. FedGCN: Convegennc-commmicatto
tradeofTs in lederated training of graph convolutional networks. In Thirly-sevenlle Conference on Newral Information Processing Systems, 2023. URL https:///ppnrreview wttt:/ooruriiooooysRBB...
Qiaoling Ye, Arash A. Amini, and Qing Zhou. Federated learning of generalized linear causal networks.
TEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-14, 2024. doi: 10.1109/TPAMI. 2024.3381860.
Nan Yin, Mengzhu Wan, Li Shen, Hitesh Laxmichand Patel, Baopu Li, Bin Gu, and Huan Xiong. Continuous spiking graph neural networks. arXiv preprinl. arXiv:2404.00 0188997 2024.
Kainan Zhang, Zhipeng Cai, and Daehee Seo. Privacy-preserving federated graph neural network learning on non-iid graph data. Wireless Communications and Mobile Computing, 2023(1):8545101, 2023. doi: https://ooi.org// 11511011155522//55555550 2023/8545101
URL hatpp:://omliinellbrrryy wiley..w........................ http:/ccc//cc//////a 1155/
Ke Zhang, Carl Yang, Xiaoxiao Li, Lichao Sun, and Siu Ming Yiu. Subgraph federated learning with missing neighbor generation. Advances in Neural Information Processing Systems, 34:6671-6682 2021.
Aiqing Zhu, Pengzhan Jin, Beibei Zhu, and Yifa Tang. On numerical integration in neural ordinary differ- ential equations. In International Conference on Machine Learning, pp. 27527-275:2 PMLR, 2022.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021, pp. 1215-1226, 2021.
Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, and James S Duncan. Ordinary differential equations on graph networks. 2020.
16

---

Published in Transactions on Machine Learning Research (04/2025)
A Appendix
A.1 Datasets
Dataset
Type
Nodes
Edges
Features
Classes
Cora
Homophily
2,708
5,429
1,433
7
Citeseer
Homophily
3,327
4,732
3,703
6
Photo
Homophily
7,650
11,081
745
8
Chameleon
Heterophily
2,277
36,101
2,325
5
Squirrel
Heterophily
5,201
217,073
2,089
5
Actor
Heterophily
7,600
33,544
932
5
Table 5: Surmmary of Datasets used for node classification
Cora: This is a citation network graph dataset often used in graph machine learning tasks, known for its homophilic properties. In this dataset, nodes represent documents, and edges represent citation links between them. Each node comes with a sparse feature voctor, representing the keywords in the document, and class label.
2. Citeseer: Similar to cora, this dataset is also homophilic citation network dalaset with chal- lenging classification task due to its sparse connectivity.
Amazon Photo a homophilic dataset and consists segments of the Amazon co-purchase graph (McAuley et al.) 2015), where nodes represent goods, edges indicate that two goods are frequently bought together, node features are bag-of-words encoded product reviews, and class labels are given by the product category.
Photo:
This dataset consists of nodes that represent articles from the English Wikipedia, with edges indicating mutual links between these articles. Each node has features based on the presence of specific nouns the article. The nodes are categorized into five classes according their average monthly traffic. Due to its low level of homophily, the dataset, poses more challenging classification problem (Rozemberezki et al. 2021
Chameleon:
5. Squirrel: Similar to Chameleon, this dataset is heterophilic and is part of a collection of datasets representing pagc-page networks on specific topics such as chameleons, crocodiles, and squirrels. these datasels, nodes correspond Wikipedia articles, and edges indicate mutual links bet ween them. Node features are derived from informative nouns appearing in the article text, while target labels represent the average monthly traffic for cach article from October 2017 to November 2018. The datasets include detailed statistics the nunber of nodes and edges (Rozemberezki et al. 2021).
This dataset represents the subgraph that includes only actors from a larger film-director- actor-writer network (Tang et al. 2200)). Each node represents an actor, and an edge between two nodes indicates that the actors appear logether on the same Wikipedia page. The features of each node are derived from specific keywords found those Wikipedia pages. The nodes are categorized into five groups based on the words in the actors' Wikipedia entries.
Actor:
A.2 Discussion on Results from Table (a = 0.05 across graphs)
In homophilic graphs (Cora, CiteSeer, and Photo datasets), GNODEFormer (RK-2 and RK-4) consistently outperforms the baseline models (FedGCN-0, FedGCN-1, FedGCN-2, and FedSage+).. Notably, on the Cora datasot, GNODEFormer (RK-2) achieves the highest accuracy of 87.52%, surpassing the next-best model (FedSago+) by margin of 2.25%. A similar trend observed on the CiteSeer and Photo datasets, where GNODEFormer (RK-2) and GNODEFormer (RK-4) provide the top performances, with GNODEFormer
17

---

Published in Transactions on Machine Learning Research (04/2025)
(RK-4) achieving the best accuracy of 88.69% the Photo dataset. These results reinforce the adaptabil- ity of GNODEFormer to homophilic structures, demonstrating its ability to leverage node similarity more effectively than the traditional federated GCN and GAT models.
For heterophilic graphs (Chameleon, Squirrel, and Actor datasets), GNODEFormer shows a particularly strong performance, especially the RK-4 setting. On the Chameleon dataset, GNODEFormer (RK- 4) achieves 65.03%, outperforming all other models, including FedSage+ Similarly, in the Squirrel and Actor datasets, GNODEFormer (RK-4) yields the highest accuracies of 46.22% and 51.53%, respectively, showcasing its superior ability to manage heterophilic relationships, where the node connections are more complex and less homogeneous. However, as noted carlier, the model's improved performance comes at the cost of increased computational requirements, particularly in larger and more complex heterophilic graphs, which may slow down convergence in real-world federated learning scenarios. While the model excels in accuracy, especially in the RK-4 configuration, future work could focus on optimizing the computational efficiency to ensure faster convergence, particularly for larger, heterophilic datasets.
We also highlight very insteresting observation. In extreme non-lID settings, since the data distribution across clients is very uneven. For example, users with similar interests demographics could be clustered within particular country, leading to an imbalanced distribution of node classes across different clients (Yao et al. 2023 In such situations, aggregating information over many hops could lead to the model primarily learning from nodes with similar labels within each client. This can lead to node embeddings that are too similar, ultimately hindering the model's ability to discriminate between different node classes, which the essence of oversmoothing (Rusch et al. 2023). This leads to FedGCN(2-hop) should perform worse than (I-hCN()-hop) in extreme non-IID) settings.
Table 6: Results for a=0.05 for across graphs
Client=5
Homophilli
Heterophällie
Dataset
Cora
Citeseer
ploto
chauneleon
squirrel
actoor
0.0G
0.05
0.05
0.05
0.03
FedGCN
85.27% + 2.17%
71.3% 4 0.46%
21.7%
11.61%
27.05%
5.01%
22.59%
4.11%
1880
FedGCN
62. 10% + 0.64%
70.70% +: 0.57%
33.70%
3.30%
25.56% + 6.34%
20.84% + 1.80%
nan d: nAn
FedGCN
80.83% 0.81%
68.13%
0.66%
33.76%
3.33%
22.71%
2.27%
21.28%
usn
FedSage+
85.27% + 2.17%
74.37%
0.40%
33.84%
3.41%
27.05%
5.01%
22.50%
4.1%%
nan nan
2DDEphormer(RK-2
87.52% + 1.97%
74.58%
+ 0.80%
93.70%
= 2.86%
64.29%
+ 2.44%
46.78%
= 2.95%
44.00% = 17.30%
NODEphormer (RK-4)
87.30% 2.28%
74.44%
1.02%
88.00%
0.49%
65.03%
5.06%
48.22%
4.85%
51.53% 25.28%
A.3 Cost and Communication analysis
We provide cost and communication analysis of our proposed architecture in FL. setup. In communication analysis table we provide statistic for parameters and memory usage of transferred model in bytes.
Table 7: Communication analysis of GNODEFormer
Dataset
Parameters
Size in B
cora
140218
560872
citeseer
285426
1141704
photo 96258
385032
chameleon
197162
788648
squirrel
182058
728232
actor
108010
432040
Table & provides an analysis of the time required for Fed-GNNODEormmer to complete an entire experiment as well as the time taken for one local epoch across various datasets, comparing the second-order Runge-Kulla and fourth-order Runge-Kutta methods.
It is evident that RK-4, though delivering superior performance in terms of accuracy (as discussed in previous sections), comes at the cost of increased computational time. Across all datasets, the complete experiment
18

---

Published in Transactions on Machine Learning Research (04/2025)
time for RK-4 is significantly longer compared to RK-2. For instance, the Photo dataset, RK-4 takes 392.61 seconds, which is approximately 1.65x longer than RK-2, which completes in 237.371 seconds. Sim- ilarly, the Squirrel dataset shows a similar trend, where RK-4 requires 245.274 seconds, while RK-2 only takes 187.184 seconds.
This increased computational time with RK-4 also reflected the time per local epoch. For all datasets, the time taken per epoch is consistently higher for RK-4. For example, on the Actor dataset, RK-4 takes 3.946 seconds per epoch, which is significantly higher than the 2.361 seconds required for RK-2. Similarly, the Photo dataset, RK-4 takes nearly 4 seconds (3.919 seconds), which is again much longer than the 2.368 seconds for RK-2.
From these results, it clear that while RK-4 provides betler model performance in terms of accuracy, especially on complex datasets like heterophilic graphs (Chameleon, Squirrel, Actor), it requires more com- putational time, both at the level of single epoch and the overall training process. The increased complexity of RK-4, particularly its ability to better capture the complex relationships in heterophilic datasets, seems to directly translate to increased resource demands.
This trade-off between performance and computation must be carcfully considered in real-world scenarios, especially where computational resources are limited or when working with larger datasets. In such cases, RK-2 may be preferred as it offers a balance between reasonable accuracy and faster computation times, especially in homophilic graphs like Cora and CiteSeer. However, for tasks where accuracy and model performance are critical, and computational resources are less of concern, RK-4 remains a better choice, despite its slower convergence.
Table Time Analysis of GNODEFormer
Dataset
Method
Complete Experiment (Seconds)
One local epoch (Seconds)
RK-2
129.715
1.293
Cora
RK-4
149.631
1.492
RK-2
133.908
1.335
Citeseer
RK-4
175.14
1.747
RK-2
237.371
2.368
Photo
RK-4
392.61
3.919
RK-2
128.997
1.286
Chameleon
RK-4
144.147
1.437
RK-2
187.184
1.867
Squirrel
RK-4
245.274
2.447
RK-2
263.627
2.361
Actor
RK-4
395.259
3.946
A.4 Compute Resources
We utilized an NVIDIA GeForce RIX 4090 with 24 GB of memory and NVIDIA A.00 GPUs configured with MIG instances, each equipped with 20 GB of memory, for our Federated Learning (FL. setup. All datascts were trained in federated setting using the NVIDIA GeForce RTX 4090 and A100. For centralized lesting, the NVIDIA A100 was used for the Cora, Citeseer, Chameleon, and Squirrel datasets, while the NVIDIA A40 with 48 GB of memory was employed for training the Photo and Actor datascts using the Rungo-Kutta 4th-order (RK-4) method. For cost and communication analysis, single NVIDIA A100 GPU operating in MIG mode with 20 GB of memory utilized.
19

---

Published in Transactions on Machine Learning Research (04/2025)
A.5 Comparing GNODEFormer across graph types
In Figure plot accuracy vs. epochs to show convergence of our centralised model on both type of graphs.
Corn
Cora
60-
Tox ALCUP
Tos: ACCURIC
380
208
Chamaleon RK2
Chamelson RK-4
Toa ACC/NO2
Tot ACO
024
07
Epech
Epoch
Figure 4: Comparative performance of Cora (Homophl))) and Chameleon (Heterophii) datasets using RK-2 and RK-4 methods.
20

---

Published in Transactions on Machine Learning Research (04/2025)
B More Ablation
B.1 Visualizations and Plots for Learned Spectral Filters in GNODEFormer
We see the filter learned by the our model for homophillic graphs are low-pass filters and are better than Specformer. Same is true for band-pass filters for heterophillic graphs too.
(a) Cora
(b) Citeseer
(c) Photo
Sicbome
(d) Chamelcon
(e) Squirrel
(I) Actor
Figure 5: Eigenvalue plots for different datasets.
C Mathematical Representation of ODE Integration in Transformer Blocks
Here provide mathematical foundation for integrating Ordinary Differential Equations (ODEs) into Transformer architectures.
C.1 Residual Block as an Euler Discretization
A standard residual block in a Transformer can be expressed as:
V++1 1/+ F(yn, 0c),
where is the input at step t, Og represents block parameters, and F(y,, 0,) denotes the function computed by the block (c.g., self-attention or foedforward network).
This formulation aligns with the Fuler method for solving ODFs:
ytt + At) = v(t) + At P(v(t),((t)),
where At the step size (set to in standard Transformers). As At + 0, this discretization leads to the continuous ODE:
dy F(y(t), 0(t))
dt
Thus, residual block corresponds to single step in solving this ODE.
21

---

Published in Transactions on Machine Learning Research (04/2025)
C.2 Higher-Orde Order Runge-Kutta Methods
To improve accuracy, higher-order ODE solvers, such as the Runge-Kulte (RK) methods, are proposed for designing Transformer blocks. The general form of an explicit n-step RK method is;
yt+ y
+
F.
where Fú intermediate approximations computed iteratively as:
i-1 Big F
=
F
+
1=1
and 76. Biy are method-specifie coefficients.
This equation can be directly mapped to neural network's layer-based representation:
I+1
wifo
x' + Σ
aiski
j=l
where:
( is the input to the layer (analogous to ve),
((+1) the output of the layer (analogous to yc+1),
p is the order of the Runge-Kutta method (c.g., 2, 3, or 4),
fe represents the neural network layer function with parameters 0,
Wt corresponds to "i serving as weights for combining each intermediate Ass
a. corresponds to By, defining the coefficients for intermediate steps,
h are intermediate values representing estimates of changes over step.
In this formulation:
Each layer corresponds to temporal evolution step in solving the ODE.
The weights 7/s in the RK method are represented by w, in the network.
The coefficients Bij map to aij, controlling intermediate computations.
The function F(y,, 06) the RK method modeled as fe, parameterized by the network.
This parallel demonstrates how higher-order RK methods naturally extend to neural networks, enabling precise approximations ODE solutions while maintaining interpretability in terms of temporal dynamics.
We have used popular variants like RK2 (second-order) and RK4 (fourth-order) as they offer more precise approximations of ODE solutions within each Transformer block.
C.3 Coefficient Learning
Learning coefficients 7i (or weights w, the network) during training further optimizes model performance by adaptively weighting intermediate approximations. This enhances the expressiveness and efficiency of the Transformer architecture.
22

---

Published in Transactions on Machine Learning Research (04/2025)
C.4 Working of Runge- Kutta variants
Based on the equations below
111 2 +
Σ wifo + -aaahh aigks
where 20)) is the imput to the layer, 2(+1) is the output of the layer, is the order of the Sunge-Kutte method (2, 3, or 4), fo represents the neural network layer function with parameters 0, w; are the weights for combining each &ę term, di are the coefficients for the intermediate steps, kę are intermediate values representing estimates of the change x over a step, and k are the previously calculated values used in computing the current kg
We show the calculation of RK2 and RK4 methods by putting relevant values.
C.4.1 Working of RK2 Method
(8)
(9)
1 0
Expanded form:
b fo(2))
(10)
22 fo (2)
(11)
The final step calculation:
1 k2
(++) x
+
(12)
C.4.2 Working of RK4 Method
W
(13)
(14)
Expanded form:
ki fa(2")
(15)
fo -0.5k1)
(16)
ka fol2' + +0.5k2)
(17)
Ra fo + -a)
(18)
((+) 3 + k +
1
+
1:
+
k4
(19)
6
3
6
23

---

Published in Transactions on Machine Learning Research (04/2025)
D Scalability with Graph Size
Our method inherits standard scaling challenges of GNNs, such as memory and computation overhead for large graphs. Existing solutions, like sampling techniques graph sparsification, can be adapted to address these issues and explored in future work. Section A.3 provides a detailed analysis of the computations ional costs and communication overhead in federated learning (FL) setup, comparing second-order (RK-2) and fourth-order (RK-4) Rutge-Kutta solvers.
Table shows that RK-4 delivers superior accuracy on complex heterophilic graphs but at higher compu- tational cost (e.g., 1.65x longer the Photo dataset). RK-2, while faster, is effective for homophilic graphs resource-constrainned scenarios. This trade-off demonstrates that our method scales elfectively, balancing computational demands and performance based on application requirements.
We also conducted extensive experiments with GNODEFormer on largo-scale homophilic and heterophilic graphs. Specifically, we tested on Penn94 (heterophilic) (Lim et al. 2021). which contains 41,554 nodes and 1,362,229 edges, and ogbu-arXiv (homophilic) (Lim al. 2021 edges. Below the experimental results:
which has 169,343 nodes and 1,166,243
Model
Penn94
OGBN-arXiv
GCN
82.47 + 0.27
71.74 + 0.29
GAT
81.53 + 0.55
71.82 + 0.23
JacobiConv
83.35 + 0.11
72.14 + 0.17
Graphormer
OOM
0OM 72.37 0.18
SpecFormer
84.32 + 0.32
GNODEFormer (RK-2)
84.96 + 0.06
72.76 0.17
GNODEFormer (RK-4)
84.98 + 0.04
72.97 0.30
Table 9: Performance comparison large-scale graphs. OOM: Out of Memory.
Additionally, we analyzed the filters learned by GNODEFormer and compared them with those of SpecFormer the 25th epoch, the 250th epoch, and the final epoch for both the Arxiv and Penn94 da datasets. Given the large scale of these graphs, this intermediate analysis essential for evaluating how model learns over time and for understanding the evolution of the learned filters.
(a)
(b)
(d)
(f)
Figure 6: Learned filters at the 25th, 250th, and final epochs for Arxiv (a c, top) and Penn94 (d bottom).
24

---

Published in Transactions on Machine Learning Research (04/2025)
E Hyperparameter Tuning Strategy
We followed standard hyperparamcter tuning practices to ensure fair and consistent evaluation across all baselines and variants of our proposed model. Specifically, we performed grid search over the following parameters:
Learning rates: {le-4, 4, le- 3e-33
ODE solver steps: {2
In addition; we made slight manual adjustments based on empirical observations during inference. For example, we used a slightly lower learning rate cases with extremely low values (i.c., high non-IID data settings) and significant heterophily, this improved stability and convergence.
We also conducted ablation study to evaluate the impact of key hyperparameters, including the ODE solver step size and model depth. These results are summarized in Appendix 4.5
We also observed that model performance exhibits nuanced behavior across different a values. In general, higher a (i.e., more IID data) tends to improve average accuracy, though not uniformly across models.
25


Reviews:
[{"review_id": "JHF9yH94wk", "reviewer": "Reviewer_v6xZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. **Innovative Architecture**: The combination of spectral GNNs and Neural ODEs to form the GNODEFormer architecture, and its extension to the federated learning scenario as Fed-GNODEFormer, is novel. This integration allows for better capturing of complex relationships and dynamic changes between nodes in graph data, especially under non-IID conditions, bringing new technical ideas to federated graph learning.\n2. **Strong Capability for Non-IID Data**: The experimental results show that this method outperforms existing methods under extremely non-IID conditions (e.g., α=0.01), effectively addressing the issue of uneven data distribution across different clients. This is significant for many real-world applications such as social network analysis and recommendation systems, where data is often highly heterogeneous and unevenly distributed.\n3. **Wide Applicability**: Fed-GNODEFormer not only performs well on non-IID data but also achieves comparable performance on IID data, demonstrating its adaptability and robustness across different data distributions. Moreover, its good performance on both homophilic and heterophilic graphs further proves its universality in diverse graph structures.\n4. **Privacy and Communication Optimization**: The authors emphasize the design of Fed-GNODEFormer in terms of data privacy protection and communication cost optimization, which are crucial for practical applications of federated learning. In real-world scenarios, data privacy and communication efficiency are often key factors restricting the development of federated learning, and Fed-GNODEFormer achieves a good balance between them, making it closer to the requirements of practical applications.\n\n###  Weaknesses\n1. **Computational Efficiency**: Despite its great performance, Fed-GNODEFormer suffers from relatively low computational efficiency, especially when dealing with larger heterophilic graph datasets. For example, on the Squirrel and Actor datasets, the time required for each local training epoch is quite long when using the fourth-order Runge-Kutta method (RK-4), which may limit its application in large-scale or resource-constrained environments. \n2. **Model Complexity and Scalability**: As the scale of graph data increases and the graph structure becomes more complex, the complexity of Fed-GNODEFormer may rise further. The paper does not provide detailed discussions on how to effectively scale this method to large-scale graph data, which may affect its applicability in industrial-level applications. \n3. **Hyperparameter Selection and Tuning**: The high-order Runge-Kutta methods in Neural ODEs involve multiple hyperparameters (such as step size and order), which significantly affect the model's performance and computational efficiency. However, the paper provides relatively little discussion on these hyperparameters and lacks systematic strategies for hyperparameter selection and tuning. addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a novel federated learning method called Fed-GNODEFormer for training GNNs on non-IID graph data. By integrating spectral GNNs and Neural Ordinary Differential Equations, this method aims to better capture complex relationships in graph structures and demonstrates superior performance on both homophilic and heterophilic graphs. The authors provided a detailed description of the model architecture, including the centralized GNODEFormer and its extension to the federated learning scenario, Fed-GNODEFormer. Extensive experiments validate the effectiveness of this approach under various data distribution conditions. Overall, this research is innovative and practical in the field of federated graph learning, offering a new perspective for handling complex graph data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See  weaknesses", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sg2tGIi0cC", "reviewer": "Reviewer_87kj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper provides a novel idea of using high-order ODE solvers to simulate the message-passing processing in the GNN.\n2. Numerical results on the datasets demonstrate strong improvement compared with existing methods.\n\nWeakness:\n1. Missing discussion on related works. Specifically, the author should provide more discussion on the connection between the proposed method and other GNN+ODE/PDE works (e.g., [R1], [R2]).\n2. Unclear FL setting. FL+Graph data has many different settings, including graph/subgraph/edge/node/feature distributed settings. The paper should clarify the setting in the abstract and in the introduction.\n3. It is counter-intuitive that more iid data (larger $\\alpha$) results in worse performance than non-iid data. Although the authors claim it is due to a lack of sufficient parameter tuning, it is not convincing. More careful experiments are desired.\n \n[R1] Zhuang J, Dvornek N, Li X, Duncan JS. Ordinary differential equations on graph networks.\n[R2] Eliasof M, Haber E, Treister E. Pde-gcn: Novel architectures for graph neural networks motivated by partial differential equations. Advances in neural information processing systems. 2021 Dec 6;34:3836-49.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes using the Runge-Kutta method to simulate the process of graph signal propagation in the frequency domain. The paper further extends the proposed GNN to the federated learning setting. The paper provides experiments on both centralized and federated learning settings to demonstrate that the proposed GNN has a better capacity to express the graphs (node classification) on various datasets under homogeneous and heterogeneous data distributions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses above.\n\nMinor:\n1. replace \"alpha\" with $\\alpha$.\n2. Correctly using \\citet{} and \\citep{}\n3. The notations are confusing. $n$ is used as both the number of nodes, the number of layers, and the number of clients. $i,j$ are used as node index, steps in RK method, client index. $s_i$ and $c_i$ are used to denote client, etc. The paper is hard to understand.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sc0sB2ntvs", "reviewer": "Reviewer_AfWR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The paper is well-organized, with each component of the proposed method clearly explained and seems easy to follow.\n2. The existing numerical results seems good and demonstrate the effectiveness of the proposed method. \n\n**Weaknesses**:\n1. I’m a little bit unclear on how client selection is conducted in your experiments, and the current setup with 5 clients seems somewhat limited.\n2. Figure 2 is a bit confusing to me. I understand that GNODEFormer is designed for undirected graphs, but the figure depicts an example of a directed graph. Additionally, the proposed Fed-GNODEFormer appears to assume different graph structures across clients by “partitioning the global graph G into m subgraphs.” However, this partitioning process isn’t clearly illustrated in Figure 2.\n3. In my view, the key factor driving Fed-GNODEFormer’s performance improvement in FL scenarios primarily stems from the advantages of the GNODEFormer architecture itself. This includes its strong performance under highly non-IID conditions, suggesting that much of the benefit can be attributed to GNODEFormer rather than the adjustment for federated learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores Fed-GNODEFormer, a GNN model designed for FL that addresses the challenge of heterogeneous graph data distribution across clients. The model leverages spectral decomposition and ODEs to capture both structural and dynamic graph features, enhancing performance on complex and diverse graph datasets. The authors present numerical experiments to validate their methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Regarding the aforementioned W1, I recommend that the authors include a discussion on client assignment and consider conducting additional experiments with a larger number of clients, if needed.\n* Regarding W2, I suggest the authors provide clearer explanations or revisions to the figures to enhance clarity and avoid potential confusion.\n* Some citation styles in the paper need to be corrected. The authors should clarify the use of \\citep and \\citet.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NKEVvDcDRc", "reviewer": "Reviewer_tZrz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a clear motivation for the proposed method, emphasizing the practical need for privacy-preserving graph learning.\n2. The experimental results are comprehensive, demonstrating the method's effectiveness across a variety of graph datasets and non-IID conditions.\n\nWeaknesses:\n1. How does the proposed method scale with the size of the graph, especially when dealing with very large and complex graphs?\n2. Are there any specific hyperparameter tuning strategies mentioned for the neural ODE components that are crucial for the model's performance?\n3. There are several relevant methods missing in the manuscript that need to be discussed and analyzed: A Survey of Graph Neural Networks in Real world: Imbalance, Noise, Privacy and OOD Challenges.\n4. What are the computational costs associated with the higher-order Runge-Kutta methods compared to simpler ODE solvers, and does this impact the practicality of the approach?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel federated learning approach for Graph Neural Networks (GNNs) that combines spectral GNNs with neural ordinary differential equations (ODEs) to address non-IID data challenges in both homophilic and heterophilic graph settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel federated learning approach for GNNs using spectral methods and neural ODEs. The results are promising, especially for non-IID heterophilic graphs. Minor revisions are needed to clarify the experimental setup and provide more detailed comparisons with existing methods.
Decision: Accept (poster)