FedSSP: Federated Graph Learning with Spectral Knowledge and Personalized Preference

Authors: zihan  tan
guancheng  wan
wenke  huang
mang  ye
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 09:00:57

Abstract: Personalized Federated Graph Learning (pFGL) facilitates the decentralized training of Graph Neural Networks (GNNs) without compromising privacy while accommodating personalized requirements for non-IID participants. In cross-domain scenarios, structural heterogeneity poses significant challenges for pFGL. Nevertheless, previous pFGL methods incorrectly share non-generic knowledge globally and fail to tailor personalized solutions locally under domain structural shift. We innovatively reveal that the spectral nature of graphs can well reflect inherent domain structural shifts. Correspondingly, our method overcomes it by sharing generic spectral knowledge. Moreover, we indicate the biased message-passing schemes for graph structures and propose the personalized preference module. Combining both strategies, we propose our pFGL framework $\textbf{FedSSP}$ which $\textbf{S}$hares generic $\textbf{S}$pectral knowledge while satisfying graph $\textbf{P}$references. Furthermore, We perform extensive experiments on cross-dataset and cross-domain settings to demonstrate the superiority of our framework. The code is available at https://github.com/OakleyTan/FedSSP.

Main paper: 
 introduction: 
 Graph Neural Networks (GNNs) [56, 63, 48, 42] have demonstrated their superiority in modeling graph data which frequently emerges in a variety of scenarios [73, 71] , as exemplified by graph clustering [45, 46, 44] , graph contrastive learning [64, 80] , anatomy detection [75, 88] , knowledge graph [79, 78, 38] , structural inference [66, 68, 65, 67] and so on. However, large amounts of graph data are generated by edge devices in reality, which brings in privacy concerns and the challenges of data silos [89, 24, 28] . To address these difficulties, federated learning (FL) has recently been applied to graph learning [18, 20, 25] . It allows models on various clients to collaborate without sharing local data [26, 15, 23, 27, 83] and makes federated graph learning (FGL) a promising direction. Nonetheless, the non-IID problem remains a major challenge in FGL, as graph data from different clients usually vary significantly. In such scenarios, a single global model struggles to adapt well to the local data of each client with inconsistent data distributions [60, 58] . To tackle these challenges, personalized federated graph learning (pFGL) has emerged, offering customized GNNs for each client to achieve satisfying local performance [1, 61] .However, pFGL still encounters substantial challenges from structural heterogeneity [29] , especially in domain shift tasks, for instance, between social networks [94, 95] and molecular structures [59, 55] . There are two significant drawbacks to previous algorithms as Fig. 1 demonstrates. For global The aggregated message-passing scheme suffers from inconsistent preferences that remain unsatisfied of specific datasets in this scenario. Consequently, it leads to unsuitable features of graphs in local applications. (c) The heat map of Jensen-Shannon divergence of algebraic connectivity [17] and eigenvalues distributions among six datasets from three different domains. Spectral characteristics exhibit significant biases across domains but are more similar within a same domain.collaboration, the considerable domain structural shifts inevitably lead to non-generic knowledge, thus resulting in knowledge conflict. Both current methods suffer from conflict and are trapped in unpromising collaboration. Specifically, [61] share non-generic structural encoding and struggle with structural knowledge conflict, while strategy [77] mitigating conflicts by limiting the potential for collaboration. The key to addressing knowledge conflict is pursuing a way to share generic knowledge that benefits all clients. Based on this observation, we raise the question: 1) how to address the knowledge conflict under domain structural shift by extracting and sharing generic knowledge?For local applications, each client owns its specific dataset with distinct structural characteristics in this cross-dataset scenario. Due to the GNN message-passing nature, distinct graph structures stored in different clients prefer different message-passing schemes. Consequently, the scheme provided by aggregated GNN exhibits biases from the optimum when applied locally, thus leading to unsuitable features. Both current methods neglect the preferences of various clients for specific graph structures. This deficiency leads us to consider: 2) how to design personalized plans to deal with inconsistent preferences of specific graph datasets from various clients?To address problem 1), given that structural shifts make it hard to directly achieve generic sharing at the structural level, we propose to explore the structure shifts from another spectral perspective since previous works have demonstrated the strong correlation between graph structure and spectra [2, 81, 43, 32] . The major advantage of spectra is the detailed propagation and processing of graph signals on the graph structure, thus facilitating the discovery of generic knowledge in several certain processes unaffected by structural shifts. To validate our assumption, we first conduct experiments and explicitly reveal the domain spectral biases that directly reflect domain structural shifts on spectra as Fig. 1 demonstrates. To tackle these spectral biases directly to overcome structural shifts, we design Generic Spectral Knowledge Sharing (GSKS) to share generic spectral knowledge extracted from spectral encoders. It enables clients to benefit from others through collaboration without knowledge conflict. Conversely, other components containing non-generic knowledge are retained locally. Correspondingly, clients can customize powerful graph convolutions for their local graph characteristics while benefiting from generic knowledge without conflict. Through this strategy, we promote the sharing of generic spectral knowledge and the personalizing of non-generic knowledge, thus achieving effective collaboration against knowledge conflict. Moreover, we attempt to achieve target 2) and design suitable personalized plans for each client graph structure locally. Specifically, we explore the message-passing nature of GNN [5, 16, 62] . From the spectral perspective, spectral encoders strongly affect message transmission. Therefore, when aggregated spectral encoders are applied to distinct graph structures locally, they tend to deviate from the optimal message-passing scheme for the client [49] . Consequently, GNNs extract inappropriate frequency messages which lead to unsuitable features. To meet the inconsistent graph preferences, we innovatively configure a learnable preference for each client and propose Personalized Graph Preference Adjustment (PGPA). These personalized preference modules apply adjustments to the feature extracted with the participation of global spectral encoders. It allows the feature to suit the specific graph structures of each client. Moreover, to address the issue of over-reliance when applying the preference module independently, a regularization term is introduced. Combining both strategies for effective global collaboration and personalized local application, we propose our pFGL framework FedSSP. In conclusion, our key contributions are:• We are the first to reveal domain structural shifts through spectral biases, as well as consider the inconsistent preferences of distinct datasets from various clients.• We propose FedSSP, which innovatively overcomes knowledge conflicts from a spectral perspective and implements personalized graph preference adjustments for each client.• We conduct experiments in various cross-dataset and cross-domain settings, proving that our approach outperforms several current state-of-the-art methods and achieves optimal results.2 Related Work 
spectral gnns: 
 Spectral GNNs [4, 12] are based on spectral graph filters set in the spectral domain, providing powerful models for graph neural networks [76, 87, 86, 47, 72, 10, 70, 8, 69, 7] . Spectral GNNs can generally be categorized into two types: those with fixed filters and those with learnable filters. Fixed filter spectral GNNs, such as APPNP [19] , utilize personalized PageRank (PPR) [53] to construct their filtering functions. GNN-LF/HF [96] designs filter weights from the perspective of graph optimization functions. Learnable filter spectral GNNs include subclass that approximates arbitrary filters using various types of orthogonal polynomials, including Bernstein [22] , Chebyshev [21] , and Jacobi [74] . Another subclass parameterizes the filters by neural networks, including LanczosNet [40] and Specformer [3] . The robust modeling capability of spectral graph neural networks on data inspires us to leverage this foundation to tackle the issue of structural heterogeneity across domains.Federated learning [84, 34, 28, 14, 82] facilitates privacy-preserving collaborative learning on local data, introducing methods like FedAvg [51] to address this. Yet, it struggles with non-IID data across clients. Several techniques aim to address the challenge [33, 35, 30] , but achieving a global model that suits all local data remains difficult [29] . Personalized Federated Learning (pFL) has attracted increasing attention for its ability to address the non-IID problem [13, 39, 60] . Research has approached improvements from various aspects. In personalized-aggregation-based methods, FedPHP aggregates the global model and old personalized models locally to preserve historical information [36] , FedALA achieves personalized aggregation through personalized masks [91] , and APPLE uploads only core models and employs directed relationship vectors for downloading [50] . In model-splitting-based approaches, FedRoD [6] learns with a global feature extractor and two heads for both global and personalized tasks. FedCP decouples features suitable for global and local heads through a conditional policy scheme [92] . Moreover, methods based on regularization and knowledge distillation have also been utilized to enhance pFL. However, pFL methods lack targeted strategy designs for graphs, making them not particularly suited for pFGL scenarios.Recent studies have utilized the FL framework for distributed training of GNNs without sharing graph data [20, 41, 28, 9] . Current Federated Graph Learning (FGL) research can be categorized into two types: intra-graph and inter-graph FGL. In inter-graph FGL, each client has distinct graphs, and they jointly participate in federated learning to either improve GNN modeling of local data or achieve a model that can generalize across different datasets [77, 61] . Intra-graph FGL, on the other hand, aims to deal with challenges such as missing link prediction [11] , subgraph community detection [93, 1] , and node classification [25, 37] . However, most FGL methods lack specific design considerations for our scenario. More precisely, there is a general absence of consideration for the heterogeneity of graph-level structures and the personalized needs of different clients brought about by structural characteristics. In this paper, we focus on inter-graph FGL, taking into account spectral domain biases and the uniqueness of graph structures that result in client-specific preferences, to customize a personalized optimal model for each client specifically for graph classification tasks.3 Preliminary 
graph signal filter: 
 Assume that we have a graph G = (V, E), where V represents the node set with |V| = n and E is the edge set. The corresponding adjacency matrix is defined as A ∈ {0, 1} n×n , where A ab = 1 if there is an edge between nodes a and b, and A ab = 0 otherwise. The normalized graph Laplacian matrix is defined asL = I n -D -1/2 AD -1/2, where I n denotes the n × n identity matrix and D is the diagonal degree matrix. We assume G is undirected. L is a real symmetric matrix, whose spectral decomposition can be written as L = U ΛU T , where the columns of U are the eigenvectors and Λ = diag(λ 1 , λ 2 , . . . , λ n ) are the corresponding eigenvalues ranged in [0, 2]. The Graph Fourier transform of a signal x ∈ R n×d is defined as x = U T x ∈ R n×d . The inverse transform is defined as x = U x [57] . The i-th column of U denotes a frequency component corresponding to the eigenvalue λ i . Let xλ = U T λ x, where U λ represents the eigenvector corresponding to λ, be the frequency component of x at λ frequency. We can use a function g : [0, 2] → R to filter each frequency component by multiplying g(λ). By defining Λ = diag(λ), g implements filtering on Λ , thus ultimately implementing filtering on the graph signal x. The whole process is defined as follows:EQUATIONBy defining g( L) = K k=0 α k Lk , where g is often set to be a polynomial of degree K for parameterizing the filter, the filtering process can be rewritten as follows:U g(Λ)U T x = g( L)x.(2)Traditional FL leverages isolated data of distributed clients and collaboratively learns models M for a generalizable global model without leaking privacy. Specifically, the goal is to minimize:EQUATIONwhere f g (•) denotes the global objective. It is computed as the weighted sum of the N local objectives, with N being the number of clients and w i ≥ 0 being the weights. The local objective M i (•) is often defined as the expected error over all data under local data D i .In the context of personalized federated learning, the global objective takes a more flexible form:EQUATIONwhere f p (•) is the global objective for the personalized algorithms, andΘ = [θ 1 , θ 2 , . . . , θ N ]is the matrix with all personalized models. In this work, we aim to obtain the optimal Θ * = arg min Θ f p (Θ), which equivalently represents the set of optimal personalized models θ i , i ∈ [N ].Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
generic spectral knowledge sharing (gsks): 
 Motivation. Current methods suffer from knowledge conflict arising from non-generic sharing under domain structural shifts. Since structural shifts impede the direct generic sharing at the structural level, we are the first to reveal and resolve knowledge conflicts from the spectral perspective. To explicitly address the spectral biases that reflect structural shifts in Fig. 1 , we base our pFGL strategy on spectral GNNs and further propose Generic Spectral Knowledge Sharing (GSKS). Effective collaboration that overcomes spectral bias and structural shift is achieved, thereby addressing knowledge conflict. Details of GSKS are presented in Fig. 2 (a) .Eigenvalue filtering. Aiming at more expressive representations of frequency information, the eigenvalues are firstly mapped from scalars to meaningful vectors for subsequent learning of frequency for satisfying inconsistent preferences and achieving suitable feature of datasets locally. These two boxes correspondingly refer to the two core strategies of our framework FedSSP.interrelation in the multi-head attention module as follows:EQUATIONwhere c keeps eigenvalues within a suitable numerical range to distinguish different eigenvalues for trigonometric functions. q ∈ Z ∩ [0, d -1] is the index for dimension d while β controlling the importance of λ with defaulted value 10000. Moreover, θ e denotes parameters of the eigenvalue encoder ϕ e , by which eigenvalues are mapped from scalars to vectors for richer frequency information. Consequently, they are more expressive for filtering by the attention module and decoder through R 1 → R d . The initial representations are the concatenation of eigenvalues and their encodings: d+1) . Then the multi-head attention module is leveraged. After stacking multiple transformer blocks, spectral decoder ψ d for eigenvalue decoding can learn new eigenvalues from the expressive representations of spectra:λ ′ = [concat[λ 1 , ϕ e (θ e ; λ 1 )], . . . , concat[λ n , ϕ e (θ e ; λ n )]] T ∈ R n×(EQUATIONWhere the representations learned by the m-th head are fed into ψ d , while ψ d denotes a combination of liner and optional activation. λ m ∈ R n×1 is the filtered eigenvalues by the m-th head. The whole process in Eq. ( 6) acts as a spectral filter g for the origin eigenvalues in Eq. (1).To address the challenge of knowledge conflict, we attempt to explore the functionality of each module. The eigenvalue encoder ϕ e captures multi-scale representations of eigenvalues and provides meaningful vectors of distinct frequencies. Since the mapping from eigenvalues to vectors by ϕ e is independent of the domain characteristics, θ e of ϕ e is considered to contain generic knowledge. In contrast, as the spectral biases we reveal in Figure 1 demonstrate, biases exist in eigenvalue distribution across domains. In contrast, spectral characteristics within the same domain are more similar. Therefore, the attention module learns the non-generic magnitudes and relative dependencies specific to the spectral characteristics of each client. Correspondingly, the eigenvalue decoder focuses on decoding the most suitable message-passing scheme and client-specific frequency components from the representation processed by the attention module. Attention module and decoder together formed g in Eq. ( 1), aiming at designing personalized filtered eigenvalue that guides message-passing at a personalized suitable frequency. Therefore, θ e is shared in our strategy to achieve generic spectral knowledge sharing and effective collaboration unaffected by knowledge conflict.Specifically, client i uploads its update of θ e i . At the t-th iteration (t ≥ 0), the central server distributes global spectral encoder weights θ t g . Accordingly, client i updates local GNN weights including θ e i with their dataset D i and send these updates as ∆θ t i = θ t i -θ t g to the central server. Then the server aggregates the received local updates and modifies the global weight θ t+1 g as follows:EQUATIONnotably, aggregation based on sample size is unsuitable in this scenario for effective collaboration across various domains and datasets. Since clients here possess specific datasets with significant quantitative variance, those with larger datasets tend to dominate the collaboration. Thereby preventing them from benefiting from the spectral and frequency knowledge of clients with fewer samples. Correspondingly, clients with fewer graphs are almost entirely dominated by knowledge that does not originate from their local data. To address the problem, we leverage a direct average of spectral encoder weights from all clients to achieve fair collaboration and cross-dataset knowledge sharing.Personalized graph convolution constructing. After getting M filtered eigenvalues, filter encoder:ϕ f (θ f ; B) R M +1 → R d isleveraged to construct the bases for personalized graph convolution. To avoid confusion and distinguish from the mentioned filter g on eigenvalues in Eq. ( 1), filter here in filter encoder refers to the filtering on feature message-passing through bases in graph convolution. New bases are first reconstructed and concatenated along the channel dimension. Specifically, by defining Λ m = diag(λ m ), they are fed into filter encoder ϕ f as follows:EQUATIONEQUATIONwhere Eq. ( 7), thereby achieving a comprehensive understanding of different frequency messages in graphs. Subsequently, it enables the construction of powerful personalized graph convolutions as follows:B = [B 1 , B 2 , . . . , B M ] ∈ R n×n×M while B m ∈ REQUATIONwhere x v is the node v representation from the previous layer, while x ′ v represents the output of the current layer. θ Conv denotes weights of graph convolution, and σ refers to the optional activation. Ultimately, the representations of all nodes within a graph are aggregated by an average pooling layer to form the overall feature representation of graph G l in dataset D i of client i as follows:EQUATIONwhere h l is defined as the average of all node features in graph G l , namely the graph feature, while V refers to the node quantities in graph l here. By sharing generic spectral knowledge and retaining client-specific knowledge we successfully achieve effective collaboration that overcomes spectral bias, thereby domain structural shift from the spectral perspective.Motivation. Due to the GNN message-passing nature, distinct graph structures prefer different message-passing schemes, especially when meeting the specificity of datasets in cross-dataset and cross-domain scenarios. Consequently, The spectral encoders under global collaboration fail to satisfy the inconsistent local preferences of graphs. Correspondingly, graph convolutions tend to learn biased message-passing schemes, thereby extracting unsuitable graph features. Our approach provides personalized adjustments to address this challenge based on client preference. Moreover, we solve the over-reliance issue that arises from the process of satisfying various preferences. Details of GSKS are presented in Fig. 2 (b) .To satisfy the various preferences and make the graph features more suitable for graphs, we propose a learnable preference module that adjusts to features extracted by client i to satisfy local graph structure preference explicitly. The module includes learnable parameters matched in size with the feature space, thus acting as a refined tool to flexibly satisfy the preferences of each client during local training. Considering local model M including feature extractor F(θ F ; G), classification head C(θ C , h), and preference module P(δ), where G represents graphs contained in dataset D of a client. The whole graph feature-extracting process can be defined as follows in our strategy:EQUATIONby integrating the original feature h with preference adjustments δ, h ′ becomes the ultimately suitable feature that satisfies client preference. Now we leverage adjusted feature h ′ for z ′ = C(θ C ; h ′ ) instead of the original unsuitable representation h . Specifically, the local loss for client i is:EQUATIONwhere the Cross-entropy (CE) loss measures the difference between the predicted probability and the true label. Nevertheless, the preference module learns not only the client-specific preference but also aspects that should be handled by the feature extractor F without a guide for preference. As a result, the local feature extractor F might overly rely on adjustments provided by the preference module during training, thereby hindering its capability. Correspondingly, this over-reliance can negatively impact federated collaboration. When the capability of F is degraded, the shared spectral encoders fail to convey beneficial knowledge to others, leading to unpromising collaboration.Therefore, it is essential to guide the preference module to focus solely on the aspects of client preferences, rather than interfering with the feature extraction guided by collaboration. We achieve this by guiding the output of the feature extractor to align more closely with global graph features. Consequently, the PGPA module is directed to concentrate on client preferences. To implement this, we first calculate the mean of local graph features in each iteration:EQUATIONwhere µ denotes the momentum we introduce for bringing graph modeling patterns from previous batches to the current batch in the same local epoch. h pre i and h cur i represent the local mean graph feature of the previous batches and the current batch. It is necessary to distinguish between mean and prototype. In this scenario, clients own various datasets, thus the class information is client-specific. Correspondingly, the mean h i which represents the average modeling for graphs in client i is unrelated to class information. After local training, client i uploads its mean to the server for global consensus aggregation. Based on our exploration of Eq. ( 7), a direct average is leveraged here:EQUATIONwhere hg refers to the global graph modeling consensus calculated from all samples across all clients. Accordingly, we employ the Mean Squared Error (MSE) to measure the distance between the local graph feature mean and the global graph mean obtained from the previous round. This measurement serves as a regularization term to encourage the local graph feature to align closer to the global modeling consensus, thus guiding the preference module to focus on preference and correspondingly address the over-reliance issue. Specifically, local feature extractors are encouraged to extract certain frequency messages in graphs that reflect the global modeling consensus, making the PGPA only responsible for client-specific preference. The local loss of client i is now defined as:EQUATIONBy implementing MSE( hi , hg ), we explicitly align local graph modelings with the global consensus, thus guiding the preference module P to focus on preference and addressing the considered issue of over-reliance by forcing the preference module to focus on client-specific graph preference.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We fully disclose all the information needed to reproduce the main experimental results in this paper and our code. We are convinced that the obtained results can be reproduced. 
experimental setup: 
 We perform experiments on graph classification tasks in various cross-dataset and cross-domain scenarios to validate the superiority of our framework FedSSP.Datasets. Follow the settings in [61] , we use 15 public graph classification datasets from four different domains, including Small Molecules (MUTAG, BZR, COX2, DHFR, PTC_MR, AIDS, NCI1), Bioinformatics (PROTEIN, OHSU, Peking_1), Social Networks(IMDB-BINARY, IMDB-MULTI), and Computer Vision (Letter-low, Letter-high, Letter-med) [52] . Node features are available in some datasets, and graph labels are either binary or multi-class. We create six different settings:(1) cross-dataset setting based on seven small molecules datasets (SM); ( 2)-( 6) both cross-dataset and cross-domain settings based on datasets from two different domains(BIO-SM, SM-CV) and three different domains(BIO-SM-SN, BIO-SN-CV, CHEM-SN-CV)Baselines. We compare ours with several SOTA federated approaches. (1)Local as the first baseline;(2)FedAvg [51] ; (3)FedProx [35] which address heterogeneity issues in FL; (4)APPLE [50] and (5)FedCP [92] , two state-of-the-art pFL method;(6)FedSage [93] , (7)GCFL [77] ,(8)FGSSL [25] , and ( 9)FedStar [61] , four state-of-the-art FGL methods.Implementation Details. The experiments are conducted using NVIDIA GeForce RTX 3090 GPUs as the hardware platform, coupled with Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz. For each setting, every client holds its unique graph dataset, among which 10% are held out for testing, 10% for validation, and 80% for training. We leverage the AdamW optimizer [31] for local GNNs with learning rate 0.001, the default parameter of ϵ = 1e -8, and (β 1 , β 2 ) = (0.99, 0.999), as suggested by [54, 85] . The number of communication rounds is 200 for all FL methods. We report the results with an average of over 5 runs of different random seeds.Performance Comparison We show the federated graph classification results of all methods under six different non-IID settings, including one cross-dataset setting (SM), two cross-double-domain settings (BIO-SM, SM-CV) cross-multi-domain settings (BIO-SM-SN, BIO-SN-CV, SM-SN-CV). We summarize the final average test accuracy in Tab. 1. These results indicate that FedSSP outperforms all other baselines in five out of the six settings. Notably, traditional FL algorithms such as FedAvg and FedProx failed to outperform self-training due to the strong cross-datasets and cross-domain non-IID challenge of this scenario. Correspondingly, algorithms such as FedStar and FedCP which are designed specifically for pFGL or pFL scenarios perform better here. Convergence Analysis Fig. 3 shows the curves of the average test accuracy with standard deviation during the training process across five random runs of three settings (SM, SM-CV, SM-SN-CV) representing single-domain, double-domain, and multi-domain scenarios, including the results of various baselines. As is shown, traditional FL methods such as FedAvg or FedProx own higher standard deviations and are more unstable while methods designed specifically for pFGL scenarios such as GCFL and FedStar are more stable with lower standard deviations.Effects of Key Components Mechanism of FedSSP To better understand the impact of specific design components on the overall performance of FedSSP, we conducted an ablation study in which we varied these components of single, double, and multi-domain settings(SM, SM-CV, SM-SN-CV).As shown in Tab. 2, compared to FedAvg, GSKS significantly enhances accuracy when applied independently. Correspondingly, as a further exploration of the nature of GNN message passing, PGPA achieves noticeable success in adjusting client preferences.Effects of Key Component Mechanism of GSKS Tab. 3 discuss the key component of GSKS.We demonstrated the impact of different sharing strategies. Specifically, sharing only non-generic spectral GNN components or all components often fails to outperform self-training, while GSKS successfully dominates all the strategies. Accordingly, the results fully validate the effectiveness of GSKS in sharing universal knowledge and promoting effective collaboration in this scenario. We compare the graph classification performance under different values of PGPA parameter τ , momentum µ, number of attention heads, and hidden dimension. Where Fig. 4 shows the results when these hyper-parameters are fixed at different scales and values. It indicates that the choosing of τ can affect the strength of PGPA while performance is not influenced much unless they are set to extreme values. All studies of τ and µ here outperform the baseline. We also test the performances under different values of attention heads and hidden dimensions. For results in Tab. 1, we set up 4 heads for the multi-head attention mechanism while 128 for the hidden dimension.  
discussion: 
 Even though FedSSP has achieved significant success in cross-domain federated graph learning collaborations, it still faces certain limitations as a spectral GNN-based approach. Compared to spatial GNNs, while spectral GNNs have the advantage of overcoming structural heterogeneity from the spectral domain, many spectral GNNs require eigenvectors and eigenvalues, which adds to the computational overhead of data preprocessing and subsequent storage burden.Furthermore, we notice a similar approach in DBE [90] which employs static global consensus in FL to separate personalized and global information. Nevertheless, it inevitably struggles to handle scenarios where the message-passing of multiple GNNs is continuously updated. It merely provides a static anchor point, making it difficult to establish a global graph modeling consensus that could guide the local GNNs in capturing graph signals. Instead, we align the GNN backbone with dynamic global graph modeling to avoid the preference module from overly extracting features that should be captured by the GNN itself, which could lead to decreased GNN performance and hinder global collaboration. This approach allows for real-time adjustment of message-passing across different client GNNs, focusing the preference module solely on personalization. Additionally, to address issues such as sample size disparity between domains and dominance of large domains in model parameter aggregation, we adopt a direct averaging strategy in dynamic global aggregation instead of conventional weighted averaging to mitigate these concerns. 
conclusion: 
 In this paper, we pioneer an innovative exploration of cross-domain personalized Federated Graph Learning. To achieve this goal, we achieve improvements in two aspects: seeking effective global collaboration and suitable local application, thus proposing a novel framework FedSSP. For global collaboration, GSKS is leveraged to facilitate the sharing of generic spectral knowledge, overcoming knowledge conflict by domain structural shift from a spectral perspective. For local applications, we design PGPA to satisfy inconsistent preferences of specific datasets contained in clients. By integrating these two strategies, FedSSP outperforms various state-of-the-art methods on various cross-dataset and cross-domain pFGL scenarios in graph classification. 
acknowledgment: 
 This work is supported by National Natural Science Foundation of China under Grant (62361166629, 62176188, 623B2080), Key Research and Development Project of Hubei Province (2022BAD175), and the Luojia Undergraduate Innovation Research Fund Project of Wuhan University. The numerical calculations in this paper have been supported by the super-computing system in the Supercomputing Center of Wuhan University.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The research presented in this paper is foundational. It is not directly tied to any specific applications or deployments.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: This paper does not use existing assets.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conducted in the paper strictly adheres to the NeurIPS Code of Ethics, ensuring that all aspects of the work are in compliance with the guidelines provided.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper focuses on structural heterogeneity in personalized Federated Graph Learning. Specifically, it effectively addresses the domain structural shift from a spectral perspective, tackling issues such as knowledge conflict and spectral bias. The framework includes Generic Spectral Knowledge Sharing (GSKS), which shares generic spectral knowledge across clients within certain components, while applying Personalized Graph Preference Adjustment (PGPA) to meet the preferences of clients for more personalized results. The experimental settings are appropriate to demonstrate the superiority of the proposed methods, although some limitations (e.g., number of clients, non-i.i.d. scenarios) were highlighted by the reviewers. The reviewers initially noted both strong points and concerns.

Strong points:

- All reviewers praise the clear motivation. Alleviating knowledge conflicts from a spectral perspective is a novel and important challenge in personalized Federated Graph Learning.

- The method is thoughtfully designed, and its performance, as well as the harmony between modules, is commendable.

- Comprehensive experiments demonstrate the effectiveness of the proposed method.

Weak points:

- Analysis of the scalability with respect to the number of clients, performance in non-i.i.d. settings, and communication cost is required.

- Additional explanation of the motivation experiments (Figure 1(c)) and justifications for sharing the filter encoder are needed.

- A more detailed discussion of related works is necessary.

The authors have adequately addressed the reviewers’ concerns. Please be sure to incorporate the rebuttal into the revised version, particularly the clarification of the motivation behind sharing the filter encoder and the detailed explanation of the motivation experiment shown in Figure 1. Moreover, the methodology section should be more refined (e.g., where is the parameter for the eigenvalue encoder (i.e., $\theta^e$) used? How is $B$ calculated?). Lastly, it would be beneficial to include an experiment that effectively demonstrates that generic knowledge has less spectral bias than other knowledge, similar to the motivation experiment in Figure 1(c). Despite some limitations, this paper has strong points that address important challenges from a novel perspective (i.e., spectral), and it holds significant potential to provide valuable insights for future work in graph federated learning.

Reviews: review: paper_summary: paper_summary: FedSSP tackles structural heterogeneity well in personalized Federated Graph Learning. When it comes to cross-domain scenarios, the structural heterogeneity becomes more negative than usual. It is crucial to mitigate the impact of domain structural shifts. FedSSP proposes two strategies to address these challenges from two directions. It firstly overcomes knowledge conflict by generic spectral encoder weights to seek better collaboration across various datasets. Besides, considering the uniqueness of clients in this scenario, it then satisfies the special preferences of the graphs by feature adjusting.
strength_weakness: strength_weakness: The client scale considered in the experiments is quite small. Cross-device federated learning typically involves a significantly larger client scale to ensure the robustness and generalizability of the results. Therefore, it would be beneficial to consider expanding the client scale in experiments to better reflect practical scenarios and to validate the scalability of the proposed method.
questions: questions: Please refer to the weaknesses.
limitations: limitations: The experiment should include performance under varying conditions to ensure the method has no limitations in this aspect, such as different numbers of training rounds.
paper_summary: paper_summary: This work proposed a personalized federated graph learning framework for federated graph classification. The framework includes strategies for sharing generic knowledge and satisfying personalized preferences. The authors evaluated six different cross-dataset and cross-domain settings and showed good performance.
strength_weakness: strength_weakness: 1. In the motivation, the authors do not clearly explain the relationship between the two spectral bias metrics and the graph structure.
2. The scalability of the method is not well analyzed in the experiment. 
3. Absence of a detailed discussion on the necessity of the shared filter encoder in the proposed method, particularly in relation to its specific architecture.
questions: questions: Q1) I have some confusion regarding the two spectral bias metrics used in the motivation. Could the authors specifically point out the relationship between these two spectral bias metrics and graph structure?
Q2) Could the authors add experiments on the scalability of the method, such as performance with varying numbers of clients?
Q3) The global consensus in the method is similar to the prototype. If it differs from prototype, authors should provide distinctions and explanations.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces a novel method FedSSP designed to address the current limitations of personalized Federated Graph Learning methods. The author highlighten that existing methods fail to deal with domain structural shift and ignore the uniqueness of datasets in cross-dataset scenarios. To address the limitations, FedSSP innovatively utilizes spectral GNNs and shares key layers in them to facilitate generic communication. Meanwhile, the authors propose preference module for the preferences derive from different graph datasets.
strength_weakness: strength_weakness: 1. Since the method of spectral encoder sharing essentially belongs to the personalized layer methods, I believe the authors need to discuss the differences between this method and other personalized layer methods in this paper, which they did not, such as APPLE.
2. How the node-level method FGSSL is used here in the experiments? What is the core model of APPLE here in the federated graph learning scenario? Could the authors provide the implementation details of the baselines?
3. The author changed the model architecture here but has not compared its communication cost with baselines.
questions: questions: Please refer to the Weakness part.
limitations: limitations: Authors should discuss communication cost for this potential limitation.
paper_summary: paper_summary: This paper focuses on cross-domain Federated Graph Learning, where graph data stored in clients exists negative domain structural shifts. Authors observe the presence of spectral biases as a reflection of structural shifts. Thereafter, this work proposes the Generic Spectral Knowledge Sharing (GSKS), which allows clients to share certain components containing generic spectral knowledge. Moreover, it proposes Personalized Graph Preference Adjustment (PGPA) to satisfy preferences of clients for more suitable local applications. Extensive experiments on different scenarios demonstrates the superiority of the proposed methods.
strength_weakness: strength_weakness: 1. Lacking of comprehensive comparative analysis of existing FGL methods. For instance, FGSSL appears in the performance table but is not comparatively analyzed in the introduction. The authors are supposed to clarify the shortcomings of this method in the given scenario.
questions: questions: I expected a more detailed comparative analysis of FGL baselines mentioned in the weakness. Could the authors provide a discussion on it?
limitations: limitations: This work solves the problems in cross-domain settings well. I wonder whether it is generalizable enough to be extended to other non-iid problems, such as each client having a portion of the same graph dataset. It’s necessary to add relevant experiments to demonstrate that this method is not limited.

Comments: comment: Dear authors,

I've read all the replies and decided to keep my scoring.
Dear Reviewer cuMU,

We highly value the time and effort you have dedicated to evaluating our work. We are fully aware of the importance of your time and strive to respect it. In light of this, we would greatly appreciate any additional feedback or confirmation that our rebuttal has effectively addressed your comments. Our goal is to ensure that we have comprehensively addressed your concerns.

Thank you so much for your time and consideration.

Authors
Dear Reviewer LD7P,

We sincerely appreciate the time and expertise you have devoted to reviewing our submission. Acknowledging the demands on your schedule, we are mindful not to intrude on your time. However, we would be grateful if you confirm that our rebuttal has addressed your concerns adequately.

Thank you in advance for your consideration.

Authors
Dear Reviewer 3kee,

We would like to express our heartfelt thanks for your consistent support. Your positive assessment of our work is greatly appreciated. Please feel free to reach out if you have any additional thoughts or suggestions.

Thank you for your consideration and valuable time.

Authors
Dear Reviewer jSYo,

Thank you very much for your prompt response and for reconsidering your evaluation. We are truly grateful for your valuable feedback and the time you have invested in our manuscript. Your comments have greatly contributed to improving our work, and we appreciate your support in advancing it through this process.

If there is anything further you believe could be refined or enhanced, please let us know.

Authors
Thank you for the detailed response. I  maintain my score.
Thanks for the rebuttal. After reading other reviewers' comments, I decided to raise my original score.