Title: FedGMark: Certifiably Robust Watermarking for Federated Graph Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Watermark

Federated Graph Learning
Abstract: Federated graph learning (FedGL) is an emerging learning paradigm to collaboratively train graph data from various clients. However, during the development and deployment of FedGL models, they are susceptible to illegal copying and model theft. Backdoor-based watermarking is a well-known method for mitigating these attacks, as it offers ownership verification to the model owner. We take the first step to protect the ownership of FedGL models via backdoor-based watermarking. Existing techniques have challenges in achieving the goal: 1) they either cannot be directly applied or yield unsatisfactory performance; 2) they are vulnerable to watermark removal attacks; and 3) they lack of formal guarantees. To address all the challenges, we propose FedGMark, the first certified robust backdoor-based watermarking for FedGL. FedGMark leverages the unique graph structure and client information in FedGL to learn customized and diverse watermarks. It also designs a novel GL architecture that facilitates defending against both the empirical and theoretically worst-case watermark removal attacks. Extensive experiments validate the promising empirical and provable watermarking performance of FedGMark. Source code is available at: https://github.com/Yuxin104/FedGMark.

FedGMark: Certifiably Robust Watermarking for Federated Graph Learning

Abstract

Federated graph learning (FedGL) is an emerging learning paradigm to collaboratively train graph data from various clients. However, during the development and deployment of FedGL models, they are susceptible to illegal copying and model theft. Backdoor-based watermarking is a well-known method for mitigating these attacks, as it offers ownership verification to the model owner. We take the first step to protect the ownership of FedGL models via backdoor-based watermarking. Existing techniques have challenges in achieving the goal: 1) they either cannot be directly applied or yield unsatisfactory performance; 2) they are vulnerable to watermark removal attacks; and 3) they lack of formal guarantees. To address all the challenges, we propose FedGMark, the first certified robust backdoor-based watermarking for FedGL. FedGMark leverages the unique graph structure and client information in FedGL to learn customized and diverse watermarks. It also designs a novel GL architecture that facilitates defending against both the empirical and theoretically worst-case watermark removal attacks. Extensive experiments validate the promising empirical and provable watermarking performance of FedGMark. Source code is available at: https://github.com/Yuxin104/FedGMark .

1 Introduction

Federated Graph Learning (FedGL) [Xie et al. , 2021; Wang et al. , 2022a; Tan et al. , 2023; Yao et al. , 2024] leverages a server and multiple clients to collaboratively train GL methods [Kipf and Welling, 2017; Hamilton et al. , 2017] via federated learning (FL) [McMahan et al. , 2017; Li et al. , 2021; Karimireddy et al. , 2020]. In recent years, FedGL has attracted increasing interest in domains such as disease prediction [Peng et al. , 2022], recommendation systems [Baek et al. , 2023; Wu et al. , 2022; Li et al. , 2022b], and molecular classification [He et al. , 2022]. In addition, several industries have deployed/open-sourced their FedGL frameworks, such as Alibaba's FederatedScope-GNN [Wang et al. , 2022b] and Amazon's FedML-GNN [Vidya et al.]. However, FedGL models are typically left unprotected, rendering them vulnerable to threats like illegal copying, model theft, and malicious distribution. For instance, a business competitor may replicate a model to gain competitive advantages or a malicious user may sell the model for profits. These threats waste the model owner's investment (e.g., labor costs, time, and energy) and infringe upon the legitimate copyrights of the model.

Backdoor-based watermarking [Uchida et al. , 2017; Adi et al. , 2018; Bansal et al. , 2022] is a de facto model ownership verification technique to mitigate the above threats. This technique typically consists of two steps: 1) Embedding the target model with a watermark. The model owner injects a specific backdoor trigger (i.e., watermark) into some clean samples and trains the target model with

this watermarked data along with the remaining clean data. Then the trained target (watermarked) model could have both high watermark accuracy (i.e., accurately classify testing data with the same watermark as the owner desires) and main task accuracy (i.e., accurately classify clean testing data). 2) Model ownership verification. When suspecting the target model is illegally used by others, the model owner can recruit a trusted third party for model ownership verification. Particularly, the true model owner knows how the target model behaves as expected by providing the trusted third party the carefully designed watermarked data, while the illegal parties cannot do so. Notice that, since all the clients have devoted computation and data to the training, they have a strong intention to jointly protect their ownership of the model.

In this paper, we aim to protect the ownership of FedGL models via backdoor-based watermarking. We observe backdoor-based watermarking methods for protecting the ownership of FL model on non-graph data [Li et al. , 2022a; Tekgul et al. , 2021; Shao et al. , 2022; Yang et al. , 2023; Lansari et al. , 2023] or centralized GL model on graph data [Xu et al. , 2023] have been recently developed. However, applying these methods for protecting FedGL models faces challenges and weaknesses.

Inapplicable or ineffective: Existing methods for non-graph data cannot be directly applied for graph data. For instance, they require input data have same size, while graphs can have varying sizes; they are unable to consider the connectivity information such as edges connecting nodes in the graph data. The only method for graph data [Xu et al. , 2023] uses a naive random graph (e.g., generated by the ER model [Gilbert, 1959]) as a watermark. Extending this watermark from centralized GL to FedGL models exhibits unsatisfactory performance , as shown in Table 1. For instance, the watermark accuracy is less than 60% in all the studied graph datasets and FedGL models. The core reason is the random graph watermark does not use any graph structure information or client information that are unique in FedGL.

Vulnerable to watermark removal attacks: They are vulnerable to existing watermark removal techniques such as distillation and finetuning [Bansal et al. , 2022] (more details in Section 2.3). For instance, as illustrated in Table 1, distillation can reduce the watermark accuracy to less than 30%.

Lack or weak formal guarantees: All these methods do not provide formal robustness guarantees against watermark removal attacks. This could make them even vulnerable to more advanced attacks. For instance, our proposed layer-perturbation attack can further reduce the watermark accuracy, e.g., perturbing only 1-layer parameters of the watermarked model yields only 10% watermark accuracy (while main accuracy is marginally affected). Bansal et al. [2022] proposed the first certified watermark for centralized non-graph learning models against l 2 model parameter perturbation. However, its certified radius is only 1.2, meaning the l 2 norm of a (usually milliondimensional) perturbation vector cannot exceed 1.2 to maintain the watermark accuracy.

We address all the above issues by proposing a certified robust backdoor-based watermark method for FedGL, called FedGMark . 2 FedGMark enjoys several properties: 1) Its designed watermarks can handle varying size graphs and utilize both graph structure and client information unique in FedGL models; 2) It is empirically robust to both existing watermark removal attacks and the proposed layerperturbation attack; and 3) more importantly, it is provably robust to the layer-perturbation attack (the layer parameters can be arbitrarily perturbed), when the number of the perturbed layers is bounded.

Z

Conv

ie[1T]

0

Ensemble classifier g = vote(A, B,.,A)

Linear

Server

Global model

f(0,Gw)

={0.,}

{…0}=0

Submodel1

(A)

Clean Graphs

Edge Encoding

GatingNet

*

>0.5

Submodel1

*+*

Submodel S

Submodel1

SubmodelS

:Watermarked

graph Gw

Submodel2

f(02,Gw)

(B)

(A)

Watermarked:

Key

SubmodelS

f(0s,G)(A)

(e.g,Client ID）Key Encoding

KeyNet

Global model (1-Layer Pert.)

Client1

ClientT

CustomizedWatermark Generator(CWG)

Robust Model Loader(RML)

CertifiedRobustness Guarantee

Specifically, as depicted in Figure 1, FedGMark consists of two modules: 1) Customized Watermark Generator (CWG) : it learns the customized watermark for individual graphs and clients in FedGL, by integrating the edge information from the client graphs and the unique key features of the clients. CWGcan significantly enhance the diversity and effectiveness of the generated watermarks. 2) Robust Model Loader (RML) . RML designs a new GL model that consists of multiple submodels, where each submodel can be any existing GL model. It also introduces a voting classifier for assembling the submodels' predictions. Such a design can facilitate deriving the certified watermark performance against the (worst-case) layer-perturbation attack.

We evaluate FedGMark on four real-world graph datasets (MUTAG, PROTEINS, DD, and COLLAB) and three FedGL models including Fed-GIN, Fed-GSAGE, and Fed-GCN, whose base GL models are GIN [Xu et al. , 2019], GSAGE [Hamilton et al. , 2017], and GCN [Kipf and Welling, 2017], respectively. Extensive experimental results show FedGMark achieves high main accuracy and watermark accuracy under no attacks and watermark removal attacks, high certified watermark accuracy, and significantly outperforms the existing method. Such good results demonstrate the potential of FedGMark as a watermarking method to protect the ownership of FedGL models.

We summarize our main contributions of this paper as follows:

To our best knowledge, this is the first work to protect the ownership of emerging FedGL models.

We propose a certifiably robust backdoor-based watermarking method FedGMark for FedGL.

Wevalidate the effectiveness of FedGMark in multiple FedGL models and real-world graph datasets under no attack, existing backdoor removal attacks, and worst-case layer-perturbation attacks.

2 PRELIMINARIES

2.1 Federated Graph Learning (FedGL)

Given a graph G = ( V , E ) as input, a GL model for graph classification learns a graph classifier f that outputs a label f ( G ) = y ∈ Y for a graph. Here, V , E , Y represent the set of nodes, edges, and labels, respectively. A ∈ { 0 , 1 } | V |×| V | is the binary adjacency matrix of G , where A [ v j , v k ] = 1 if there exists an edge between nodes v j and v k , and 0 otherwise, with | V | the total number of nodes. FedGL employs FL techniques [McMahan et al. , 2017] to collaboratively train GL models with a set of (e.g., T ) clients T = { 1 , · · · , T } and a server. Assuming each client i ∈ T has a set of graphs G i , we illustrate the training process of FedGL using the e -th epoch as an example: 1) Initially, the server distributes the global model parameters θ e to a randomly selected subset of clients T e , where T e ⊆ T . 2) Upon receiving θ e , each client i trains its local model parameter θ i e with its own graphs G i and updates its model parameters via SGD, i.e., θ i e = θ i e -1 -η∂ θ e L ( θ e ; G i ) , where L ( θ e ; G i ) represents a loss function, e.g., cross-entropy loss. After training, client i submits its update model parameters θ i e to the server. 3) The server aggregates local model parameters of the selected clients i.e., { θ i e : i ∈ T e } and updates the global model parameter, e.g., θ e +1 = 1 | T e | ∑ i ∈ T e θ i e via the average aggregation [McMahan et al. , 2017], for the next epoch. This iterative process continues until the global model converges or reaches the maximum number of epochs.

2.2 Backdoor-based Watermarking for GL

Backdoor-based watermarking [Uchida et al. , 2017; Adi et al. , 2018; Bansal et al. , 2022; Xu et al. , 2023] adopts the idea of backdoor attack [Bagdasaryan et al. , 2020; Wang et al. , 2020; Saha et al. , 2020] from the adversarial realm to facilitate model ownership verification. To watermark the GL

model, assume the model owner has a set of clean graphs G and selects a subset of graphs G w ⊂ G to inject the watermark. In the existing method [Xu et al. , 2023], the model owner first generates a random graph (e.g., via the ER-model) as the watermark for each to-be-watermarked graph. For instance, for a graph G ∈ G w with label y , the generated random graph is G s (its size is often smaller than G ). The owner then attaches G s to G to produce the watermarked graph G w , where nodes in G are randomly chosen and the edge status of these nodes are replaced by edges in G s . Finally, the owner assigns a desired label different from y to G w . The watermarked graphs together with the clean graphs are used to train the GL model. During model ownership verification, the one who can predict a high accuracy on these watermarked graphs can claim to be the model owner.

We note this method can be extended to watermark FedGL models, where each client can generate its own random graphs as the watermark and train its local model with the watermarked graphs and clean graphs. The server then aggregates the watermarked local models to update the global model.

2.3 Watermark Removal Attacks

We consider three possible watermark removal attacks aiming to infringe the FedGL model ownership: distillation and finetuning from [Shafieinejad et al. , 2021], and our proposed layer-perturbation attack. In all attacks, the attacker (e.g., malicious user) is assumed to know the target watermarked model.

1) Distillation. This attack has access to some unlabeled data sampled from the same data distribution. To remove watermarks without affecting the target model's main task performance, the attacker uses the unlabeled data to distill the target model during training. Specifically, the attacker initializes its model with the target model and labels the unlabeled data by querying the target model. The attacker's model is then updated with these unlabeled data and their predicted labels.

2) Finetuning. This attack assumes the attacker has some labeled data. The attacker then leverages the labeled data to further finetune the target model in order to forget the watermark. This attack is shown to pose a greater threat than the distillation attack [Bansal et al. , 2022].

3) Layer-perturbation attack. This attack also assumes the attacker has some labeled data. As knowing the target watermarked model (and hence the architecture), the attacker can mimic training an unwatermarked model with the same architecture as the target model using the labeled data. To further test the model robustness, we assume the attacker also knows some true watermarked samples, similar to [Jiang et al. , 2023]. Then, the attacker can replace any layer(s)' parameters of the target model with those from the unwatermarked model to maximally reduce the watermark accuracy on its watermarked samples, while maintaining the main task performance. Our results (e.g., in Table 1) show this layer-perturbation attack (even only perturbing 1 layer parameters) is much more effective than the other two attacks (even though the whole model parameters can be perturbed).

2.4 Threat Model

We follow existing methods [Shafieinejad et al. , 2021; Bansal et al. , 2022; Xu et al. , 2023; Jiang et al. , 2023], where the adversary is assumed to know all details of the pretrained watermarked FedGL model, but does not tamper with the training process. This means all clients and the server are benign and follow the federated training protocol, and the attack happens at the testing/inference time. We highlight this is in stark contrast to the training-time Byzantine attack on FL where some clients are malicious and they manipulate the training process.

Attacker's knowledge. The attacker has white-box access to the pretrained watermarked FedGL model. In addition, the attacker may also know some clean (unlabeled or labeled) training data, as well as watermarked data. Note that this setting actually makes our defense design the most challenging. If the defense can successfully defend against the strongest white-box attack on the watermarked FedGL model, it will also be effective against weaker attacks, such as black-box attacks.

Attacker's capability. The attacker can modify the pretrained model via leveraging its white-box access to the trained model and its hold training and watermarked data. For instance, the attacker can finetune the pretrained model via the labeled training data. More details of the capabilities of considered attacks are described in Section 2.3.

Attacker's goal. The attacker aims to remove the watermark based on its knowledge and capability, while maintaining the model utility. This allows it to illegally use the model without detection.

3 FedGMark: Our Certified Robust Watermark for FedGL

3.1 Motivation and Overview

Recall that the random graph based watermark is unable to ensure high watermark accuracy for protecting FedGL (as shown in Table 1). This is because such random watermark does not use any graph structure or client information during FedGL training. Our results also show this method is vulnerable to the three watermark removal attacks. These weaknesses inspire us to design a more effective and robust watermarking method specially for FedGL model ownership verification.

We propose FedGMark, the first certified robust backdoor-based watermarking method for FedGL. FedGMark comprises two main components: Customized Watermark Generator (CWG) and Robust Model Loader (RML) (as depicted in Figure 1). The CWG module utilizes the unique property of each client, as different clients could have different properties (e.g., distributions of their graph data) and their optimal watermark could be different. Particularly, CWG learns the customized watermark for each graph using its structure information, and outputs a set of diversified watermarked graphs for each client. Further, inspired by existing GNNs [Xu et al. , 2019], the RML module designs a new GL model that consists of multiple submodels, each being any existing GL model. It also introduces a voting classifier for aggregating the prediction results from the submodels. Under this design, FedGMark can be proved to be certified robust against the worst-case layer-perturbation attack, once the number of perturbed layers is bounded. The model owner (e.g., participating clients in FedGL) adopts the designed GL model to train the local watermarked model with the learnt watermarked graphs and the remaining clean graphs. After the server-client training terminates, the ownership of the trained FedGL model can be verified via measuring its performance on a set of testing graphs injected with the global watermark , which is the integration of all clients' local watermarks.

3.2 Customized Watermark Generator (CWG)

CWG consists of two networks: GatingNet and KeyNet . GatingNet designs the watermark for each graph separately using the edge information, while KeyNet learns client-wise watermarking style using predefined keys (e.g., client ID in this paper). The customized watermark for each client's graph is then decided using the output of GatingNet and KeyNet . Detailed network architectures of CWG can be seen in Table 7 in Appendix C. We demonstrate how CWG can learn a customized watermark using a graph G i = ( V i , E i ) from client i as an instance. The details are as follows:

We first randomly select n w nodes V i w = { v 1 , · · · , v n w } from V i as watermark nodes and construct a corresponding mask matrix M i ∈ { 0 , 1 } | V i |×| V i | such that M i [ v j , v k ] = 1 if v j , v k ∈ V i w , and 0 otherwise. We also update the adjacency matrix A i of G i according to V i w , i.e., setting A i [ v j , v k ] = 0 , ∀ v j , v k ∈ V i w . This allows us focus on learning the edge status between watermarked nodes.

Given the client i 's ID string k i , we utilize a cryptographic hash function, such as MD5, to convert it into an integer (e.g., 128 -bit long with the integer range [0 , 2 128 -1] ). This integer is then employed as a seed to produce a key matrix K i ∈ R | V i |×| V i | . Then, we employ GatingNet and KeyNet to extract edge features and key features, resulting in ˜ A i = GatingNet( A i ) ∈ [0 , 1] | V i |×| V i | and ˜ K i = KeyNet( K i ) ∈ [0 , 1] | V i |×| V i | , respectively.

We finally learn the customized watermark for G i by integrating ˜ A i , ˜ K i , and M i , and obtain the corresponding watermarked graph as G i w = ( V i , E i w ) . Here E i w is the set of edges according to the updated adjacency matrix A i ⊕ W i , where ⊕ is the element-wise addition and W i = I (( ˜ A i ⊙ ˜ K i ) > 0 . 5) ⊙ M i contains the edge status between the watermark nodes V i w . Here, ⊙ is the element-wise product, I ( p ) is an indicator function returning 1 if p is true, and 0 otherwise. We adopt 0 . 5 as a threshold to decide the presence of edges between watermarked nodes.

3.3 Robust Model Loader (RML)

This module aims to design a new GL model that is provably robust to the layer-perturbation attack. Towards this end, we design a GL model architecture to incorporate multiple submodels; and devise a majority voting-based ensemble classifier on top of the predictions of these submodels.

Architecture of the proposed GL model. Intuitively, each client can take a base GL model (e.g., GIN [Xu et al. , 2019]) and split it according to the layer indexes to obtain multiple submodels. For instance, a 8 -layer GIN can be represented with layer indexes { l 1 , · · · , l 8 } . Splitting this GIN into 4 submodels { GIN 1 , · · · , GIN 4 } with layer indexes { l 1 , l 2 } , · · · , { l 7 , l 8 } means GIN i contains layers

{ l 2 i -1 , l 2 i } , from the GIN. However, submodels splitted in this way are coupled from each other, making them unable to defend against layer-perturbation attacks. To tackle this problem, we design the novel GL model θ that is an ensemble of a set of S independent submodels { ϑ 1 , ϑ 2 , · · · , ϑ S } , where each submodel ϑ i is a base GL model. This approach can hence be easily adapted to any existing FedGL. Further, to prevent homogeneity, we define varying channels for these submodels to diversify them. Details of the model architecture are shown in Table 8 in Appendix C.

A majority-voting based ensemble classifier. The designed GL model architecture inspires us to leverage the idea of ensemble classifier, which can combine the predictions of base 'weak' classifiers. Specifically, we propose a majority voting-based ensemble classifier to combine the predictions of the submodels. Given a testing graph G and a graph classifier f , we denote the prediction of the submodel ϑ i for G as y = f ( ϑ i , G ) ∈ Y . For a GL model θ with S submodels { ϑ 1 , ϑ 2 , · · · , ϑ S } , we can count the submodels that classify G to be y as N y = ∑ S i =1 ■ ( f ( ϑ i , G ) = y ) . Then we introduce our majority-voting based ensemble classifier g to classify G as: g ( θ, G ) = arg max y ∈ Y N y . In cases of ties, our ensemble classifier g selects the label with a smaller index.

3.4 Training the Proposed FedGL Model

The overall training process consists of three iterative steps: 1) training the proposed GL model in all clients; 2) training the CWG module in watermarked clients , i.e., the clients that aim to inject watermarked graphs for protecting the model ownership; and 3) aggregating the clients' GL models to produce the target watermarked model. The final global model is the learnt watermarked FedGL model. Details of training can be seen in Algorithm 1 in the Appendix.

Step 1: Training the proposed GL model. Assume we have T w watermarked clients with indexes [1 , T w ] . For each watermarked client i , we split its training graphs G i into the watermarked graphs G i w with a target label, say y w , and remaining clean graphs G i c , and then customize the watermark for each graph in G i w using the CWG module (see Step 2 ). Given the client's GL model θ i with S submodels { ϑ i 1 , ϑ i 2 , · · · , ϑ i S } , we train each submodel ϑ i j via minimizing the loss on G i c and G i w , i.e., ϑ i j = arg min ϑ i j L ( ϑ i j ; G i c ⋃ G i w ) . For an unwatermarked client k ∈ [ T w +1 , T ] , we utilize all clean graphs G k to train each submodel ϑ k j separately, i.e., ϑ k j = arg min ϑ k j L ( ϑ k j ; G k ) .

Step 2: Training the CWG. We denote the parameters of the CWG module for a watermarked client i ∈ [1 , T w ] , as ω i . The parameters include two networks, GatingNet and KeyNet . Each client i trains its CWG ω i to ensure that the generated watermarks be effective and diverse. Formally, we have ω i = arg min ω i L ( θ i ; G i w ) , i ∈ [1 , T w ] .

Step 3: Aggregating clients' GL models. The server averages GL models { θ i } i ∈ T to produce the global model θ , and distributes this model to selected clients in the next iteration.

3.5 Model Ownership Verification

When suspecting the target FedGMark model θ is illegally used by others, the model owner (all the participating clients or their representative) can recruit a trusted judge for model ownership verification. Typically, the judge requests both the true model owner and the illegal party to provide some test data for verification. Only when the one knows the predictions by the target model for the provided test data by both parties, the judge will confirm this party the model ownership. In particular, besides providing the clean data G i c by both parties that behave normally, the true model owner especially provides the designed watermarked data G i w that only s/he knows the model behaves on. As a result, both parties know the prediction results on G i c , but the illegal party is hard to predict accurately on G i w provided by the true model owner.

3.6 Certified Robustness Guarantees against Layer-Perturbation Attacks

We show the above design, with any layer-perturbation attack, ensures the predictions of the learnt watermarked FedGL model and its compromised counterpart for the watermarked graphs are consistent, once the number of perturbed layers is bounded. Given the target watermarked FedGL model θ and its S submodels { ϑ 1 , · · · , ϑ S } , we denote θ ′ as the comprised model and { ϑ ′ 1 , · · · , ϑ ′ S } as its S submodels. For each watermarked graph G w , we use the ensemble classifier g on submodels' predictions, i.e., its predictions on θ and θ ′ are g ( θ, G w ) = arg max y ∈ Y N y , and g ( θ ′ , G w ) = arg max y ∈ Y N ′ y , respectively, where N y = ∑ S i =1 ■ ( f ( ϑ i , G ) = y ) and N ′ y = ∑ S i =1 ■ ( f ( ϑ ′ i , G ) = y ) . Then we have the following result on guaranteeing the number of perturbed layers on the target watermarked model.

Theorem 1 (Certified number of perturbed layers r .) Let θ , θ ′ , g , and G w be above defined. Suppose N A and N B are the largest and second largest count outputted by g on G w , For any layerperturbation attack, we have g ( θ, G w ) = g ( θ ′ , G w ) , when the number of perturbed layers r satisfies:

■ We also show the tightness of our derived r ∗ in the following theorem:

■ where [ · ] is the indicator function and r is called the certified number of perturbed layers.

Theorem 2 (Tightness of r ∗ .) Without using extra information of f , our derived r ∗ in Theorem 1 is tight. I.e., r ∗ is the maximum number of perturbed layers tolerated by our target watermarked model.

The proofs of Theorems 1 and 2 are deferred to Appendix A.

4 Experiments

In this section, we comprehensively evaluate FedGMark on multiple datasets, FedGL models, attack baselines, and experimental settings. More experimental results and discussions are in Appendix.

4.1 Experimental Setup

Datasets and models. We evaluate our FedGMark on four real-world graph datasets for graph classification: MUTAG [Debnath et al. , 1991], PROTEINS [Borgwardt et al. , 2005], DD [Dobson and Doig, 2003], and COLLAB [Yanardag and Vishwanathan, 2015]. Details about the statistics of those datasets are shown in Table 6 in Appendix C. Following prior work Shen et al. [2022]; Xia et al. [2024], we choose the well-known GIN [Xu et al. , 2019], GSAGE [Hamilton et al. , 2017], and GCN [Kipf and Welling, 2017] as the GL model. All these network architectures involved in the experiments are detailed in Table 8 in Appendix C.

Parameter setting. We implement our method using one NVIDIA GeForce GTX 1080 Ti GPU. In FedGL, we use T = 40 clients in total and train the model 200 iterations. The server randomly selects 50% clients in each iteration. We define the target label of watermarking graphs as 1 , and each participating client randomly selects 10% graphs with labels not 1 as the watermarking graphs. We extensively validate the effectiveness and robustness of the proposed watermarking method with the following hyperparameters details: the number of submodels S = { 4 , 8 , 16 } , the number watermarked clients T w = { 5 , 10 , 20 } (both S and T w are halved on MUTAG due to less data), the watermarked nodes n w = { 3 , 4 , 5 } , and the number of perturbed layers r = { 1 , · · · , 5 } in the layer-perturbation attack. By default, we set S = 4 , T w = 10 , n w = 4 , r = 1 . While studying the impact of a hyperparameter, we fix the others as the default value.

Evaluation metric. We use three metrics for evaluation: the main task accuracy (MA), watermark accuracy (WA), and certified WA (CWA @ r ). An effective and robust watermarked model is expected to achieve both high MA and WA. CWA evaluates the certified robustness of FedGMark against layer-perturbation attacks. CWA @ r is defined as the fraction of testing graphs that are provably predicted as the target label, when at most r layers in client models can be arbitrarily perturbed.

Attack baselines. We evaluate FedGMark against existing watermark removal attacks including distillation and finetuning, and our proposed layer-perturbation attack. In our setting, the layerperturbation attack can replace any layer(s)' parameters of the target watermarked model with those from the unwatermarked model to maximally reduce the watermark accuracy on its watermarked graphs. When perturbing multiple layers, we utilize a greedy algorithm to decide the perturbed layers-we search for one optimal perturbed layer at each step. Specifically, we first traverse perturbing layers in the watermarked model and find the one that maximally reduces the watermark accuracy. We then search for the remaining layers based on this one and continue the process.

4.2 Empirical Results: MA and WA

4.2.1 Results under the Default Setting

We first assess our FedGMark against empirical and layer-perturbation attacks under the default setting. Experimental results are provided in Table 2. We have the following observations:

1) Our FedGMark significantly outperforms the existing method under no attack. Recall in Table 1 that [Xu et al. , 2023] obtains < 60% WAs across all datasets and FedGL models. In contrast, FedGMark can achieve WAs > 70% in almost all cases, while having similar MAs.

clienti

clientj

clienti

clientj

clienti

clientj

clienti

clientj

MUTAG

PROTEINS

DD

COLLAB

Such significant improvements confirm the superiority of our learnt watermarks over the random watermarks in [Xu et al. , 2023]. Figure 2 also visualizes example learnt watermarks and we can see these watermarks are diversified due to the proposed CWG.

2) Our FedGMark exhibits resistance to existing empirical attacks. As shown in Table 1, existing methods are vulnerable to distillation and finetuning attacks. In contrast, the WAs of FedGMark under these two attacks are almost the same as those under no attack, demonstrating FedGMark is robust to the existing attacks.

3) Our FedGMark is resilient to the proposed layer perturbation attack. We notice that the existing method has difficulties in defending against the proposed layer-perturbation attacks and shows an unsatisfactory WA, e.g., < 25% WAin almost all cases under the 1-layer perturbation attack. Conversely, our FedGMark can obtain a close WA compared to that without attack. This verifies our ensemble classifier in FedGMark is capable of resisting the 1-layer perturbation attack.

4.2.2 Impact of Hyperparameters

This section studies the impact of hyperparameters in FedGMark against the layer-perturbation attack.

Impact of #submodels S . We first examine the impact of S on the performance of FedGMark and report the results against the 1-layer perturbation attack in Table 3. We observe that the learnt watermarked model can resist to the 1-layer perturbation attack under all S and both the MAs and WAs are also similar. This indicates the number of submodels marginally affects the watermarking performance against the 1-layer perturbation attack.

Fed-GIN: CWA@r

1.0

None

..None

AS=8

S=4

S=8

S=4

S=16

S=16

None

None

S=8

S=4

S=8

S=4

0.0

S=16

S=16

Fed-GSAGE:CWA@r

None

.None

S=4

S=4

S=8

S=8

S=16

S=16

None

None

S=4

S=4

S=8

S=8

0.0

S=16

S=16

Fed-GCN: CWA@r

None

None

+

S=8

S=4

S=8

S=4

S=16

S=16

None

S=4

S=8

0.0

012345678

9

012345678

9

012345678

9

0123456789

#Perturbed Layers:r

#Perturbed Layers:r

#Perturbed Layers:r

#Perturbed Layers:r

MUTAG

PROTEINS

DD

COLLAB

Impact of #watermarking clients T w . The results on T w = 5 and T w = 20 are shown in Figure 4 in Appendix. We see that: the larger number of clients generating watermarks, the better our watermarking performance. For instance, the WA improves by 11% , 9% , and 7% , respectively on the three FedGL models on PROTEINS when T w increases from 5 to 20 . This is because more watermark clients (thus more watermarked graphs) can ensure the FedGL model better learns the relation between the watermarks and the target label.

Impact of the watermark size n w . We also investigate n w = 3 and n w = 5 , and the results are presented in Figure 5 in Appendix. Similarly, FedGMark obtains higher WA with larger n w . This is because a larger watermark size can facilitate the trained FedGL model better learn the watermark.

Impact of #perturbed layers. Finally, Table 4 shows the results of FedGMark vs. different #perturbed layers. We observe that: 1) The WA decreases as increasing the #perturbed layers. This is because the attacker has more attack capability by perturbing more layers. 2) Despite the theoretically guaranteed CWA being 0 at #perturbed layers ≥ 3 as later depicted in the Figure 3, our empirical watermarking performance can still achieves WA from 48% to 58% . Note that the change in MA is < 4% in all cases, compared with Table 2.

4.2.3 Ablation Study

We investigate the contribution of each module in FedGMark, including the GatingNet and KeyNet in CWG, and the submodels used in RML. The results under the default setting and no attack are shown in Table 5. First, GatingNet plays a crucial role on generating more

effective watermarks and results in an improvement in WA from 8% to 21% . Second, KeyNet facilitates watermark differentiation between clients, leading to an improve on WA from 5% to 8%. Third, the submodels in RML has a negligible impact on WA/MA. However, it is important to defend against the layer-perturbation attack, as shown in Section 4.3.

4.3 Certified Robustness Results: Certified WA

In this section, we evaluate the certified robustness of FedGMark against the layer perturbation attack. The CWAs on the three FedGL models and four datasets are depicted in Figure 3. We have several observations: 1) FedGMark achieves promising provable robustness results against the worst-case layer perturbation attack, when the #perturbed layers is within the certified range in Eqn (1). For instance, when S = 4 and 0 < r ≤ 2 , CWA is close to WA without attack ( r = 0 ). 2) As S increases, the certified #perturbed layers also increases, showing that a more number of submodules in RML can better provably defend against the layer-perturbation attack. For instance, when S = 16 , the CWA is 86% on MUTAG even when 5 any layers in the global model are arbitrarily perturbed, while it is 0 when S = 4 . However, this is at the cost of requiring more computational resources (e.g., GPU memory, runtime). As shown in Table 10 in Appendix, the runtime is linearly to S .

5 Related Work

Backdoor-based watermarking for centralized models on non-graph data. Many backdoor-based watermarking methods [Lv et al. , 2023; Yan et al. , 2023; Tekgul et al. , 2021; Yang et al. , 2023; Li et al. , 2022a; Shao et al. , 2022; Lansari et al. , 2023] have been proposed that can empirically protect the model ownership. These methods mainly focus on centralized learning models on non-graph (e.g., image) data. Compared with non-graph data, graph data have unique graph structure information, e.g., entities are connected by links. Similarly, compared with centralized models, FL models are collaboratively trained by multiple clients, which could have their own uniqueness. Bansal et al. [2022] is the first watermarking method for centralized non-graph models with certified guarantee. However, its certified robustness performance is unsatisfactory.

Backdoor-based watermarking for FL models on non-graph data. A few recent works Tekgul et al. [2021]; Li et al. [2022a]; Bansal et al. [2022] design backdoor-based watermarks for protecting the ownership of FL models on non-graph data, where the watermark can be injected into client's data or server's validation data. For instance, Tekgul et al. [2021] presented WAFFLE, an approach to watermark DNN models by incorporating a re-training step via watermarked data at the server. Li et al. [2022a] leveraged each client's private watermark to verify FL model ownership, ensuring non-conflicting watermarks across different clients. However, these techniques cannot be directly applied to graph data, as they often require fixed input data, while graph data often have varying sizes. Moreover, all these methods do not provide guaranteed watermarking performance under the attack.

Backdoor-based watermarking for centralized GL models on graph data. The only work [Xu et al. , 2023] handling graph data uses random graph as the watermark. This method can be extended to the FedGL model, but its watermarking performance is far from satisfactory-especially vulnerable to watermark removal attacks such as distillation, finetuning, and the layer perturbation attack.

Backdoor attacks for centralized and federated GL models on graph data. Several works [Zhang et al. , 2021; Yang et al. , 2024] design backdoor attacks to manipulate GL models, enabling the attacker to influence the learned model to serve its purpose-the model will predict the attacker-chosen label for test graphs once they contain a predefined trigger. For instance, Zhang et al. [2021] uses random subgraphs as the trigger to backdoor centralized GL models, while Yang et al. [2024] learns subgraph trigger to backdoor FedGL models. Note that the goal of these works is orthogonal to ours.

6 Conclusion

We protect the model ownership of emerging FedGL trained on distributed graph data, and use the de facto backdoor-based watermarking method. We develop the first certifiably robust backdoor-based watermarking method FedGMark for FedGL. FedGMark demonstrates the capability of achieving high empirical watermarking performance under no attack, under existing backdoor removal attacks and the proposed stronger layer-perturbation attack. FedGMark is also provably robust against the worst-case layer-perturbation attack, once the number of perturbed layers is bounded by Theorem 1.

Acknowledgments. We thank all anonymous reviewers for the constructive comments. Li is partially supported by the National Natural Science Foundation of China under Grant No. 62072208, Key Research and Development Projects of Jilin Province under Grant No. 20240302090GX. Hong is partially supported by the National Science Foundation under grant No. CNS-2302689, CNS2308730, CNS-2319277 and CMMI-2326341. Wang is partially supported by the National Science Foundation under grant No. ECCS-2216926, CNS-2241713, CNS-2331302 and CNS-2339686.

References

Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX Security Symposium (USENIX Security 18) , pages 1615-1631, 2018.

Jinheon Baek, Wonyong Jeong, Jiongdao Jin, Jaehong Yoon, and Sung Ju Hwang. Personalized subgraph federated learning. In International Conference on Machine Learning , pages 1396-1415. PMLR, 2023.

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International conference on artificial intelligence and statistics , pages 2938-2948. PMLR, 2020.

Arpit Bansal, Ping-yeh Chiang, Michael J Curry, Rajiv Jain, Curtis Wigington, Varun Manjunatha, John P Dickerson, and Tom Goldstein. Certified neural network watermarks with randomized smoothing. In International Conference on Machine Learning , pages 1450-1465. PMLR, 2022.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems , 30, 2017.

Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics , 21(suppl_1):i47i56, 2005.

Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry , 34(2):786-797, 1991.

Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology , 330(4):771-783, 2003.

Edgar N Gilbert. Random graphs. The Annals of Mathematical Statistics , 30(4):1141-1144, 1959.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Chaoyang He, Emir Ceyani, Keshav Balasubramanian, Murali Annavaram, and Salman Avestimehr. Spreadgnn: Decentralized multi-task federated learning for graph neural networks on molecular data. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 68656873, 2022.

Zhengyuan Jiang, Minghong Fang, and Neil Zhenqiang Gong. Ipcert: Provably robust intellectual property protection for machine learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3612-3621, 2023.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Mohammed Lansari, Reda Bellafqira, Katarzyna Kapusta, Vincent Thouvenot, Olivier Bettan, and Gouenou Coatrieux. When federated learning meets watermarking: A comprehensive overview of techniques for intellectual property protection. Machine Learning and Knowledge Extraction , 5(4):1382-1406, 2023.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International conference on machine learning , pages 63576368. PMLR, 2021.

Bowen Li, Lixin Fan, Hanlin Gu, Jie Li, and Qiang Yang. Fedipr: Ownership verification for federated deep neural network models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4521-4536, 2022.

Zheng Li, Muhammad Bilal, Xiaolong Xu, Jielin Jiang, and Yan Cui. Federated learning-based cross-enterprise recommendation with graph neural networks. IEEE Transactions on Industrial Informatics , 19(1):673-682, 2022.

Peizhuo Lv, Pan Li, Shengzhi Zhang, Kai Chen, Ruigang Liang, Hualong Ma, Yue Zhao, and Yingjiu Li. A robustness-assured white-box watermark in neural networks. IEEE Transactions on Dependable and Secure Computing , 2023.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Liang Peng, Nan Wang, Nicha Dvornek, Xiaofeng Zhu, and Xiaoxiao Li. Fedni: Federated graph learning with network inpainting for population-based disease prediction. IEEE Transactions on Medical Imaging , 2022.

Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11957-11965, 2020.

Masoumeh Shafieinejad, Nils Lukas, Jiaqi Wang, Xinda Li, and Florian Kerschbaum. On the robustness of backdoor-based watermarking in deep neural networks. In Proceedings of the 2021 ACM workshop on information hiding and multimedia security , pages 177-188, 2021.

Shuo Shao, Wenyuan Yang, Hanlin Gu, Zhan Qin, Lixin Fan, Qiang Yang, and Kui Ren. Fedtracker: Furnishing ownership verification and traceability for federated learning model. arXiv preprint arXiv:2211.07160 , 2022.

Yun Shen, Xinlei He, Yufei Han, and Yang Zhang. Model stealing attacks against inductive graph neural networks. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1175-1192. IEEE, 2022.

Yue Tan, Yixin Liu, Guodong Long, Jing Jiang, Qinghua Lu, and Chengqi Zhang. Federated learning on non-iid graphs via structural knowledge sharing. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 9953-9961, 2023.

Buse GA Tekgul, Yuxi Xia, Samuel Marchal, and N Asokan. Waffle: Watermarking in federated learning. In 2021 40th International Symposium on Reliable Distributed Systems (SRDS) , pages 310-320. IEEE, 2021.

Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on international conference on multimedia retrieval , pages 269-277, 2017.

Vidya et al. Fedml supports several out-of-the-box deep learning algorithms for various data types, such as tabular, text, image, graphs, and internet of things (iot) data. https://aws.amazon.com/blogs/machine-learning/ part-2-federated-learning-on-aws-with-fedml-health-analytics-without-shar ing-sensitive-data/ .

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems , 33:1607016084, 2020.

Binghui Wang, Ang Li, Meng Pang, Hai Li, and Yiran Chen. Graphfl: A federated learning framework for semi-supervised node classification on graphs. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 498-507. IEEE, 2022.

Zhen Wang, Weirui Kuang, Yuexiang Xie, Liuyi Yao, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-gnn: Towards a unified, comprehensive and efficient package for federated graph learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4110-4120, 2022.

Peter Wills and François G Meyer. Metrics for graph comparison: a practitioner's guide. Plos one , 15(2):e0228728, 2020.

Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Tao Qi, Yongfeng Huang, and Xing Xie. A federated graph neural network framework for privacy-preserving personalization. Nature Communications , 13(1):3091, 2022.

Zaishuo Xia, Han Yang, Binghui Wang, and Jinyuan Jia. Gnncert: Deterministic certification of graph neural networks against adversarial perturbations. In International Conference on Learning Representations , 2024.

Han Xie, Jing Ma, Li Xiong, and Carl Yang. Federated graph classification over non-iid graphs. Advances in neural information processing systems , 34:18839-18852, 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

Jing Xu, Stefanos Koffas, O˘ guzhan Ersoy, and Stjepan Picek. Watermarking graph neural networks based on backdoor attacks. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P) , pages 1179-1197. IEEE, 2023.

Yifan Yan, Xudong Pan, Mi Zhang, and Min Yang. Rethinking { White-Box } watermarks on deep learning models under neural structural obfuscation. In 32nd USENIX Security Symposium (USENIX Security 23) , pages 2347-2364, 2023.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1365-1374, 2015.

Wenyuan Yang, Shuo Shao, Yue Yang, Xiyao Liu, Ximeng Liu, Zhihua Xia, Gerald Schaefer, and Hui Fang. Watermarking in secure federated learning: A verification framework based on client-side backdooring. ACM Transactions on Intelligent Systems and Technology , 15(1):1-25, 2023.

Yuxin Yang, Qiang Li, Jinyuan Jia, Yuan Hong, and Binghui Wang. Distributed backdoor attacks on federated graph learning and certified defenses. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security , 2024.

Yuhang Yao, Weizhao Jin, Srivatsan Ravi, and Carlee Joe-Wong. Fedgcn: Convergencecommunication tradeoffs in federated training of graph convolutional networks. Advances in Neural Information Processing Systems , 36, 2024.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International conference on machine learning , pages 5650-5659. Pmlr, 2018.

Zaixi Zhang, Jinyuan Jia, Binghui Wang, and Neil Zhenqiang Gong. Backdoor attacks to graph neural networks. In SACMAT , 2021.

A Proofs

Proof of Theorem 1: Given a watermarked graph G w , a base graph classifier f , our ensemble classifier g , and the target watermarked FedGL model θ = { ϑ i } S i =1 . Let θ ′ be the watermarked FedGL model after the layer-perturbation attack. We denote by y = f ( ϑ i , G w ) the prediction of G w by submodel i , and g ( θ, G w ) = arg max y ∈ Y N y the prediction of G w by the target model θ , where N y = ∑ S i =1 ■ ( f ( ϑ i , G w ) = y ) and θ = { ϑ i } S i =1 . Assume N A and N B are the largest and second largest votes of the output of the ensemble classifier g on the target model θ and G w , respectively. Similarly, let N ′ A and N ′ B respectively denote the corresponding votes of the model θ ′ . Under our GL model architecture, each perturbed layer affects at most 1 submodel in the worst case. Therefore, N ′ A and N ′ B satisfy the following equations when r layers in θ are perturbed:

Thus, g ( θ ′ , G ) = g ( θ, G ) when N ′ A > N ′ B , This means N A -r > N B + r -✶ ( A < B ) , where ■ ( A < B ) indicates we select a label with a smaller index in case of ties. Hence, r satisfies:

̸

✶ Or N ′ A ≤ N ′ B . Hence, g ( θ, G w ) = A = B = g ( θ ′ , G w ) , if A > B .

B Algorithm 1

Algorithm 1 The training process of FedGMark

Input: clients [1 , T ] , watermarked clients [1 , T w ] , training graphs G i ( i ∈ [1 , T ] ), initial global model θ 1 , initial CWG ω 1 , #Perturbed Layers r , unique key k i for client i ( i ∈ [1 , T w ] ), the number of submodels S .

Output: Target global model θ epoch , and target local models θ j epoch ( j ∈ [1 , T w ]) .

Proof of Theorem 2: We use proof by contradiction, i.e., when r > r ∗ , we can build a base classifier f ′ such that there exists g ( θ, G w ) = g ( θ ′ , G w ) . We select the smallest r = r ∗ +1 for simplicity, indicating that r ∗ +1 layers whose model parameters can be arbitrarily perturbed. Assume there exist a graph classifier f ′ under which, ( r ∗ +1) submodels whose original predictions on G w are the label A and now become B after r ∗ +1 layers are perturbed. Then we have N ′ A = N A -( r ∗ +1) and N ′ B = N B +( r ∗ +1) . Since r ∗ is the largest number such that Equation (2) is satisfied, i.e., N A -r ∗ > N B -✶ ( A < B ) + r ∗ holds. Thus, N A -( r ∗ +1) ≤ N B -( A < B ) + ( r ∗ +1) . (4)

̸

C More Experimental Details

Table 6 shows the statistics of the used graph datasets. Table 7 shows the network architectures of the CWGcomponent. Table 8 shows the detailed network architectures of Fed-GIN, Fed-GSAGE, and Fed-GCN models.

1MA

WA

Fed-GIN

Fed-GSAGE

Fed-GCN

MA / WA

7

MA / WA

1

0

Tw=3:None Distillation

Finetuning Tw=10: None Distillation

Finetuning

Tw= 5:None Distillation

Finetuning Tw= 20: None Distillation

Finetuning

(a)MUTAG

(b)PROTEINS

MA / WA

1

MA / WA

1

0

Tw=5:NoneDistillation

Finetuning Tw=20: None Distillation

Finetuning

Tw=5:NoneDistillation

Finetuning Tw= 20: None Distillation

Finetuning

(c)DD

(d)COLLAB

MA

WA

Fed-GIN

Fed-GSAGE

Fed-GCN

WA

MA / WA

1

MA/

0

nw=3:None Distillation

Finetuning nw=5:None Distillation

Finetuning

nw=3:NoneDistillation

Finetuning nw=5:None Distillation

Finetuning

(a)MUTAG

(b)PROTEINS

MA / WA

1

MA / WA

1

1888

0

0

nw=3:None Distillation

Finetuning nw= 5: None Distillation

Finetuning

nw=3:NoneDistillation

Finetuning nw= 5:None Distillation

Finetuning

(c)DD

(d)COLLAB

D More Experimental Results

Impact of #watermarking clients T w and watermark size n w . See results in Figure 4 and Figure 5, respectively. We can see FedGMark achieves better watermarking performance with a larger number of clients generating watermarks; and a larger watermark size.

Impact of learning rate ( lr ) and #local epochs ( le ). The results with varying lr and le are shown in Table 9 in Appendix. We can see that a large lr may reduce WA, and WA increases slightly as le grows, indicating more thorough training makes our method perform better.

Scalability of FedGMark. Compared with graph-based or non-robust watermarking methods, the computation overhead of FedGMark is mainly from the introduced submodel models (fixing all the other parameters, such as #clients, #iterations, to be the same). Particularly, the overhead scales linearly with the number of submodels S , and the runtime results are shown in Table 10.

FedGMark under watermark/backdoor detection attacks. Many existing works [Zhang et al. , 2021] show the trigger-reverse based backdoor detection is ineffective to 'stealthy' backdoor. This is because the effectiveness of trigger reverse attacks largely depends on the statistical differences between clean data and backdoored data. Since we do not notice any graph backdoor trigger-reverse attack, we instead propose to quantitatively test the structure similarity between the generated watermarked graphs and the clean graphs. Here we use the metrics NetSim and DeltaCon proposed in [Wills and Meyer, 2020], with the range [0 , 1] and the higher value the larger similarity. As shown in Table 11, we observe the watermarked graphs and their clean counterparts are structurally very close. This implies that the proposed watermarks are hard to be detected.

FedGMark on non-IID/heterogeneous datasets. Recall that the local watermarks in FedGMark are learnt by considering the unique properties in each client. Such unique properties may include the heterogeneity across clients' data. To validate this, we also test FedGMark with non-IID graphs across clients and show results in Table 12, where each client holds a single label data. We observe that FedGMark demonstrates strong performance on non-IID datasets, indicating that the learnt customized watermarks effectively capture the heterogeneity of clients' graphs.

FedGMark against more knowledgeable/stronger adversaries. In our threat model, we assume all clients and the server are benign and assume the attacker does not know our CWG. Here, we test FedGMark against stronger adversaries, where some clients are malicious and the these malicious clients also has access to the CWG component to manipulate the FedGL training. First, we consider a passive attack where all malicious clients do not use CWG to generate customized local watermarks. Model (b) in Table 5 shows the maximum WAdecrease is 9%, where all clients do not use CWG.

Second, we test an active attack where malicious clients modify their watermark data's label to obfuscate the training. Specifically, all malicious clients' watermark data are labeled (e.g., 2) differently from the target label (e.g., 1) and then follow the federated training. The results in Table 13 show MA/WA is marginally affected even with 20% malicious clients.

FedGMark with alternative aggregation methods. The current FedGMark's evaluation focuses on FedAvg for aggregating client models. Here, we evaluate FedGMark using Multi-Krum [Blanchard et al. , 2017] and Trim-mean [Yin et al. , 2018] aggregation methods that consider data quality (e.g., remove outlier clients). Specifically, Multi-Krum filters a set of p clients whose gradients largely deviated from others, while Trim-mean trims off q highest and lowest values for each parameter in clients' models. Table 14 shows the results with p = 10 and q = 10 . We can see these robust aggregators achieve a robustness-utility tradeoff, and MA and WA are not largely different.

0.9

0.9

0.7

0.8

0.8

0.6

WA

WA

WA

WA

0.7

0.8

0.7

local 1

global

global

local 1

0.5

global

local 1

global

local 1

local 2

local 2

local 2

local 2

local 3

local 3

local 3

0.6

local.3

0.6

local 4

0.7

local 4

local 4

local 4

Fed-GINFed-GSAGEFed-GCN

Fed-GINFed-GSAGEFed-GCN

Fed-GINFed-GSAGEFed-GCN

Fed-GINFed-GSAGEFed-GCN

MUTAG

PROTEINS

DD

COLLAB

FedGMark with alternative triggers such as feature-based triggers and hybrid triggers. The current FedGMark relies on graph structural information to learn the trigger used in watermark. Here, we adjust FedGMark to learn feature-based triggers and hybrid feature-structure triggers.

To learn feature-based triggers, we first select a set of nodes from a graph as the target nodes, and learn the watermarked features for the target nodes (we do not watermark structure). We use a graph G i = ( V i , E i , X i ) from client i for illustration, where X i is the node feature matrix. We then define a feature-mask M i f [ v j ] = 1 if v j ∈ V i w and 0 otherwise, where V i w is the watermark node set described in the paper. Then, we introduce a feature network ( FeaNet ) that learns watermarked node features as X i w = FeaNet( X i ) ⊙ M i f . The FeaNet takes input X i and outputs a matrix having the same size as X i , e.g., it has the same architecture as GatingNet but adjusts the input size. The corresponding watermarked graph is defined as G i w = ( V i , E i , X i w ) . By generating a set of watermarked graphs G i w for client i , we minimize the loss on client i 's both clean graphs G i c and G i w .

Further, to learn feature-structure triggers, we combine FeaNet (that gets X i w ) with GatingNet / KeyNet (that gets E i w ), and the watermarked graphs are G i w = ( V i , E i w , X i w ) . We then minimize the loss on G i c and G i w . More details about training refer to Section 3.4.

We evaluate these triggers and the results are in Table 15. We observe that structure information alone is sufficient to enable designing effective triggers.

E Discussion

Global watermark vs. local watermarks. In this paper, each client employs CWG to generate watermarks (i.e., local watermarks) to train the local model. During testing, we can use either local watermarks or a global watermark, i.e., fully connect local watermarks. Here, we further evaluate the impact of global watermark vs. local watermarks of the target watermarked model under default settings. For better clarity, we randomly select four local watermarks labeled as local watermarks 1 to 4 . The experimental results depicted in Figure 6 demonstrate that, despite not being part of the training process, the global watermark performs slightly better than local watermarks. This implies that the distributed training of FedGL can aggregate the effects of local watermarks, providing a foundation for utilizing the global watermark for verification.

Ensemble classifier with majority voting vs. summation on submodels' outputs. In this paper, we propose an ensemble classifier g to aggregate the predictions of each submodel ϑ i through a voting mechanism and classify a graph G as g ( θ, G ) = arg max y ∈ Y N y , where N y = ∑ S i =1 ■ ( f ( ϑ i , G ) = y ) . Another common strategy, as used in GL models such as GIN [Xu et al. , 2019], is to sum the prediction confidence vector of each submodel ϑ i , i.e., g ( θ, G ) = arg max y ∈ Y f G , where f G = ∑ S i =1 f ( ϑ i , G ) and f outputs a probabilistic confidence vector whose summation is 1. Here, we also test our FedGMark with this strategy against the 1-layer perturbation attack, and the results are shown in Table 16. We notice the results do not remain consistent across different numbers of submodels and are worse than those in Table 3, especially when S is small. This suggests that the summing-based ensemble classifier is not robust to the layer-perturbation attack, emphasizing the necessity of our proposed majority-voting based ensemble classifier.

FedGMark against offline clients that cannot provide watermark data or malicious clients that provide fake watermark samples during ownership verification. We deem that our ownership verification is still robust against offline clients and malicious clients, if its number is less than 50%. During ownership verification, each client provides its own watermark data to the trusted judge. When some clients are offline, the trusted judge can simply neglect them and only use participating clients' watermark data for verification.

When facing malicious clients, their negative effect can be circumvented through a majority votingbased approach. Specifically, all clients provide their own watermark data to the judge and obtain the watermark accuracy per client. Though the watermark accuracy on malicious clients could be very low, the majority of benign clients can produce more number of high watermark accuracy, compared to the number of low accuracy. When the judge uses the majority-vote strategy, the final watermark accuracy is still high, ensuring the accurate ownership claim for benign clients.

Using clients' training graphs as the watermark samples and sending them to the trust judge may lead to privacy leakage. We clarify that the watermark data are not necessarily generated from the training/test samples. Remember the primary goal of backdoor-based watermarking is to force the model to memorize the relationship between the backdoor trigger (in the watermark samples) and the target label, while the samples to inject the trigger do not have constraints, i.e., they can be from training samples or artificially synthesized (which does not contain privacy information of any training/test data). For conveniences, existing methods inject backdoor triggers into the training/test samples. To validate this, we synthesize a set of random graphs using the popular Erd˝ os-Rényi model (via the NetworkX toolbox) and the watermark samples are generated by injecting the learnt watermark on the synthesized graphs. Under the default setting, we test on Fed-GIN and show results in Table 17, where we observe WAs are very close to those shown in the paper on the four datasets.

Furthermore, since all clients intend to verify model ownership, it is reasonable to believe that these clients are willing to provide their watermark data-whether generated from private training/test data or non-private synthesized data-exclusively to a trusted judge, with informed consent and in accordance with legal and ethical standards. From this perspective, the data is confidential between each client and the trusted judge. We acknowledge it is very interesting future work to design a provably private mechanism for model ownership verification that the verifier cannot access the watermark data but can guarantee the correctness of verification.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the research problem, motivation, and contributions of this paper in the Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the approach in the Discussion section, e.g., using clients' training graphs as watermark samples and sending them to the trusted judge could lead to privacy leakage.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide complete theory background in the main text, and present proofs in the supplementary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our experimental results are reproducible. We provide detailed experimental settings in the main text.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include links to the source code in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide training and test details in the experimental part.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experimental results are stable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments can be reproduced using one NVIDIA GeForce GTX 1080 Ti GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Positive societal impacts: Our proposed watermark i.e. FedGMark can protect the ownership of FedGL models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no the above risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the reference of datasets employed in this paper following their requirements.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: There is no dataset contribution in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigated the problem of watermarking the Federated Graph Learning (FGL) models. This paper proposed the first backdoor-based FGL watermarking framework, called FedGMark. Specifically, to tackle the issues of ineffectiveness and vulnerability of existing methods, FedGMark designed two modules respectively. One is a Customized Watermark Generator (CWG). CWG aimed to generate the watermarked trigger samples (graphs) using each client's secret key. The other is the Robust Model Loader (RML). RML guaranteed that the watermarked models were certifiably robust against layer perturbation attacks.\n\nSTRENGTHS:\n- The first attempt to watermark federated graph learning models.\n- The watermarked models are certifiably robust against attacks.\n- Experiments on various datasets and models validate the effectiveness of FedGMark.\n\nWEAKNESSES:\nMy major concerns are as follows.\n1. Unclear threat model: The threat model and the problem formulation of this paper is unclear. What's the capability of the adversary and the defender? And more importantly, who is the adversary to steal the FGL model? This paper proposed to watermark the FGL model from the client side, which means the clients should be trustworthy. Is the central server an adversary in this paper? To my best knowledge, the typical threat model of various attacks in FL (e.g., backdoor attacks or Byzantine attacks) assumes that some of the clients may be malicious. The author should add a section on the threat model or problem formulation and clarify why they make these assumptions. This may be helpful to better understand the problem the authors tried to solve.\n2. Privacy concern: I also worry that utilizing FedGMark may raise privacy concerns. In Section 3.4, the watermarked client needs to use a subset of its training graphs as the watermarked graphs. However, in FL, the client's graphs are privacy-sensitive, and using them to verify ownership may lead to privacy leakage. This is contrary to the original purpose (preserve privacy) of FL.\n3. Missing experiments on the robustness against backdoor defense: This paper considers three different watermark removal attacks. However, since FedGMark utilizes backdoor-based watermarking methods, it is important to validate whether FedGMark is robust against backdoor defenses.\n4. Missing introduction to ownership verification: This paper lacks an important section to introduce the ownership verification procedure of FedGMark.\n\nQUESTIONS:\n1. Clarify the threat model.\n2. Address the privacy concern.\n3. Analysis or experiments on the robustness against backdoor defenses.\n4. Clarify the procedure of ownership verification in FedGMark.\n\nLIMITATIONS:\nThis paper does not include a discussion of the limitations. However, I think there is a strong assumption that the clients need to be trustworthy in FedGMark. A discussion on this assumption is necessitated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces FedGMark, a backdoor-based watermarking method specifically designed to protect Federated Graph Learning (FedGL) models from illegal copying and model theft. They claim that the proposed FedGMark is the first method to safeguard the intellectual property of FedGL models, offering certified robustness against watermark removal attacks, leveraging unique graph structures and client information to create customized and diverse watermarks. Experiments demonstrate its effectiveness and robustness.\n\nSTRENGTHS:\nThe paper introduces FedGMark to address the overlooked vulnerability of FedGL model ownership and identifies three main challenges in current watermarking techniques: inapplicability to graph data, vulnerability to removal attacks, and lack of formal guarantees. The proposed method, including CWG and RML, is clear and intuitive, and the authors have provided comprehensive experiments to support their approach.\n\nWEAKNESSES:\n1.\tI strongly recommend setting a \"Threat Model\" subsection to clarify the potential security threats to FedGL. In my opinion, since the authors consider watermark removal attacks like distillation and finetuning, FedGL operates under a white-box setting.\n2.\tThe paper assumes attackers know the internal information of the target watermarked model, enabling distillation, finetuning, and layer-perturbation attacks. However, I find the white-box setting narrow and trivial. The authors should consider black-box attacks, which are more challenging and meaningful. Many studies on black-box attacks can be found.\n3.\tIn watermarking-related literature, robustness and fidelity are more frequently used terms than watermark accuracy and task accuracy.\n4.\tIn the \"Inapplicable or Ineffective\" item, the authors state, \"For instance, they require input data to have the same size, while graphs can have varying sizes,\" which is not entirely accurate. For example, some Wavelet and DCT-based watermarking methods can be scalable.\n\nQUESTIONS:\nPlease refer to Weaknesses part\n\nLIMITATIONS:\nPlease refer to Weaknesses part", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of protecting model ownership in the emerging domain of Federated Graph Learning (FedGL) by proposing FedGMark, a backdoor-based watermarking technique. The authors argue that existing watermarking approaches are either inapplicable to graph data or exhibit weaknesses in terms of robustness against removal attacks and lack of formal guarantees. FedGMark aims to overcome these limitations by leveraging graph structure and client information to learn customized watermarks, employing a novel graph learning (GL) architecture that enhances robustness, and providing certified robustness guarantees against layer-perturbation attacks.\n\nSTRENGTHS:\n- The paper clearly outlines the limitations of existing watermarking techniques and presents a well-motivated approach to address them. The design of FedGMark, with its CWG and RML modules, is tailored to the specific challenges of watermarking in FedGL.\n-  FedGMark demonstrates promising empirical performance in terms of both main task accuracy and watermark accuracy. It outperforms the baseline approach (random graph-based watermarking) significantly, especially under watermark removal attacks.\n- The paper provides theoretical guarantees for the robustness of FedGMark against layer-perturbation attacks, a unique and valuable contribution in the watermarking literature.\n\nWEAKNESSES:\n1. The reliance on pre-defined private keys for watermark generation may not be practical in all scenarios, and alternative key management methods should be explored.\n2. The assumption of limited attacker knowledge about the watermarked model may not hold in practice. Evaluating FedGMark against more knowledgeable adversaries would provide a more realistic assessment.\n3. The focus on FedAvg for model aggregation limits the exploration of other aggregation methods and their impact on watermark robustness.\n\nQUESTIONS:\n1. Could you quantify the communication overhead of FedGMark during federated training, especially compared to random graph-based watermarking (in terms of local training time, size of watermarked data, etc.)?\n2. How do you envision FedGMark being deployed in a real-world FedGL system? What practical challenges might arise during implementation and watermark verification?\n3. How would the certified robustness guarantees be affected by more advanced watermark removal attacks beyond layer perturbation (e.g., those involving trigger reverse engineering)?\n4. How would the effectiveness of FedGMark be affected if the attacker had more knowledge about the watermarking process, such as access to the CWG architecture or the private key generation method?\n\nLIMITATIONS:\n1. FedGMark's evaluation focuses solely on FedAvg for aggregating client models. The impact of alternative aggregation methods (e.g., those prioritizing clients based on data quality or model performance) on both watermark robustness and overall FedGL model performance remains unexplored.\n2. The paper acknowledges the increased computational cost of using more submodels (S) in RML but doesn't fully analyze the scalability of FedGMark. Further investigation is needed to understand how performance scales with different numbers of clients.\n3. FedGMark relies heavily on structural modifications of the graph as the watermark.  The effectiveness and robustness of alternative trigger designs, such as feature-based triggers, hybrid triggers, or combinations of different trigger types, have not been explored. \n4. The paper lacks specific details about the hyperparameters used for training the GL models on the client-side. The impact of client training dynamics, particularly the choice of learning rate and the number of local epochs, on the watermarking performance and robustness of FedGMark remains unclear and requires further investigation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies watermarking for federated graph learning (FGL) to protect the ownership of participants. It proposes a customized watermark generator for local clients that can capture the local graph structure and private client information, and a robust model loader consisting of multiple GL submodels and a majority-voting-based ensemble classifier, which can defend against the proposed layer-perturbation attack.\n\nSTRENGTHS:\n1. This work claims to be the first to study watermarking for FGL models.\n\n2. The method can leverage local graph and client information to generate customized watermarks.\n\n3. The paper introduces a layer-perturbation attack to further demonstrate the certifiably robustness of the proposed backdoor-based watermarking for FGL.\n\n4. The work is well-motivated with preliminary studies.\n\nWEAKNESSES:\n1. The concept of ownership in FGL can be confusing and is not well-defined in this paper. For example, can every client claim ownership of the federated trained model? Since the watermarks from different clients are different, can any single client claim entire ownership? Additionally, for clients who participate in the FL but do not have watermarks, how can they claim ownership?\n\n2. The motivation for using local customized watermarks is not clear. The following problems arise: (1) It is unclear how to conduct ownership verification. Should it use the global watermark or the local watermarks? (2) If using a global watermark, what is the necessity of employing customized watermarks, or what is the adequate way to aggregate the global watermark from customized watermarks? If using local watermarks, how can the customized watermarks be used across clients?\n\n3. The method requires specific GL models (to be split to multiple submodels), which can be hard to adapt to existing FGL methods, especially for advanced FGL methods.\n\n4. The motivation for incorporating submodels for GL is missing. Why is this design necessary?\n\n5. (1) What does “layer indexes” for splitting GL models mean? From section 3.3, it is not clear how the submodels are split and how the split submodels are decoupled from each other regarding cascaded structures. (2) Additionally, structural information can be important for graph learning. How would discarding such structural information impact in this setting?\n\n6. The global model is obtained by simply averaging uploaded clients’ models (not weighted by data size, or applying proxy terms for regularization). Can this method address the potential heterogeneity issue when local watermarks are highly disparate from each other?\n\n7. The proposed method can introduce efficiency issues, as it significantly increases the number of parameters and computation time.\n\nQUESTIONS:\n1. When the set of selected clients for aggregation is different from the set of watermarked clients, can the method achieve stable convergence?\n\n2. Is the layer-perturbation attack applied before or after submodel splitting? If it is applied after, does it perturb all submodels or not?\n\n3. Out of curiosity, is it possible to federated learn the local watermarking? How do you expect this would perform?\n\nLIMITATIONS:\nPlease see Weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}