Node-Level Differentially Private Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful modeling tools that capture structural information provided by a graph . Consequently , they have become popular in a wide array of domains such as biology ( Ktena et al. , 2018 ) , medicine ( Ahmedt-Aristizabal et al. , 2021 ) , chemistry ( McCloskey et al. , 2019 ) , computer vision ( Wang et al. , 2019 ) , and text classification ( Yao et al. , 2019 ) . GNNs allow aggregation of data from the neighbors of a given node in the graph , thus evading the challenge of data scarcity per node . Naturally , such solutions are quite attractive in modeling users – each node of the graph is represented by the user and the connections represent interactions between the users – for a variety of recommendation/ranking tasks , where it is challenging to obtain and store user data ( Fan et al. , 2019 ; Budhiraja et al. , 2020 ; Levy et al. , 2021 ) . However , such solutions are challenging to deploy as they are susceptible to leaking highly sensitive private information about the users . It is well-known that standard ML models – without GNN style data aggregation – can leak highly sensitive information about the training data ( Carlini et al. , 2019 ) . The risk of leakage is significantly higher in GNNs as each prediction is based on not just the individual node , but also an aggregation of data from the neighborhood of the given node . In fact , there are two types of highly-sensitive information about an individual node that can be leaked : a ) the features associated with each node/user , b ) the connectivity information of an individual node/user . In this work , we study the problem of designing algorithms to learn GNNs while preserving nodelevel privacy , i.e. , preserving both the features as well as connectivity information of an individual node . We use differential privacy as the notion of privacy ( Dwork et al. , 2006 ) of a node , which roughly-speaking requires that the algorithm should learn similar GNNs despite perturbation of an entire node and all the data points or predictions associated with that node . Example scenarios for such a solution include ranking/recommendation of entities like documents/emails in an organization . Here , the graph can be formed by a variety of means like how users interact with each other , and the goal would be to learn user features that can enable more accurate ranking of emails/documents . Naturally , user interaction data as well as individual users ’ features ( like the topics in which user is interested in ) would be critical to preserve , and any revelation of such data can be catastrophic . Furthermore , once GNNs are learned to model users while preserving privacy , they can be used in different settings based on the problem requirement . For example , in settings where a node can access it ’ s r-hop neighbors data , we can directly apply r-layer GNNs ( if they are trained with DP ) . Similarly , in certain scenarios , we would want to learn GNNs over a large enterprise and deploy the same model for a small enterprise , where at inference time neighborhood information ( like managerial reporting structure ) might be publicly accessible within the enterprise but not across enterprises . See Section 4 for a detailed discussion . Recent works have explored the problem of differentially private learning of GNNs , but they either consider a restricted setting of edge-level privacy which is often insufficient for real-world problems or they restrict themselves to simpler settings like bipartite graphs or node-level privacy without preserving individual connectivity information ( Wu et al. , 2021a ; b ; Zhou et al. , 2020 ) . In contrast , our proposed method preserves the privacy of the features of each node ( ‘ user ’ ) , their labels as well as their connectivity information . To this end , we adapt the standard DP-SGD method ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) to our setting . But , analysis of the standard DP-SGD method does not directly extend to GNNs , as each gradient term in GNNs can depend on multiple nodes . The key technical contribution of our work is two-fold : i ) we provide a careful sensitivity analysis for the special case of 1-layer GNNs , ii ) we extend the standard privacy by amplification technique to GNNs where one gradient term can depend on multiple users . Note that the standard privacy by amplification method only applies to scenarios where each point corresponds to one user/entity . By combining the above two results with the standard Rényi Differential Privacy ( RDP ) accounting , we obtain a formal proof of privacy for our method . Finally , we evaluate our DP-GNN method on standard benchmarks . We demonstrate that DP-GNN is reasonably accurate compared to the standard 1-layer GCN models , while providing privacy parameters of about ≤ 30 which are close to the industry standard . More critically , compared to standard MLP ( multi-layer perceptron ) based methods that completely discard graph side-information , our method can be 5-6 % more accurate while still providing strong privacy guarantees . That is , we demonstrate that GNN based techniques can indeed be deployed in practice with the benefits of improved accuracy over vanilla MLP style methods while still preserving sensitive user data . Contributions : We propose a Node-Level Differentially Private Graph Neural Network that works well in practice and provides formal privacy guarantees . This is the first work , to the best of our knowledge , to provide such strong privacy guarantees for each individual node in the graph learning regime . Our main contributions are organised as follows : • Formulation : In Section 3 , we formalize the problem of node-level differentially private GNNs , and discuss various important settings in which a solution to the problem is applicable . • Method : In Section 4 , we describe our algorithm that adapts standard DP-SGD to train differentially private GNNs , with a strong privacy guarantee that extends standard privacy amplification by sampling . • Empirical Evaluation : In Section 5 , we evaluate our framework on multiple benchmark graph datasets on the task of node classification . We demonstrate that our DP-GNN method can outperform non-private and private MLP methods that can not utilize graph information . 2 RELATED WORK . Mechanisms to make the training process of machine learning models private primarily fall into two categories : model-agnostic methods such as PATE ( Papernot et al. , 2017 ) , and model-aware methods such as DP-SGD ( Abadi et al. , 2016 ) , which augment the standard paradigm of gradientbased training to be differentially private . DP-SGD , in particular , has been used successfully to train neural network models to classify images ( Abadi et al. , 2016 ) and text ( Anil et al. , 2021 ) . Today , there are many varieties of graph neural networks employed : Graph Convolutional Neural Networks ( Kipf & Welling , 2016 ) , Graph Attention Networks ( Veličković et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) , and Message-Passing Neural Networks ( Gilmer et al. , 2017 ) , to name a few . Broadly , these models compute node-level representations via aggregation of neighbourhood-level information , that can lead to diffusion of private information across multiple nodes , thus making application of standard DP-SGD like techniques non-trivial . There has been recent work in learning and evaluating edge-level private GNNs ( Wu et al. , 2021b ) but they do not preserve node-level data . Private GNNs have also been studied from the perspective of local privacy ( Sajadmanesh & Gatica-Perez , 2020 ) , where each node performs its share of the GNN computation locally . In such a setting , each node sends noisy versions of its features and labels to neighbouring nodes in order to learn shared weights , resulting in a elaborate learning algorithm that needs to correct for the bias in both the features and labels . ( Wu et al. , 2021a ) utilizes private GNNs for recommendation systems , but their method assumes a bipartite graph structure , and can not naturally handle homogeneous graphs . Other approaches employ federated learning ( Zhou et al. , 2020 ) , but only guarantee that the GNN neighbourhood aggregation step is differentially private , which is insufficient to guarantee privacy of each node ’ s neighborhood . Finally , other attempts ( Shan et al. , 2021 ) to create privacy-preserving GNNs exist , but these do not use the formal notion of DP . Model-agnostic methods , such as PATE , have recently been investigated to train GNNs ( Olatunji et al. , 2021 ) . In their current form , however , such methods require access to public data samples , which may not always be available for the task at hand . In contrast to previous approaches which protect the privacy of a node ’ s features and labels only , we additionally seek to protect every node ’ s adjacency vector , which is its private list of connections to neighbouring nodes . This is because the existence of communication between a pair of nodes can often be sensitive information in itself . Further , our approach extends the standard approaches of gradient-based training to scalably train node-level differentially private GNNs in a centralized setting , without any access to public data . Depending on the required privacy setting , this mechanism can be composed with locally differentially private mechanisms to generate node-level predictions . In different contexts , there has been extensive work on node-level DP ( Raskhodnikova & Smith , 2016 ; Karwa et al. , 2011 ; Borgs et al. , 2015 ; 2018 ) . But these methods generally deal with modeling ‘ global ’ graph-level statistics and do not support learning methods such as GNNs . In contrast , our approach aims to predict ‘ local ’ node-level statistics ( like the label of a node ) while preserving node-level privacy . 3 PROBLEM FORMULATION AND PRELIMINARIES . Consider a graph dataset G = ( V , E , X , Y ) with directed graph G = ( V , E ) represented by a adjacency matrix A ∈ { 0 , 1 } n×n . n is the number of nodes in G , V denotes the node set , E denotes the edge set . Each node v in the graph is equipped with a feature vector Xv ∈ Rd ; X ∈ Rn×d denotes the feature matrix . Y ∈ Rn×Q is the label matrix and yv is the label for the v-th node over Q classes . Note that many of the labels in the label vector can be missing , which models the semi-supervised setting . In particular , we assume that node labels yv are only provided for a subset of nodes Vtr ⊂ V , called the training set . Given the graph dataset G , the goal is to learn parameters of a one-layer GNN while preserving privacy of individual nodes . A GNN can be represented by the following operations : ŷv = GNN ( A , X , v ; Θ ) : = fdec ( fagg ( { fenc ( Xu ) |Avu 6= 0 } ) ) ( 1 ) where ŷv is the prediction from the GNN for a given node v , fenc is the encoder function that encodes node features with parameters Θenc , fagg is the neighborhood aggregation function with parameters Θagg , fdec is the prediction decoder function with parameters Θdec , and Θ : = ( Θenc , Θagg , Θdec ) . While our results apply to most 1-layer GNN models ( Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2018 ) , for simplicity , we focus on 1-layer Graph Convolutional Network ( GCN ) models1 ( Kipf & Welling , 2016 ) . These GCN models use a multi-layer perceptron ( MLP ) for encoder and decoder functions , with non-linear activation function σ : ŷv = GCN ( A , X , v ; Θ ) : = MLPdec ( Avσ ( MLPenc ( X ) ) Θagg ) ( 2 ) 1 As is common in practice , we allow any normalization and addition of self-loops to A . Thus , “ learning ” a GCN is equivalent to finding parameters Θ : = ( Θenc , Θagg , Θdec ) that minimize a suitable loss : Θ∗ = arg min Θ ∑ v∈V ` ( ŷv ; yv ) ︸ ︷︷ ︸ L ( G , Θ ) ( 3 ) where ` : RQ×Q → R is a standard loss function such as categorical cross-entropy.2 As mentioned earlier , we use differential privacy as the notion of privacy of a node . Before defining differential privacy , we first define the notion of adjacent graph datasets : Definition 1 ( Adjacent Graph Datasets ) . Two graph datasets G and G′ are said to be node-level adjacent if one can be obtained by adding or removing a node ( with its features , labels and associated edges ) to the other . That is , G and G′ are exactly the same except for the v-th node , i.e. , Xv , yv and Av differ in the two datasets . Informally , A is said to be node-level differentially-private algorithm if the addition or removal of a node in A ’ s input does not affect A ’ s output significantly . Definition 2 ( Node-level Differential Privacy ) . Consider any randomized algorithm A that takes as input a graph dataset . A is said to be ( α , γ ) node-level Rényi differentially-private ( Mironov , 2017b ) if , for every pair of node-level adjacent datasets G and G′ : Dα ( A ( G ) ‖ A ( G′ ) ) ≤ γ , where Rényi divergence Dα of order α between two random variables P and Q is defined as : Dα ( P ‖ Q ) = 1 α− 1 lnEx∼Q [ P ( x ) Q ( x ) ] α . Note that we use Rényi differentially-private ( RDP ) ( Mironov , 2017b ) as the formal notion of differential privacy ( DP ) , as it allows for tighter composition of DP across multiple steps . This notion is closely related to the standard ( ε , δ ) -differential privacy ( Dwork et al. , 2006 ) ; Proposition 3 of Mironov ( 2017b ) states that any ( α , γ ) -RDP mechanism also satisfies ( γ + log 1/δα−1 , δ ) -differential privacy for any 0 < δ < 1 . Thus , the goal is to find Θ by optimizing equation 3 while ensuring RDP ( Definition 2 ) . It is clear that node-level privacy is essential when training models on graph datasets with sensitive node-level information . However , node-level privacy is significantly harder to achieve than the weaker notion of edge-level privacy . In the context of GNNs , the representation for a node is computed using not just the node ’ s individual features , but also features of other nodes from the local neighbourhood . Thus , the removal of a node from a graph dataset affects its entire local neighbourhood , which can be a very large set of nodes . This is in contrast to the standard non-graph setting for differentially private models , where the representation of individual users would only depend on the user ’ s own data . We now define two concepts that are critical in our design and analysis of a private GNN learning method . Definition 3 . The node-level sensitivity ∆ ( f ) of a function f defined on graph datasets is : ∆ ( f ) = max node-level adjacent G , G′ ‖f ( G ) − f ( G′ ) ‖2 The K-restricted node-level sensitivity ∆K ( f ) of a function f defined on graph datasets is : ∆K ( f ) = max deg ( G ) , deg ( G′ ) ≤K node-level adjacent G , G′ ‖f ( G ) − f ( G′ ) ‖2 Definition 4 . We define the clipping operator ClipC ( . ) as : ClipC ( v ) = min ( 1 , C‖v‖F ) · v , for any vector or matrix v. 2 The analysis here holds for multi-label settings as well , which would instead use loss functions such as sigmoidal cross-entropy , for example . Algorithm 1 : DP-GNN ( SGD ) : Differentially Private Graph Neural Network with SGD Data : Graph G = ( V , E , X , Y ) , GNN definition GNN , Training set Vtr , Loss function L , Batch size m , Maximum degree K , Learning rate η , Clipping threshold C , Noise standard deviation σ , Maximum training iterations T . Result : GNN parameters ΘT . Note that Vtr is the subset of nodes for which labels are available ( see Paragraph 1 of Section 3 ) . Using Vtr , construct the set of training subgraphs Str with Algorithm 2 . Construct the 0− 1 adjacency matrix A : Avu = 1 ⇐⇒ ( v , u ) ∈ Str Initialize Θ0 randomly . for t = 0 to T do Sample set Bt ⊆ Vtr of size m uniformly at random from all subsets of Vtr . Compute the update term ut as the sum of the clipped gradient terms in the batch Bt : ut ← ∑ v∈Bt ClipC ( ∇Θ ` ( GNN ( A , X , v ; Θt ) ; yv ) ) Add independent Gaussian noise to the update term : ũt ← ut +N ( 0 , σ2I ) Update the current estimate of the parameters with the noisy update : Θt+1 ← Θt − ηm ũt end

Summary: The paper claim to proposed a novel training method of GNN under providing both node feature and adjacency privacy. Their main approach is to use existed differential privacy framework into GNN world. They claim that providing privacy on both node feature and their connectivity is novel.