Node-Level Differentially Private Graph Neural Networks

1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful modeling tools that capture structural information provided by a graph . Consequently , they have become popular in a wide array of domains such as biology ( Ktena et al. , 2018 ) , medicine ( Ahmedt-Aristizabal et al. , 2021 ) , chemistry ( McCloskey et al. , 2019 ) , computer vision ( Wang et al. , 2019 ) , and text classification ( Yao et al. , 2019 ) . GNNs allow aggregation of data from the neighbors of a given node in the graph , thus evading the challenge of data scarcity per node . Naturally , such solutions are quite attractive in modeling users – each node of the graph is represented by the user and the connections represent interactions between the users – for a variety of recommendation/ranking tasks , where it is challenging to obtain and store user data ( Fan et al. , 2019 ; Budhiraja et al. , 2020 ; Levy et al. , 2021 ) . However , such solutions are challenging to deploy as they are susceptible to leaking highly sensitive private information about the users . It is well-known that standard ML models – without GNN style data aggregation – can leak highly sensitive information about the training data ( Carlini et al. , 2019 ) . The risk of leakage is significantly higher in GNNs as each prediction is based on not just the individual node , but also an aggregation of data from the neighborhood of the given node . In fact , there are two types of highly-sensitive information about an individual node that can be leaked : a ) the features associated with each node/user , b ) the connectivity information of an individual node/user . In this work , we study the problem of designing algorithms to learn GNNs while preserving nodelevel privacy , i.e. , preserving both the features as well as connectivity information of an individual node . We use differential privacy as the notion of privacy ( Dwork et al. , 2006 ) of a node , which roughly-speaking requires that the algorithm should learn similar GNNs despite perturbation of an entire node and all the data points or predictions associated with that node . Example scenarios for such a solution include ranking/recommendation of entities like documents/emails in an organization . Here , the graph can be formed by a variety of means like how users interact with each other , and the goal would be to learn user features that can enable more accurate ranking of emails/documents . Naturally , user interaction data as well as individual users ’ features ( like the topics in which user is interested in ) would be critical to preserve , and any revelation of such data can be catastrophic . Furthermore , once GNNs are learned to model users while preserving privacy , they can be used in different settings based on the problem requirement . For example , in settings where a node can access it ’ s r-hop neighbors data , we can directly apply r-layer GNNs ( if they are trained with DP ) . Similarly , in certain scenarios , we would want to learn GNNs over a large enterprise and deploy the same model for a small enterprise , where at inference time neighborhood information ( like managerial reporting structure ) might be publicly accessible within the enterprise but not across enterprises . See Section 4 for a detailed discussion . Recent works have explored the problem of differentially private learning of GNNs , but they either consider a restricted setting of edge-level privacy which is often insufficient for real-world problems or they restrict themselves to simpler settings like bipartite graphs or node-level privacy without preserving individual connectivity information ( Wu et al. , 2021a ; b ; Zhou et al. , 2020 ) . In contrast , our proposed method preserves the privacy of the features of each node ( ‘ user ’ ) , their labels as well as their connectivity information . To this end , we adapt the standard DP-SGD method ( Song et al. , 2013 ; Bassily et al. , 2014 ; Abadi et al. , 2016 ) to our setting . But , analysis of the standard DP-SGD method does not directly extend to GNNs , as each gradient term in GNNs can depend on multiple nodes . The key technical contribution of our work is two-fold : i ) we provide a careful sensitivity analysis for the special case of 1-layer GNNs , ii ) we extend the standard privacy by amplification technique to GNNs where one gradient term can depend on multiple users . Note that the standard privacy by amplification method only applies to scenarios where each point corresponds to one user/entity . By combining the above two results with the standard Rényi Differential Privacy ( RDP ) accounting , we obtain a formal proof of privacy for our method . Finally , we evaluate our DP-GNN method on standard benchmarks . We demonstrate that DP-GNN is reasonably accurate compared to the standard 1-layer GCN models , while providing privacy parameters of about ≤ 30 which are close to the industry standard . More critically , compared to standard MLP ( multi-layer perceptron ) based methods that completely discard graph side-information , our method can be 5-6 % more accurate while still providing strong privacy guarantees . That is , we demonstrate that GNN based techniques can indeed be deployed in practice with the benefits of improved accuracy over vanilla MLP style methods while still preserving sensitive user data . Contributions : We propose a Node-Level Differentially Private Graph Neural Network that works well in practice and provides formal privacy guarantees . This is the first work , to the best of our knowledge , to provide such strong privacy guarantees for each individual node in the graph learning regime . Our main contributions are organised as follows : • Formulation : In Section 3 , we formalize the problem of node-level differentially private GNNs , and discuss various important settings in which a solution to the problem is applicable . • Method : In Section 4 , we describe our algorithm that adapts standard DP-SGD to train differentially private GNNs , with a strong privacy guarantee that extends standard privacy amplification by sampling . • Empirical Evaluation : In Section 5 , we evaluate our framework on multiple benchmark graph datasets on the task of node classification . We demonstrate that our DP-GNN method can outperform non-private and private MLP methods that can not utilize graph information . 2 RELATED WORK . Mechanisms to make the training process of machine learning models private primarily fall into two categories : model-agnostic methods such as PATE ( Papernot et al. , 2017 ) , and model-aware methods such as DP-SGD ( Abadi et al. , 2016 ) , which augment the standard paradigm of gradientbased training to be differentially private . DP-SGD , in particular , has been used successfully to train neural network models to classify images ( Abadi et al. , 2016 ) and text ( Anil et al. , 2021 ) . Today , there are many varieties of graph neural networks employed : Graph Convolutional Neural Networks ( Kipf & Welling , 2016 ) , Graph Attention Networks ( Veličković et al. , 2018 ) , GraphSAGE ( Hamilton et al. , 2017 ) , and Message-Passing Neural Networks ( Gilmer et al. , 2017 ) , to name a few . Broadly , these models compute node-level representations via aggregation of neighbourhood-level information , that can lead to diffusion of private information across multiple nodes , thus making application of standard DP-SGD like techniques non-trivial . There has been recent work in learning and evaluating edge-level private GNNs ( Wu et al. , 2021b ) but they do not preserve node-level data . Private GNNs have also been studied from the perspective of local privacy ( Sajadmanesh & Gatica-Perez , 2020 ) , where each node performs its share of the GNN computation locally . In such a setting , each node sends noisy versions of its features and labels to neighbouring nodes in order to learn shared weights , resulting in a elaborate learning algorithm that needs to correct for the bias in both the features and labels . ( Wu et al. , 2021a ) utilizes private GNNs for recommendation systems , but their method assumes a bipartite graph structure , and can not naturally handle homogeneous graphs . Other approaches employ federated learning ( Zhou et al. , 2020 ) , but only guarantee that the GNN neighbourhood aggregation step is differentially private , which is insufficient to guarantee privacy of each node ’ s neighborhood . Finally , other attempts ( Shan et al. , 2021 ) to create privacy-preserving GNNs exist , but these do not use the formal notion of DP . Model-agnostic methods , such as PATE , have recently been investigated to train GNNs ( Olatunji et al. , 2021 ) . In their current form , however , such methods require access to public data samples , which may not always be available for the task at hand . In contrast to previous approaches which protect the privacy of a node ’ s features and labels only , we additionally seek to protect every node ’ s adjacency vector , which is its private list of connections to neighbouring nodes . This is because the existence of communication between a pair of nodes can often be sensitive information in itself . Further , our approach extends the standard approaches of gradient-based training to scalably train node-level differentially private GNNs in a centralized setting , without any access to public data . Depending on the required privacy setting , this mechanism can be composed with locally differentially private mechanisms to generate node-level predictions . In different contexts , there has been extensive work on node-level DP ( Raskhodnikova & Smith , 2016 ; Karwa et al. , 2011 ; Borgs et al. , 2015 ; 2018 ) . But these methods generally deal with modeling ‘ global ’ graph-level statistics and do not support learning methods such as GNNs . In contrast , our approach aims to predict ‘ local ’ node-level statistics ( like the label of a node ) while preserving node-level privacy . 3 PROBLEM FORMULATION AND PRELIMINARIES . Consider a graph dataset G = ( V , E , X , Y ) with directed graph G = ( V , E ) represented by a adjacency matrix A ∈ { 0 , 1 } n×n . n is the number of nodes in G , V denotes the node set , E denotes the edge set . Each node v in the graph is equipped with a feature vector Xv ∈ Rd ; X ∈ Rn×d denotes the feature matrix . Y ∈ Rn×Q is the label matrix and yv is the label for the v-th node over Q classes . Note that many of the labels in the label vector can be missing , which models the semi-supervised setting . In particular , we assume that node labels yv are only provided for a subset of nodes Vtr ⊂ V , called the training set . Given the graph dataset G , the goal is to learn parameters of a one-layer GNN while preserving privacy of individual nodes . A GNN can be represented by the following operations : ŷv = GNN ( A , X , v ; Θ ) : = fdec ( fagg ( { fenc ( Xu ) |Avu 6= 0 } ) ) ( 1 ) where ŷv is the prediction from the GNN for a given node v , fenc is the encoder function that encodes node features with parameters Θenc , fagg is the neighborhood aggregation function with parameters Θagg , fdec is the prediction decoder function with parameters Θdec , and Θ : = ( Θenc , Θagg , Θdec ) . While our results apply to most 1-layer GNN models ( Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2018 ) , for simplicity , we focus on 1-layer Graph Convolutional Network ( GCN ) models1 ( Kipf & Welling , 2016 ) . These GCN models use a multi-layer perceptron ( MLP ) for encoder and decoder functions , with non-linear activation function σ : ŷv = GCN ( A , X , v ; Θ ) : = MLPdec ( Avσ ( MLPenc ( X ) ) Θagg ) ( 2 ) 1 As is common in practice , we allow any normalization and addition of self-loops to A . Thus , “ learning ” a GCN is equivalent to finding parameters Θ : = ( Θenc , Θagg , Θdec ) that minimize a suitable loss : Θ∗ = arg min Θ ∑ v∈V ` ( ŷv ; yv ) ︸ ︷︷ ︸ L ( G , Θ ) ( 3 ) where ` : RQ×Q → R is a standard loss function such as categorical cross-entropy.2 As mentioned earlier , we use differential privacy as the notion of privacy of a node . Before defining differential privacy , we first define the notion of adjacent graph datasets : Definition 1 ( Adjacent Graph Datasets ) . Two graph datasets G and G′ are said to be node-level adjacent if one can be obtained by adding or removing a node ( with its features , labels and associated edges ) to the other . That is , G and G′ are exactly the same except for the v-th node , i.e. , Xv , yv and Av differ in the two datasets . Informally , A is said to be node-level differentially-private algorithm if the addition or removal of a node in A ’ s input does not affect A ’ s output significantly . Definition 2 ( Node-level Differential Privacy ) . Consider any randomized algorithm A that takes as input a graph dataset . A is said to be ( α , γ ) node-level Rényi differentially-private ( Mironov , 2017b ) if , for every pair of node-level adjacent datasets G and G′ : Dα ( A ( G ) ‖ A ( G′ ) ) ≤ γ , where Rényi divergence Dα of order α between two random variables P and Q is defined as : Dα ( P ‖ Q ) = 1 α− 1 lnEx∼Q [ P ( x ) Q ( x ) ] α . Note that we use Rényi differentially-private ( RDP ) ( Mironov , 2017b ) as the formal notion of differential privacy ( DP ) , as it allows for tighter composition of DP across multiple steps . This notion is closely related to the standard ( ε , δ ) -differential privacy ( Dwork et al. , 2006 ) ; Proposition 3 of Mironov ( 2017b ) states that any ( α , γ ) -RDP mechanism also satisfies ( γ + log 1/δα−1 , δ ) -differential privacy for any 0 < δ < 1 . Thus , the goal is to find Θ by optimizing equation 3 while ensuring RDP ( Definition 2 ) . It is clear that node-level privacy is essential when training models on graph datasets with sensitive node-level information . However , node-level privacy is significantly harder to achieve than the weaker notion of edge-level privacy . In the context of GNNs , the representation for a node is computed using not just the node ’ s individual features , but also features of other nodes from the local neighbourhood . Thus , the removal of a node from a graph dataset affects its entire local neighbourhood , which can be a very large set of nodes . This is in contrast to the standard non-graph setting for differentially private models , where the representation of individual users would only depend on the user ’ s own data . We now define two concepts that are critical in our design and analysis of a private GNN learning method . Definition 3 . The node-level sensitivity ∆ ( f ) of a function f defined on graph datasets is : ∆ ( f ) = max node-level adjacent G , G′ ‖f ( G ) − f ( G′ ) ‖2 The K-restricted node-level sensitivity ∆K ( f ) of a function f defined on graph datasets is : ∆K ( f ) = max deg ( G ) , deg ( G′ ) ≤K node-level adjacent G , G′ ‖f ( G ) − f ( G′ ) ‖2 Definition 4 . We define the clipping operator ClipC ( . ) as : ClipC ( v ) = min ( 1 , C‖v‖F ) · v , for any vector or matrix v. 2 The analysis here holds for multi-label settings as well , which would instead use loss functions such as sigmoidal cross-entropy , for example . Algorithm 1 : DP-GNN ( SGD ) : Differentially Private Graph Neural Network with SGD Data : Graph G = ( V , E , X , Y ) , GNN definition GNN , Training set Vtr , Loss function L , Batch size m , Maximum degree K , Learning rate η , Clipping threshold C , Noise standard deviation σ , Maximum training iterations T . Result : GNN parameters ΘT . Note that Vtr is the subset of nodes for which labels are available ( see Paragraph 1 of Section 3 ) . Using Vtr , construct the set of training subgraphs Str with Algorithm 2 . Construct the 0− 1 adjacency matrix A : Avu = 1 ⇐⇒ ( v , u ) ∈ Str Initialize Θ0 randomly . for t = 0 to T do Sample set Bt ⊆ Vtr of size m uniformly at random from all subsets of Vtr . Compute the update term ut as the sum of the clipped gradient terms in the batch Bt : ut ← ∑ v∈Bt ClipC ( ∇Θ ` ( GNN ( A , X , v ; Θt ) ; yv ) ) Add independent Gaussian noise to the update term : ũt ← ut +N ( 0 , σ2I ) Update the current estimate of the parameters with the noisy update : Θt+1 ← Θt − ηm ũt end

Summary: This paper aims to achieve node-level differential privacy on GNNs. Through adding noises to calculated gradients at each optimization step, this work shows that trained GNN layer can be (α, γ) node-level Renyi differentially-private. Concretely, this work considers the 1-layer GNN case, and make the maximum degree of each node to be K. With these requirements, this work theoretically prove the scale of noises.  Its contributions are as follows: 1.	Propose the task of learning node-level differential private GNNs, 2.	Adapt DP-SGD to work on graphs, through extending amplified privacy guarantee, 3.	Evaluate proposed optimization algorithm on benchmark graph datasets.