Secure Network Release with Link Privacy

1 INTRODUCTION . Nowadays , open data of networks play a pivotal role in data mining and data analytics ( Tang et al. , 2008 ; Sen et al. , 2008 ; Blum et al. , 2013 ; Leskovec & Krevl , 2014 ) . By releasing and sharing structured relational data with research facilities and enterprise partners , data companies harvest the enormous potential value from their data , which benefits decision-making on various aspects , including social , financial , environmental , through collectively improved ads , recommendation , retention , and so on ( Yang et al. , 2017 ; 2018 ; Sigurbjörnsson & Van Zwol , 2008 ; Kuhn , 2009 ) . However , network data usually encode sensitive information not only about individuals but also their interactions , which makes direct release and exploitation rather unsafe . More importantly , even with careful anonymization , individual privacy is still at stake under collective attack models facilitated by the underlying network structure ( Zhang et al. , 2019 ; Cai et al. , 2018 ) . Can we find a way to securely release network data without drastic sanitization that essentially renders the released data useless ? In dealing with such tension between the need to release utilizable data and the concern of data owners ’ privacy , quite a few models have been proposed recently , focusing on grid-based data like images , texts and gene sequences ( Frigerio et al. , 2019 ; Papernot et al. , 2018 ; Triastcyn & Faltings , 2018 ; Narayanan & Shmatikov , 2008 ; Xie et al. , 2018 ; Chen et al. , 2018 ; Boob et al. , 2018 ; Dy & Krause , 2018 ; Lecuyer et al. , 2018 ; Zhang et al. , 2018 ) . However , none of the existing models can be directly applied to the network ( graph ) setting . While a secure generative model on grid-based data apparently aims to preserve high-level semantics ( e.g. , class distributions ) and protect detailed training data ( e.g. , exact images or sentences ) , it remains obtuse what to be preserved and what to be protected for network data , due to its modeling of complex interactive objects . Motivating scenario . In Figure 1 , a bank aims to encourage public studies on its customers ’ community structures . It does so by firstly anonymizing all customers and then sharing the network ( i.e. , ( a ) in Figure 1 ) to the public . However , an attacker interested in knowing the financial interactions ( e.g. , money transfer ) between particular customers in the bank may happen to have access to another network of a similar set of customers ( e.g. , a malicious employee of another financial company ) . The similarity of simple graph properties like node degree distribution and triangle count between the two networks can then be used to identify specific customers with high accuracy in the released network ( e.g. , customer A as the only node with degree 5 and within 1 triangle , and customer B as the only node with degree 2 and within 1 triangle ) . Thus , the attacker confidently knows the A and B ’ s identities and the fact that they have financial interactions in the bank , which seriously harms customers ’ privacy and poses potential crises . As the first contribution in this work , we define and formulate secure network release goals as preserving global network structure while protecting individual link privacy . Continue with the toy example , the solution we propose is to train a graph neural network model on the original network and release the generated networks ( e.g. , ( b ) in Figure 1 ) . Towards the utility of generated networks , we require them to be similar to the original networks from a global perspective , which can be measured by various graph global properties ( e.g. , network ( b ) has very similar degree distribution and the same triangle count as ( a ) ) . In this way , we expect many downstream data-mining and analytical tasks on them to produce similar results as on the original networks . As for privacy protection , we require that the information in the generated networks can not confidently reveal the existence or absence of any individual links in the original networks ( e.g. , the attacker may still identify customers A and B in network ( b ) , but their link structure has changed ) . Subsequently , there are two unique challenges in learning such structure-preserved and privacyprotected graph generation models , which have not been explored by existing literature so far . Challenge 1 : Rigorous protection of individual link privacy . The rich relational structures in graph data often allow attackers to recover private information through various ways of collective inference ( Zhang et al. , 2014 ; Narayanan & Shmatikov , 2009 ; Backstrom et al. , 2007 ) . Moreover , graph structure can always be converted to numerical features such as spectral embedding , after which most attacks on grid-based data like model inversion ( Fredrikson et al. , 2015 ) and membership inference ( Shokri et al. , 2017 ) can be directly applied for link identification . How can we design an effective mechanism with rigorous privacy protection on links in networks against various attacks ? Challenge 2 : Effective preservation of global network structure . To capture the global network structure , the model has to constantly compare the structures of the input graphs and currently generated graphs during training . However , unlike images and other grid-based data , graphs have flexible structures , and thus they lack efficient universal representations ( Dong et al. , 2019 ) . How can we allow a network generation model to effectively learn from the structural difference between two graphs , without conducting very time-costly operations like isomorphism tests all the time ? Present work . In this work , for the first time , we draw attention to the secure release of network data with deep generative models . Technically , towards the aforementioned two challenges , we develop Differentially Private Graph Generative Nets ( DPGGAN ) , which imposes DP training over a link reconstruction based network generation model for rigorous individual link privacy protection , and further ensures structure-oriented graph comparison for effective global network structure preservation . In particular , we first formulate and enforce edge-DP via Gaussian gradient distortion by injecting designed noise into the sensitive modules during model training . Then we leverage graph convolutional networks ( Kipf & Welling , 2017 ) through a variational generative adversarial network architecture ( Gu et al. , 2019 ; Larsen et al. , 2016 ) to enable structure-oriented network comparison . To evaluate the effectiveness of DPGGAN , we conduct extensive experiments on two real-world network datasets . On one hand , we evaluate the utility of generated networks by computing a suite of commonly concerned graph properties to compare the global structure of generated networks with the original ones . On the other hand , we validate the privacy of individual links by evaluating links predicted from the generated networks on the original networks . Consistent experimental results show that DPGGAN is able to effectively generate networks that are similar to the original ones regarding global network structure , while at the same time useless towards individual link prediction . 2 RELATED WORK . Differential Privacy ( DP ) . Differential privacy is a statistical approach in addressing the paradox of learning nothing about an individual while learning useful information about a population ( Dwork et al. , 2006 ) . Recent advances in deep learning have led to the rapid development of DP-oriented learning schemes . Among them , the Gaussian Mechanism ( Dwork et al. , 2014 ) , defined as follows , provides a neat and compatible framework for DP analysis over machine learning models . Definition 1 ( Gaussian Mechanism ( Dwork et al. , 2014 ) ) . For a deterministic function f with its ` 2-norm sensitivity as ∆2f = max ‖G−G′‖1=1 ‖f ( G ) − f ( G′ ) ‖2 , we have : Mf ( G ) , f ( G ) +N ( 0 , ∆2f2σ2 ) , ( 1 ) where N ( 0 , ∆2f2σ2 ) is a random variable obeying the Gaussian distribution with mean 0 and standard deviation ∆2fσ . The randomized mechanism Mf ( G ) is ( ε , δ ) -DP if σ ≥ ∆2f √ 2 ln ( 1.25/δ ) /ε and ε < 1 . Following this framework , ( Abadi et al. , 2016 ) proposes a general training strategy called DPSGD , which looses the condition on the overall privacy loss than that in Definition 1 by tracking detailed information of the SGD process to achieve an adaptive Gaussian Mechanism . DP learning has also been widely adapted to generative models ( Frigerio et al. , 2019 ; Papernot et al. , 2018 ; Triastcyn & Faltings , 2018 ; Narayanan & Shmatikov , 2008 ; Mohammed et al. , 2011 ; Xie et al. , 2018 ; Chen et al. , 2018 ; Boob et al. , 2018 ; Dy & Krause , 2018 ; Lecuyer et al. , 2018 ; Zhang et al. , 2018 ) . For example , ( Frigerio et al. , 2019 ; Chen et al. , 2018 ; Boob et al. , 2018 ; Zhang et al. , 2018 ) share the same spirit by enforcing DP on the discriminators , and thus inductively on the generators , in a generative adversarial network ( GAN ) scheme . However , none of them can be directly applied to graph data due to the lack of consideration of structure generation . For graphs ’ structural data , two types of privacy constraints can be applied , i.e. , node-DP ( Kasiviswanathan et al. , 2013 ) and edge-DP ( Blocki et al. , 2012 ) , which define two neighboring graphs to differ by at most one node or edge . In this work , we aim at the secure release of network data , and particularly , we focus on edge privacy because it is essential for the protection of object interactions unique for network data compared with other types of data . Several existing works have studied the protection of edge-DP . For example , ( Sala et al. , 2011 ) generates graphs based on the statistical representations extracted from the original graphs blurred by designed noise , whereas ( Wang & Wu , 2013 ) enforces the parameters of dK-graph models to be private . However , based on shallow graph generation models , they do not flexibly capture global network structure that can support various unknown downstream analytical tasks ( Zhang et al. , 2019 ; Wasserman & Zhou , 2010 ) . Graph Generation ( GGen ) . GGen has been studied for decades and is widely used to synthesize network data used to develop various collective analysis and mining models ( Evans & Lambiotte , 2009 ; Hallac et al. , 2017 ) . Earlier works mainly use probabilistic models to generate graphs with certain properties ( Erdős & Rényi , 1960 ; Watts & Strogatz , 1998 ; Barabási & Albert , 1999 ; Newman , 2001 ) , which are manually designed based on sheer observations and prior assumptions . Thanks to the surge of deep learning , many advanced GGen models have been developed recently , which leverage different powerful neural networks in a learn-to-generate manner ( Kipf & Welling , 2016 ; Bojchevski et al. , 2018 ; You et al. , 2018b ; Simonovsky & Komodakis , 2018 ; Li et al. , 2018 ; You et al. , 2018a ; Jin et al. , 2018 ; Grover et al. , 2017 ; De Cao & Kipf , 2018 ; Zou & Lerman , 2018 ; Ma et al. , 2018 ) . For example , NetGAN ( Bojchevski et al. , 2018 ) converts graphs into biased random walks , learns the generation of walks with GAN , and assembles the generated walks into graphs ; GraphRNN ( You et al. , 2018b ) regards the generation of graphs as node-and-edge addition sequences , and models it with a heuristic breadth-first-search scheme and hierarchical RNN . These neural network based models can often generate graphs with much richer properties , and flexible structures learned from real-world graphs . To the best of our knowledge , no existing work on deep GGen has looked into the potential privacy threats laid during the learning and releasing of the powerful models . Such concerns are rather urgent in the network setting , where sensitive information can often be more easily compromised in a collective manner ( Dai et al. , 2018 ; Backstrom et al. , 2007 ; Zhang et al. , 2014 ) and privacy leakage can easily further propagate ( Narayanan & Shmatikov , 2009 ; Zügner et al. , 2018 ) .

Summary: This paper considers the problem of releasing sensible structured data, where there are two inlined challenges: 1. The global network structure should be effectively preserved; 2. The link privacy should be rigorously protected. This paper looks at the secure release of network data with deep generative models. Specifically, the paper develops two models, DPGVAE and DPGGan, which can be viewed as a combination of DP-SGD and graph generation techniques. Extensive experiments are carried out on real-world network datasets, and the positive results have shown the effectiveness of the new models.