PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning

1 INTRODUCTION . Synthesizing data under differential privacy ( DP ) ( Dwork ( 2006 ; 2011 ) ; Dwork & Roth ( 2014 ) ) enables us to share the synthetic data and generative model with rigorous privacy guarantees . Particularly , DP approaches of data synthesis involving the use of deep generative models have received attention lately ( Takagi et al . ( 2020 ) ; Xie et al . ( 2018 ) ; Torkzadehmahani et al . ( 2019 ) ; Frigerio et al . ( 2019 ) ; Yoon et al . ( 2019 ) ; Chen et al . ( 2020 ) ; Harder et al . ( 2021 ) ) . Typically , the training of such models utilizes gradient sanitization techniques ( Abadi et al . ( 2016 ) ) that add noises to the gradient updates to preserve privacy . While such methods are conducive to deep learning , due to composability , each access to data leads to degradation in privacy guarantees , and as a result , the training iteration is limited by the privacy budget . Recently , Harder et al . ( 2021 ) has proposed DP-MERF , which first represents the sensitive data as random features in a DP manner and then learns a generator by minimizing the discrepancy between the ( fixed ) representation and generated data points . DP-MERF can iterate the learning process of the generator without further consuming the privacy budget ; however , it is limited in the learning and generalization capabilities due to its fixed representation . In this work , we seek a strategy of training deep generative models privately that is able to resolve the aforementioned shortcomings , and is practical in terms of privacy ( e.g. , usable image data at ' 1 . ) We propose a private learning framework called PEARL ( Private Embeddings and Adversarial Reconstruction Learning ) . In this framework , we have i ) no limitation in learning iterations , and ii ) well-reconstruction capability . Towards those preferable properties , our framework first obtains ( 1 ) informative embedding of sensitive data and ( 2 ) auxiliary information ( e.g. , hyperparameters ) useful for training , both in a differentially private manner , then ( 3 ) the generative model is trained implicitly like GANs via the private embedding and auiliary information , where the learning is based on a stochastic procedure that generates data , and ( 4 ) a critic distinguishing between the real and generated data . The overview of PEARL is illustrated in Fig . 1 . As a concrete realization of PEARL , We first identify that the characteristic function ( CF ) representation of data can be sanitized as the private embedding of PEARL . Consequently , it is possible to train deep generative models using an appropriately defined metric measuring the discrepancy between the real ( but sanitized ) and generated data distribution based on the CF without re-using the original data . As will be explained in detail in later Sections , the generative modelling approach using CFs also involves sampling “ frequencies ” from an ad hoc distribution , to project the data to the embedding . It is desirable to optimize the sampling distribution to better represent the data as an embedding , but the naive way of optimizing it would require re-accessing the data via sampling , coming at a cost of privacy budget . Henceforth , we also propose to incorporate a privacy-preserving critic to optimize the sampling strategy , which , through re-weighting , chooses the best representation from a fixed samples of frequencies without extra privacy costs . To this end , we propose the following minimax optimization training objective : inf θ∈Θ sup ω∈Ω k∑ i=1 ω ( ti ) ω0 ( ti ) ∣∣Φ̃Pr ( ti ) − Φ̂Qθ ( ti ) ∣∣2 . ( 1 ) See later parts for notations and details . Theoretically , we show that our proposed objective has properties similar to those that are suited to training GANs , i.e. , continuity and differentiability of the generator ’ s parameters , and continuity in weak topology . We also prove the consistency of our privacy-preserving sampling strategy at the asymptotic limit of infinite sampling . Empirical evaluations show that PEARL is able to high-quality synthetic data at reasonable privacy levels . Related works . Traditional methods of synthesizing data are mainly concerned with discrete data or data preprocessed to the discrete form ( Zhang et al . ( 2017 ) ; Qardaji et al . ( 2014 ) ; He et al . ( 2015 ) ; Chen et al . ( 2015 ) ; Cai et al . ( 2021 ) ; Zhang et al . ( 2021 ) ) , whereas we are interested in more general methods involving continuous data . Deep generative models under the DP setting are suitable for this type of tasks ( Takagi et al . ( 2020 ) ; Xie et al . ( 2018 ) ; Torkzadehmahani et al . ( 2019 ) ; Frigerio et al . ( 2019 ) ; Yoon et al . ( 2019 ) ; Chen et al . ( 2020 ) ; Harder et al . ( 2021 ) ) . The private training of deep generative models is usually performed using gradient sanitization methods . An exception is DP-MERF ( Harder et al . ( 2021 ) ) , which is closest to our work . There , random features used to approximate the maximum mean discrepancy ( MMD ) objective are privatized and utilized for training a generator . PEARL , which , as a realization , uses CFs , may be viewed as a generalization of DP-MERF . Additionally , PEARL has several distinctive features which are lacking in DP-MERF . The first lies in the introduction of a privacy-preserving critic , which leads to an improvement of performance . The second is the private selection of the parameter of the sampling distribution , which is also shown to be vital . Moreover , DP-MERF uses non-characteristic kernels when treating tabular data , in contrast to ours , which is characteristic and has guarantees in convergence . We finally note that generative models using CFs but only non-privately have been explored before ( Ansari et al . ( 2020 ) ; Li et al . ( 2020 ) ) . Contributions . Our contribution in this paper is three-fold : ( i ) We propose a general framework called PEARL , where , unlike gradient sanitization methods , the generator training process and iteration are unconstrained ; reliance on ad-hoc ( non-private ) hyperparameter tuning is reduced by extracting hyperparameters ( auxiliary information ) privately . ( ii ) We demonstrate a realization of our framework by making use of the characteristic function and an adversarial re-weighting objective . ( iii ) Our proposal has theoretical guarantees of performance , and empirical evaluations show that our approach outperforms competitors at reasonable levels of privacy ( ' 1 ) . 2 PRELIMINARIES . This Section gives a brief review of essential preliminaries about differential privacy , characteristic function and the related notations . 2.1 DIFFERENTIAL PRIVACY . Definition 1 ( ( , δ ) -Differential Privacy ) . Given privacy parameters ≥ 0 and δ ≥ 0 , a randomized mechanism , M : D → R with domain D and rangeR satisfies ( , δ ) -differential privacy ( DP ) if for any two adjacent inputs d , d′ ∈ D and for any subset of outputs S ⊆ R , the following holds : Pr [ M ( d ) ∈ S ] ≤ e · Pr [ M ( d′ ) ∈ S ] + δ . ( 2 ) We next consider concrete ways of sanitizing certain outputs with DP . A typical paradigm of DP is applying the randomized mechanism , M , to a certain deterministic function f : D → R such that the output of f is DP . The noise magnitude added byM is determined by the sensitivity of f , defined as ∆f = supd , d′∈D ‖f ( d ) − f ( d′ ) ‖ , where || · || is a norm function defined on f ’ s output domain . d and d′ are any adjacent pairs of dataset . Laplacian and Gaussian mechanisms are the standard randomized mechanisms . We primarily utilize the Gaussian mechanism in this paper ( Dwork & Roth ( 2014 ) ) : Definition 2 ( Gaussian Mechanism ) . Let f : X → R be an arbitrary function with sensitivity ∆f . The Gaussian Mechanism , Mσ , parameterized by σ , adds noise to the output of f as follows : Mσ ( x ) = f ( x ) +N ( 0 , σ2I ) . ( 3 ) One of the most important properties of DP relevant to our work is the post-processing theorem ( Dwork & Roth ( 2014 ) ) : Theorem 1 ( Post-processing Theorem ) . LetM : D → R be ( , δ ) -DP and let f : R → R′ be an arbitrary randomized function . Then , f ◦M : D → R′ is ( , δ ) -DP . It ensures that the DP-sanitized data can be re-used without further consuming privacy budgets . 2.2 CHARACTERISTIC FUNCTIONS . Characteristic function ( CF ) is widely utilized in statistics and probability theory , and perhaps is best known to be used to prove the central limit theorem ( Williams ( 1991 ) ) . The definition is as follows . Definition 3 ( Characteristic Function ) . Given a random variable X ⊆ Rd and P as the probability measure associated with it , and t ∈ Rd , the corresponding characteristic function ( CF ) is given by ΦP ( t ) = Ex∼P [ eit·x ] = ∫ Rd eit·xdP . ( 4 ) Here , i is the imaginary number . From the signal processing point of view , this mathematical operation is equivalent to the Fourier transformation , and ΦP ( t ) is the Fourier transform at frequency t. It is noted that we deal with the discrete approximation of CFs in practice . That is , given a dataset with n i.i.d . samples , { xj } nj=1 from P , the empirical CF is written as Φ̂P ( t ) = 1n ∑n i=j e it·xj . We next introduce characteristic function distance ( CFD ) ( Heathcote ( 1972 ) ; Chwialkowski et al . ( 2015 ) ) : Definition 4 ( Characteristic Function Distance ) . Given two distributions P and Q of random variables residing in Rd , and ω a sampling distribution on t ∈ Rd , the squared characteristic function distance ( CFD ) between P and Q is computed as : C2 ( P , Q ) = Et∼ω ( t ) [ ∣∣ΦP ( t ) − ΦQ ( t ) ∣∣2 ] = ∫ Rd ∣∣ΦP ( t ) − ΦQ ( t ) ∣∣2ω ( t ) dt . ( 5 ) Notations . Let us make a short note on the notations before continuing . Let k be the number of t drawn from ω and P be the probability measure of a random variable . We group the CFs associated to P of different frequencies , ( Φ̂P ( t1 ) , . . . , Φ̂P ( tk ) ) > more compactly as φ̂P ( x ) . To make the dependence of φ̂P ( x ) on the sampled data explicit , we also use the fol- lowing notation : φ̂P ( x ) = 1n ∑n j=1 φ̂P ( xj ) . We notice that ‖φ̂P ( x ) ‖2 ≡ √∑k m=1 |Φ̂P ( tm ) |2 =√∑k m=1 | ∑n l=1 e itm·xl/n|2 ≤ √∑k m=1 | ∑n l=1 1/n|2 = √ k , where the norm is taken over the ( complex ) frequency space . With a slight abuse of notation , we abbreviate φ̂P as φ̂ when there is no ambiguity in the underlying probability measure associated with the CF .

Summary: This paper studied an important topic in the field of data synthesis: how to train a private deep generative model without reusing the original data. In this paper, the authors proposed a new framework that uses deep generative models to synthesize data in different private ways. Unlike popular gradient cleaning methods, the framework proposed in this paper does not incur additional privacy costs or model constraints. In addition, in order to avoid the problem of reduced privacy guarantee as training iterations increase, this paper used feature functions and adversarial re-weighting objectives to solve the above problems. Both theory and extensive case studies verify the performance of the proposed framework.