Differentially Private Generative Models Through Optimal Transport

1 INTRODUCTION . As the full value of data comes to fruition through a growing number of data-centric applications ( e.g . recommender systems ( Gomez-Uribe & Hunt , 2016 ) , personalized medicine ( Ho et al. , 2020 ) , face recognition ( Wang & Deng , 2020 ) , speech synthesis ( Oord et al. , 2016 ) , etc . ) , the importance of privacy protection has become apparent to both the public and academia . At the same time , recent Machine Learning ( ML ) algorithms and applications are increasingly data hungry and the use of personal data will eventually be a necessity . Differential Privacy ( DP ) is a rigorous definition of privacy that quantifies the amount of information leaked by a user participating in any data release ( Dwork et al. , 2006 ; Dwork & Roth , 2014 ) . DP was originally designed for answering queries to statistical databases . In a typical setting , a data analyst ( party wanting to use data , such as a healthcare or marketing company ) sends a query to a data curator ( party in charge of safekeeping the database , such as a hospital ) , who makes the query on the database and replies with a semi-random answer that preserves privacy . Differentially Private Stochastic Gradient Descent ( DPSGD ) 1 ( Abadi et al. , 2016 ) is the most popular method for training general machine learning models with DP guarantees . DPSGD involves large numbers of queries , in the form of gradient computations , to be answered quickly by the curator . This requires technology transfer of model design from analyst to curator , and strong computational capacity be present at the curator . Furthermore , if the analyst wants to train on multiple tasks , the curator must subdivide the privacy budget to spend on each task . As few institutions have simultaneous access to private data , computational resources , and expertise in machine learning , these requirements significantly limit adoption of DPSGD for learning with privacy guarantees . To address this challenge , generative models—models with the capacity to synthesize new data— can be applied as a general medium for data-sharing ( Xie et al. , 2018 ; Augenstein et al. , 2020 ) . The 1Including any variants that use gradient perturbation for ensuring privacy . curator first encodes private data into a generative model ; then , the model is used by the analysts to synthesize similar yet different data that can train other ML applications . So long as the generative model is learned ” privately ” , the user can protect their privacy by controlling how specific the generative model is to their own data . Differentially private learning of generative models has been studied mostly under the Generative Adversarial Networks ( GAN ) framework ( Xie et al. , 2018 ; Torkzadehmahani et al. , 2019 ; Frigerio et al. , 2019 ; Yoon et al. , 2019 ; Chen et al. , 2020 ) . While GANs in the non-private setting have demonstrated the ability to synthesize complex data like high definition images ( Brock et al. , 2019 ; Karras et al. , 2020 ) , their application in the private setting is more challenging . This is in part because GANs suffer from training instability problems ( Arjovsky & Bottou , 2017 ; Mescheder et al. , 2018 ) , which can be exacerbated when adding noise to the network ’ s gradients during training , a common technique to implement DP . Because of that , GANs typically require careful hyperparameter tuning and supervision during training to avoid model collapse . This goes against the principle of privacy , where repeated interactions with data need to be avoided ( Chaudhuri & Vinterbo , 2013 ) . Optimal Transport ( OT ) is another method to train generative models . In the optimal transport setting , the problem of learning a generative model is framed as minimizing the optimal transport distance , a type of Wasserstein distance , between the generator-induced distribution and the real data distribution ( Bousquet et al. , 2017 ; Peyré & Cuturi , 2019 ) . Unfortunately , exactly computing the OT distance is generally expensive . Nevertheless , Wasserstein distance-based objectives are actually widely used to train GANs ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017b ) . However , these approaches typically estimate a Wasserstein distance using an adversarially trained discriminator . Hence , training instabilities remain ( Mescheder et al. , 2018 ) . An alternative to adversarial-based OT estimation is provided by the Sinkhorn divergence ( Genevay et al. , 2016 ; Feydy et al. , 2019 ; Genevay et al. , 2018 ) . The Sinkhorn divergence is an entropyregularized version of the exact OT distance , for which the optimal transport plan can be computed efficiently via the Sinkhorn algorithm ( Cuturi , 2013 ) . In this paper , we propose DP-Sinkhorn , a novel method to train differentially private generative models using the Sinkhorn divergence as objective . Since the Sinkhorn approach does not intrinsically rely on adversarial components , it avoids any potential training instabilities and removes the need for early stopping . This makes our method easy to train and deploy in practice . As a side , we also develop a simple yet effective way to perform conditional generation in the Sinkhorn framework , by forcing the optimal transport plan to couple same-label data closer together . To the best of our knowledge , DP-Sinkhorn is the first fully OT-based approach for differentially private generative modeling . Experimentally , despite its simplicity DP-Sinkhorn achieves state-of-the-art results on image-based classification benchmarks that use data generated under differential privacy for training . We can also generate informative RGB images , which , to the best of our knowledge , has not been demonstrated by any generative models trained with differential privacy and without auxiliary public data . We make the following contributions : ( i ) We propose DP-Sinkhorn , a flexible and robust optimal transport-based framework for training differentially private generative models . ( ii ) We introduce a simple technique to perform label-conditional synthesis in the Sinkhorn framework . ( iii ) We achieve state-of-the-art performance on widely used image modeling benchmarks . ( iv ) We present informative RGB images generated under strict differential privacy without the use of public data . 2 BACKGROUND . 2.1 NOTATIONS AND SETTING . Let X denote a sample space , P ( X ) all possible measures on X , and Z ⊆ Rd the latent space . We are interested in training a generative model g : Z 7→ X such that its induced distribution µ = g ◦ ξ with noise source ξ ∈ P ( Z ) is similar to observed ν through an independently sampled finite sized set of observations D = { y } N . In our case , g is a trainable parametric function with parameters θ . 2.2 GENERATIVE LEARNING WITH OPTIMAL TRANSPORT . Optimal Transport-based generative learning considers minimizing variants of the Wasserstein distance between real and generated distributions ( Bousquet et al. , 2017 ; Peyré & Cuturi , 2019 ) . Two key advantages of the Wasserstein distance over standard GANs , which optimize the JensenShannon divergence ( Goodfellow et al. , 2014 ) , are its definiteness on distributions with nonoverlapping supports , and its weak metrization of probability spaces ( Arjovsky et al. , 2017 ) . This prevents collapse during training caused by discriminators that are overfit to the training examples . Methods implementing the OT framework use either the primal or the dual formulation . For generative learning , the dual formulation has been more popular . Under the dual formulation , the distance between the generator-induced and the data distribution is computed as the expectation of potential functions over the sample space . In WGAN and variants ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017a ; Miyato et al. , 2018 ) , the dual potential is approximated by an adversarially trained discriminator network . While theoretically sound , these methods still encounter instabilities during training since the non-optimality of the discriminator can produce arbitrarily large biases in the generator gradient ( Bousquet et al. , 2017 ) . The primal formulation involves solving for the optimal transport plan—a joint distribution over the real and generated sample spaces . The distance between the two distributions is then measured as the expectation of a point-wise cost function between pairs of samples as distributed according to the transport plan . Thus , the properties of the point-wise cost function are of great importance . In practice , sufficiently convex cost functions allow for an efficient optimization of the generator . It is also possible to learn cost functions adversarially ( Szabó & Sriperumbudur , 2017 ) . However , as discussed earlier , adversarial training often comes with additional challenges , which can be especially problematic in the differentially private setting . In general , finding the optimal transport plan is a difficult optimization problem due to the constraints of equality between its marginals and the real and generated distributions . Entropy Regularized Wasserstein Distance ( ERWD ) imposes a strongly convex regularization term on the Wasserstein distance , making the OT problem between finite samples solvable in linear time ( Peyré et al. , 2019 ) . Given a positive cost function c : X × X 7→ R+ and ≥ 0 , the EERWD is defined as : Wc , ( µ , ν ) = min π∈Π ∫ c ( x , y ) π ( x , y ) + ∫ log π ( x , y ) dµ ( x ) dν ( y ) dπ ( x , y ) ( 1 ) where Π = { π ( x , y ) ∈ P ( X × X ) | ∫ π ( x , · ) dx = ν , ∫ π ( · , y ) dy = µ } . Sinkhorn divergence uses cross correlation terms to cancel out the entropic bias introduced by ERWD . This results in faithful matching between the generator and real distributions . In this paper , we use the Sinkhorn divergence as defined in Feydy et al . ( 2019 ) . Definition 2.1 ( Sinkhorn Divergence ) The Sinkhorn divergence between measures µ and ν is defined as : Sc , ( µ , ν ) = 2Wc , ( µ , ν ) −Wc , ( µ , µ ) −Wc , ( ν , ν ) ( 2 ) Works on the efficient computation of the Sinkhorn divergence have yielded algorithms that converge to the optimal transport plan within tens of iterations ( Cuturi , 2013 ; Feydy et al. , 2019 ) . Gradient computation follows by taking the Jacobian vector product between the cost matrix Jacobian and the transport weights , which is implemented in many auto-differentiation frameworks . When compared to WGAN , learning with the Sinkhorn divergence has distinct differences . First , the Sinkhorn divergence is computed under the primal formulation of OT , whereas WGAN ’ s loss is computed under the dual formulation . While both are approximations to the exact Wasserstein distance , the source of the approximation error differs . The Sinkhorn divergence uses entropic regularization to ensure linear convergence when finding the optimal transport plan π ( Cuturi , 2013 ) . Its two sources of error are the suboptimality of the transport plan and bias introduced by entropic regularization . With the guarantee of linear convergence , π can converge to optimality by using enough iterations , thereby allowing control of the first error . The second source of error can be controlled by using small values of , which we found to work well in practice . In contrast , WGAN ’ s source of error lies in the sub-optimality of the dual potential function . Since this potential function is parameterized by an adversarially trained deep neural network , it enjoys neither convergence guarantees nor feasibility guarantees . Furthermore , the adversarial training scheme can produce oscillatory behavior , where the discriminator and generator change abruptly every iteration to counter the strategy of the other player from the previous iteration ( Mescheder et al. , 2017 ) . These shortcomings contribute to WGAN ’ s problems of non-convergence , which in turn can lead to mode dropping . In contrast , training with the Sinkhorn divergence does not involve any adversarial training at all , converges more stably , and reaps the benefits of OT metrics at covering modes .

Summary: This paper presents a differentially private method for training a generative model. The proposed method takes advantage of the Sinkhorn divergence to achieve robustness against the hyperparameters' choices. The authors also introduce a cost function enabling the generative model to generate images associated with a specific class label. The experimental results show that the proposed method outperforms the existing methods for learning a generative model in a differentially private manner. Furthermore, such a high accuracy can be achieved without the use of publicity available data.