Complex-valued deep learning with differential privacy

We present ζ-DP , an extension of differential privacy ( DP ) to complex-valued functions . After introducing the complex Gaussian mechanism , whose properties we characterise in terms of pε , δq-DP , Rényi DP and Gaussian DP , we present ζ-DP stochastic gradient descent ( ζ-DP-SGD ) , a variant of DP-SGD for training complex-valued neural networks . We experimentally evaluate ζ-DP-SGD on three complex-valued tasks , i.e . electrocardiogram classification , speech classification and magnetic resonance imaging ( MRI ) reconstruction . Moreover , we benchmark the performance of our methods on real- and complex-valued variants of the CIFAR-10 and MNIST datasets as well as a large range of complex-valued activation functions . Our experiments demonstrate that DP training of complexvalued neural networks is possible with rigorous privacy guarantees and excellent utility and represents a promising technique to mitigate privacy-utility trade-offs . 1 INTRODUCTION . The ability to harness diverse , feature-rich datasets for algorithm training can allow the scientific community to create machine learning ( ML ) models capable of solving challenging data-driven tasks . These include the creation of robust autonomous vehicles ( Rao & Frtunikj , 2018 ) , earlystage cancer discovery ( Cruz & Wishart , 2006 ) or disease survival prediction ( Rau et al. , 2018 ) . A subclass of these ML problems is able to profit particularly from the ability to execute deep learning workflows over complex-valued datasets , such as magnetic resonance imaging ( MRI ) ( Virtue et al. , 2017 ) or time-series data ( Fan & Xiong , 2013 ; Kociuba & Rowe , 2016 ) . Complex-valued deep learning has seen increased traction in the past years , owing in part to the improved support by ML frameworks and the broader availability of graphics processing unit ( GPU ) hardware able to tackle the increased computational requirements ( Bassey et al. , 2021 ) . However , since complex numbers are often used to represent signals derived from sensitive biological or medical records ( Cole et al. , 2020 ; Küstner et al. , 2020 ; Peker , 2016 ) , privacy constraints can render such datasets hard to obtain . The resulting data scarcity impairs effective model training , prompting the adoption of regulationcompliant and privacy-preserving methods for data access . Distributed computation methods such as federated learning ( FL ) ( Konečný et al. , 2016 ) can partially address this requirement by only requiring participants to share results of the local computation rather than exchange data over the network . However , FL on its own has repeatedly been shown to be insufficient in the task of privacy protection ( Geiping et al. , 2020 ; Yin et al. , 2021 ) . Thus , bridging the gap between data protection and utilisation for algorithmic training requires methods able to offer objective privacy guarantees . Differential privacy ( DP ) ( Dwork et al. , 2014 ) has established itself as the cornerstone of such techniques and has been deployed in contexts like the US Census ( Abowd , 2018 ) and distributed learning on mobile devices ( Cormode et al. , 2018 ) . DP ’ s purview has been expanded to encompass deep learning through the introduction of DP stochastic gradient descent ( DP-SGD ) ( Abadi et al. , 2016 ) , allowing for the training of deep neural networks on private data . So far , however , the application of DP to complex-valued ML tasks remains drastically under-explored . Our work attempts to address this challenge through the following contributions : 1 . We extend DP to the complex domain through a collection of techniques we refer to as ζ-DP . We use this term instead of “ complex-valued DP ” for brevity and to avoid confusion with the abbreviation “ cDP ” , which is already used for concentrated DP ( Dwork & Rothblum , 2016 ) . The letter ζ alludes to the complex-valued Riemann ζ function and is intended to convey the notion of “ continuation ” to the complex domain . 2 . We define and discuss the complex Gaussian Mechanism ( GM ) in Section 4.1 and show that its properties generalise corresponding results on real-valued functions . This allows us to interpret the complex GM through the lens of previous work on pε , δq-DP , Rényi-DP ( RDP ) and Gaussian DP ( GDP ) . 3 . To enable the design and privacy-preserving training of complex-valued deep learning models , we introduce ζ-DP-SGD in Section 4.2 . 4 . In Section 5 we experimentally evaluate the aforementioned techniques on several real-life neural network training tasks , i.e . speech classification , abnormality detection in electrocardiograms and magnetic resonance imaging ( MRI ) reconstruction . Moreover , we establish baselines for future work by providing benchmark results on real- and complex-valued variants of the MNIST and CIFAR-10 datasets and on complex neural network activation functions both with and without ζ-DP-SGD . 5 . We conclude that our DP analysis and the utilisation of tight sensitivity accounting through the Wirtinger/CR-calculus allows practitioners to avail themselves of the higher learning capacity of neural networks with complex-valued weights to achieve improved utility with the same privacy guarantees as corresponding real-valued architectures . 2 RELATED WORK . Prior work has addressed several challenges in non-private complex-valued deep learning , including the introduction of appropriate activation functions , and has presented applications in domains such as MRI reconstruction ( Küstner et al. , 2020 ) or time series analysis ( Fink et al. , 2014 ) . For a detailed overview of methodology and applications , we refer to Hirose ( 2012 ) ; Bassey et al . ( 2021 ) . Until recently , deep learning frameworks did not fully support complex arithmetic and automatic differentiation . Hence , previous works ( Trabelsi et al. , 2017 ; Nazarov & Burnaev , 2020 ) express Cn as R2n and use two real-valued channels rather than complex floating-point numbers . This approach can lead to a spurious increase in function sensitivity by incorrectly computing gradient magnitudes and , by extension , to the addition of excessive noise in the private setting , adversely impacting utility . Our work specifically addresses this shortcoming through the use of complex-valued weights and the Wirtinger/CR-calculus ( Wirtinger , 1927 ; Kreutz-Delgado , 2009 ) . Only a limited number of studies have utilised DP techniques in conjunction with complex-valued data ( Fan & Xiong , 2013 ; Fioretto et al. , 2019 ) , however , to our knowledge none has formalised a notion of complex-valued DP or investigated neural network applications . The pε , δq-definition of DP and the Gaussian mechanism are essential to our formalism , and details on their real-valued definitions can be found in Dwork et al . ( 2014 ) . As stated above , DP-SGD was introduced by Abadi et al . ( 2016 ) . Rényi-DP ( RDP ) was introduced by Mironov ( 2017 ) as a relaxation of pε , δq-DP with favourable properties under composition , rendering it particularly useful for DP-SGD privacy accounting . Gaussian DP ( GDP ) was introduced by Dong et al . ( 2019 ) and tailored to the specific properties of the Gaussian mechanism , for which it provides a tight privacy analysis . 3 BACKGROUND . We begin by introducing key terminology required in the rest of our work . We assume that a trusted analyst in possession of sensitive data wishes to publish the results of some analysis performed on this data while offering the individuals to whom the data belongs a DP guarantee . We will refer to the set of all sensitive records as the sensitive database D , whereby we assume that one individual ’ s data is only present in the database once . Let X , the metric space of all sensitive databases , be equipped with the Hamming metric dX and letD P X . D ’ s adjacent databaseD1 can be constructed from D by adding or removing exactly one database row ( that is , one individual ’ s data ) , such that dXpD , D1q “ 1 . The analyst executes a query ( function ) f , for example a mean calculation , over the database . We first define the sensitivity of f : Definition 1 ( Sensitivity ∆ of f ) . Let X and dX be defined as above . f maps the elements of X to elements of a metric space Y equipped with a metric dY . The ( global ) sensitivity ∆ of f is then defined as : ∆pfq “ max D , D1PX dY pfpDq , fpD1qq dXpD , D1q , D ‰ D1 . ( 1 ) The maximum is taken over all adjacent database pairs in X . When Y is the Euclidean space and dY is the L2 metric , ∆ is referred to as the L2 sensitivity . We will only use the L2 sensitivity in this work . In private data analysis and ML , we are often concerned with differentiable functions ; for Lipschitzcontinuous query functions , the equivalence of the Lipschitz constant and the L2-sensitivity ( Raskhodnikova & Smith , 2016 ) can be exploited : Definition 2 ( Lipschitz constant K of f ) . Let X , Y , dX and dY be defined as above . Then f is said to be K-Lipschitz continuous if and only if a non-negative real number Kf exists for which the following holds : dY pfpDq , fpD1qq ď Kf dXpD , D1q . ( 2 ) Evidently , Kf ” ∆ by Equation ( 1 ) and the definition of adjacency . Moreover , let D be the differential operator ; then Kf “ sup } Dpfq } , where } ¨ } is the operator norm ( O ’ Searcoid , 2006 ) . Therefore , for a scalar-valued query function , ∆ ” Kf ” sup } ∇f } 2 . A DP mechanism adds noise to the results of f calibrated to its sensitivity . Here , we provide the definition of the ( real-valued ) Gaussian mechanism ( GM ) : Definition 3 ( Gaussian mechanism ) . The Gaussian mechanism M operates on the results of a query function f : Rn ÞÑ Rd with sensitivity ∆ over a sensitive databaseD by outputting fpDq ` ξ , where ξ „ N p0 , σ2 Idq . Here , Id denotes the identity matrix with d diagonal elements and σ2 is the variance of Gaussian noise calibrated to ∆ . The application of the GM with properly calibrated noise satisfies pε , δq-DP : Definition 4 ( pε , δq-DP ) . The randomised mechanism M preserves p , δq-DP if , for all pairs of inputs D and D1 and all subsets S of M ’ s range : PpMpfpDq P Sqq ď eε PpMpfpD1q P Sqq ` δ . ( 3 ) A number of relaxations have been proposed to characterise the properties of the GM , of which Rényi DP is arguably the most widely employed in DP deep learning frameworks owing to its favourable properties under composition . Definition 5 ( Rényi DP ) . M preserves pα , ρq-Rényi-DP ( RDP ) if , for all pairs of inputs D and D1 : Dα ` MpfpDqq ‖ MpfpD1qq ˘ ď ρ ( 4 ) where Dα denotes the Rényi divergence of order α ą 1 between MpfpDqq and MpfpD1qq . At α “ 1 , the divergence is defined by continuity as the Kullback-Leibler divergence . We note that the term “ between ” is an abuse of terminology , as the Rényi divergence is asymmetric . In general , we will use Dαpp , qq to denote suptDαpp , qq , Dαpq , pqu . Gaussian DP ( GDP ) was introduced as a variant of f-DP by Dong et al . ( 2019 ) specifically tailored to the properties of the GM , and provides the tightest possible characterisation of its properties . Relying on statistical hypothesis testing , f-DP interprets DP through a trade-off function between the Type I and Type II statistical errors faced by an adversary trying to determine whether one of the adjacent databases contains the individual or not . GDP is a specialisation of f-DP when the trade-off-function has the form Gµ : “ T pN p0 , 1q , N pµ , 1qq . Definition 6 ( Gaussian DP ) . M preserves µ-Gaussian DP ( GDP ) if , for all pairs of adjacent databases D and D1 : T pMpfpDqq , MpfpD1qqq ě Gµ ( 5 ) where T denotes a trade-off function . In this case , Gµpαq “ Φ ` Φ´1p1´ αq ´ µq ˘ where α is the Type I statistical error and Φ is the cumulative distribution of the standard , real-valued normal distribution . 4 ζ -DP In this section we introduce ζ-DP , an extension of DP to complex-valued query functions and mechanisms . ζ-DP generalises real-valued DP and allows the re-use of prior theoretical results and software implementations .

Summary: This paper proposes a novel and comprehensive direction in private deep learning, namely the complex-valued private learning. A new notion of privacy--$\xi$-DP, is defined and carefully analyzed. For this new privacy, a new Gaussian mechanism allows novel DP optimizers to be developed. Extensive experiments demonstrate good performance by combining differential privacy and complex-valued deep learning.