Complex-valued deep learning with differential privacy
Authors: Alexander Ziller
Dmitrii Usynin
Moritz Knolle
Kerstin Hammernik
Daniel Rueckert
Georgios Kaissis
Venue: ICLR 2022 Submitted (2022)
Keywords: Differential privacy
complex-valued deep learning

Abstract: We present $\zeta$-DP, an extension of differential privacy (DP) to complex-valued functions. After introducing the complex Gaussian mechanism, whose properties we characterise in terms of $(\varepsilon, \delta)$-DP and Rényi-DP, we present $\zeta$-DP stochastic gradient descent ($\zeta$-DP-SGD), a variant of DP-SGD for training complex-valued neural networks. We experimentally evaluate $\zeta$-DP-SGD on three complex-valued tasks, i.e. electrocardiogram classification, speech classification and magnetic resonance imaging (MRI) reconstruction. Moreover, we provide $\zeta$-DP-SGD benchmarks for a large variety of complex-valued activation functions and on a complex-valued variant of the MNIST dataset. Our experiments demonstrate that DP training of complex-valued neural networks is possible with rigorous privacy guarantees and excellent utility.

Under review as a conference paper at ICLR 2022
C-DP: COMPLEX- VALUED DIFFERENTIAL PRIVACY AND ITS APPLICATIONS TO NEURAL NETWORK TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
We present (-DP, an extension of differential privacy (DP) complex-valued functions. After introducing the complex Gaussian mechanism, whose proper- ties we characterise in terms of (s, 6DP Rényi DP and Gaussian DP, we present C-DP stochastic gradient descent (G-DP-SGD), a variant of DP-SGD for train- ing complex-valued neural networks. We experimentally evaluate (-DP-SGD on three complex-valued tasks, i.e. electrocardiogram classification, speech classi- fication and magnetic resonance imaging (MR) reconstruction. Moreover, we benchmark the performance of our methods real- and complex-valuee variants of the CIFAR- 10 and MNIST datasets as well as a large range of complex-valus activation functions. Our experiments demonstrate that DP training of complex- valued neural networks is possible with rigorous privacy guarantees and excellent utility and represents promising technique mitigate privacy-utility trade-offs.
INTRODUCTION
The ability to hamnsss diverse, feature-rich datasets for algorithm training can allow the scientific community create machine learning (ML) models capable of solving challenging data-driven tasks. These include the creation of robust autonomous vehicles (Rao & Frtunikj] 2018). early- stage cancer discovery (Cruz & Wishart 2006 or disease survival prediction (Rau al. 20.8)) A subclass of these MI. problems is able to profit particularly from the ability execute deep learning workflows over complex-vai valued datasets, such as magnetic resonance imaging (MRI) (Virtue et al.,
2017) time-series data (Fan & Xiong 2013; Kociuba & Rowe, 2016). Complex-valued deep learning has seen increased traction in the past years, owing in part to the improved support by ML frameworks and the broader availability of graphics processing unit (GPU) hardware able tackle the increased computational requirements (Bassey et al. 2021). However, since complex numbers often used represent signals derived from sensitive biological medical records (Cole al.) 2020 Küstner et al. 2020 Peker (0166), privacy constraints can render such datasets hard to obtain. The resulting data scarcity impairs effective model training, prompting the adoption of regulation- methods for data access.
compliant and
Distributed computation methods such federated learning (FL) (Konečný et al. 2016) can partially address this requirement by only requiring participants to share results of the local computation rather than exchange data over the network. However, FL on its own has repeatedly been shown to be insufficient in the task of privacy protection (Geiping et al. (2020] Yin et al. (02T). Thus, bridging the gap between data protection and utilisation for algorithmic training requires methods able to offer objective privacy guarantees. Differential privacy (DP) (Dwork al. (014)) has established itself as the cornerstone such techniques and has been deployed in contexts like the US Census (Abowd) 2018) and distributed learning mobile devices (Cormode et al. 2018). DP's purview has been expanded to encompass deep learning through the introduction of DP stochastic gradient descent (DP-SGD) (Abadi et al. 2016). allowing for the training of deep neural networks on private data. So far, however, the application of DP to complex-vall valued ML tasks remains drastically under-explored.
Our work attempts address this challenge through the following contributions:

---

Under review as a conference paper at ICLR 2022
1. We extend DP to the complex domain through collection of techniques we refer to as C-DP. We use this term instead of "complex-valuce DP" for brevity and to avoid confu- sion with the abbreviation "cDP", which is already used for concentrated DP (Dwork & [Rothblum 2016). The letter Ç alludes to the complex-value Riemann Ç function and is intended to convey the notion of "continuation"
the complex domain.
2. We define and discuss the complex Gaussian Mechanism (GM) in Section and show that its properties generalise corresponding results on real-valued functions. This allows us to interpret the complex GM through the lens of previous work on (c, 6)-DP, Rényi-DP (RDP) and Gaussian DP (GDP).
3. To enable the design and privacy-preserviie training of complex- valuud deep learning mod- els, we introduce <DDP-SGD in Section
4. In Section we experimentally evaluate the aforementioned techniques on several real-life neural network training tasks, i.c. speech classification, abnormality detection in electro- cardiograms and magnetic resonance imaging (MRI) reconstruction. Moreover, we estab- lish baselines for future work by providing benchmark results on real- and complex-value variants of the MNIST and CIFAR-10 10 datasets and on complex neural network activation functions both with and without (-DP-SGD.
5. We conclude that our DP analysis and the utilisation of tight sensitivity accounting through the Wirtinger/CR-caleubb allows practitioners to avail themselves of the higher learning capacity of neural networks with complex-valll valuedd weights to achieve improved utility with the same privacy guarantees corresponding real-valued architectures.
2 RELATED WORK
Prior work has addressed several challenges in non-private complex-valuee deep learning, including the introduction of appropriate activation functions, and has presented applications in domains such as MRI reconstruction (Küstner et al. (022)) or time series amalysis (Fink el al. (2014). For a detailed overview of methodology and applications, we refer to Hirose (2012); Bassey al. (2021). Until recently, deep learning frameworks did not fully support complex arithmetic and automatic differ- entiation. Hence, previous works (Trabelsi et al. 2017; Nazarov & Burnaev 2020) express C as
R and use two real-valued channels rather than complex floating-poii point numbers. This approach can lead to a spurious increase function sensitivity by incorrectly computing gradient magnitudes and, by extension, to the addition of excessive noise in the private setting, adversely impacting util- ity. Our work specifically addresses this shortcoming through the use of complex-valuse and the Wirtinger/CRRccrt -aalculus (Wirtinger 1927 Kreutz- Delgado 2009). Only limited number of studies have utilised DP techniques in conjunction with complex- complex-valued data (Fan & Xiong, 2013] Fioretto et al., 2019), however, to our knowledge none has formalised notion of complex-valued DP or investigated neural network applications.
weights
The (c, -definiiioo of DP and the Gaussian mechanism essential our formalism, and details on their real-valued definitions can be found in Dwork et al. (2014). As stated above, DP-SGD was introduced by Abadi al. (2016). Rényi-DP (RDP) was introduced by MMronov] (2017)
relaxation of (s, 6D- with favourable properties under composition, rendering it particularly useful for DP-SGD privacy accounting. Gaussian DP (GDP) was introduced by Dong al. (2019) and tailored to the specific properties of the Gaussian mechanism, for which it provides tight privacy analysis.
3 BACKGROUND
We begin by introducing key terminology required in the rest of our work. We assume that a trusted analyst in possession of sensitive data wishes to publish the results of some analysis performed on this data while offering the individuals to whom the data belongs DP guarantee. We will refer to the set of all sensitive records as the sensitive database D, whereby we assume that one individual's data is only present in the database once. Let X, the metric space of all sensitive databases, be equipped with the Hamming metric dx and let D E X. D's adjacent database D' can be constructed from D by adding removing exactly one database row (that is, one individual's data), such that
2

---

Under review as a conference paper at ICLR 2022
D') 1. The analyst executes query (function) f. for example mean calculation, over the database. We first define the sensitivity of f:
Definition 1 (Sensitivity A of f). Let X and dx be defined as above. f maps the elements of X to elements of a metric space equipped with metric dy. The (global) sensitivity A of f is then defined as:
max dy dv(J(D),J(D)) D D'.
A
(1)
D.D'EX
dx
The maximum is taken over all adjacent database pairs in X. When Y is the Euclidean space and dy is the L2 metric, is referred to as the L2 sensitivity. We will only use the L2 sensitivity in this work.
In private data analysis and ML, we are often concerned with differentiable functions; for Lipschitz- continuous query functions, the equivalence of the Lipschitz constant and the Le-seniitivity Raskhodnikkov Smith 2016) can be exploited:
Definition 2 (Lipschitz constant K of Π). Let X,Y,d, d.x and dy be defined as above. Then f is said to be K- KiLiscchitz continuous if and only if a non-negative real number K exists for which the following holds:
dy(f(D). f(D') < dx,,
Evidently, Ky = by Equation (1 and the definition of adjacency. Moreover, let be the dif- ferential operator; then Kイ sup I()), where || is the operator norm (O'rearcoid 2006 Therefore, for a scalar-valued query function, A = K = sup ffl22-
A DP mechanism adds noise the results of f calibrated to its sensitivity. Here, provide the definition of the (real-valued) Gaussian mechanism (GM):
Definition 3 (Gaussian mechanism). The Gaussian mechanism M operates the results of a query function f : IR" + I with sensitivity Å over a sensitive database D by outputting f(D) + E, where & (0, o² Ia). Here, Ia denotes the identity matrix with d diagonal elements and o² is the variance of Gaussian noise calibrated to A.
The application of the GM with properly calibrated noise satisfies (e. 6)-DP: Definition 4 ((E, 6)-DP). The randomised mechanism M preserves (c, 8)PP) if. for all pairs of inputs D and D' and all subsets S of M'ss range:
P(M(f(D) E 5)) < ef P(M(f(D)) E S)) +
(3)
A number of relaxations have been proposed characterise the properties of the GM, of which Rényi DP is arguably the most widely employed in DP deep learning frameworks owing to its favourable properties under composition.
Definition 5 (Rényi DP). M preserves (a, ()-ee)-ed (RDP) if. for all pairs of inputs D and D':
Da (M(F(D) M(F(D')) <
(4)
where Da denotes the Rényi divergence of order a between M(f(D)) and M(f(D)). At = 1, the divergence is defined by continuity as the term "between" an abuse of terminology, the Rényi divergence is asymmetric. In general, we will use D. Dv.v) q) to denote sup(Da<p 4), Da(g,))).
divergence. We note that the
Gaussian DP (GDP) was introduced variant of f-DP by Dong al. (019) specifically tailored to the properties of the GM, and provides the tightest possible characterisation of its properties. Relying on statistical hypothesis testing, f-DP interprets DP through a trade-off function between the Type / and Type II statistical errors faced by an adversary trying to determine whether one of the adjacent databases contains the individual or not. GDP specialisation of f-DP when the made-offf-uncctinn has the form G₁ = T (O, 1), 1)).
Definition (Gaussian DP). M preserves μa- u--auusiin DP (GDP) if. for all pairs of adjacent databases D and D':
((M > G₁ where T denotes trade-off function. In this case, Ga(a) + (a-1(1-0) - nl) where is the Type statistical error and & is the cumulative distribution of the standard, real-valued normal distribution.
(5)
3

---

Under review as a conference paper at ICLR 2022
4 <-DP
In this section we introduce Ç-DP, an extension of DP to complex-valued query functions and mech- anisms. (-DP generalises real-valued DP and allows the re-use of prior theoretical results and soft- ware implementations.
4.1 THE COMPLEX GAUSSIAN MECHANISM
We begin by introducing variant of the GM suitable to query functions with codomain C. Definition (Complex Gaussian mechanism). The complex Gaussian mechanism Mc on f C C² outpus f(D)+E, where 5 ~ NN(( o² La) and (C o²) denotes circularly symmetric complex- valued Gaussian noise with variance o².
Of note, random variable X NC can be constructed by independently drawing two random variables A, from real-valued normal distribution (O, 5-) and outputting X A+ Bi, where i is the imaginary unit. This property is unique to the complex GM and leverages the fact that the inner product in C is non-bilinear to add noise scaled by v2- a to each component of the complex number. Naively "simulating" C as R.2", the latter being equipped with bi-linear inner product, would instead require the addition of independent Gaussian noise to each component of vector with resulting scale 2 a.
We now state our main theoretical results and proof sketches. Detailed proofs can be found in Appendix A.I
Theorem Let f : C Cd be a query function with sensitivity A. Then, Mc. preserves (e, DD))--DD if and only if the following holds Ve > 0, E
[0, 1]:
A
A
8(E)
4
c
(6)
20
20
A
where & denotes the cumulative distribution function of the standard (real-valued) normal distribu- tion.
Proof (Sketch). suffices to show that the magnitude of the privacy-loss random variable 12 bounded by with probability 1 8. This holds, as Q2 is distributed as real-valued normal dis- tribution even when E C. where O is any output of f. seeing as the mean Ω is distributed as (f(D) F(D)))
H(D')) e R, where H denotes the Hermitian transpose.
Theorem 2. Let f be defined as above. Then, Me preserves (a, ()-P) if:
1aA²
P
Proof (Sketcht) The Rényi divergence of order a > 1 between two circularly symmetric, complex- valued normal distributions with means li C" and μn E C" and common variance I,,, is:
(no, M1
Dn (N(Ho, (N((,, a(S(D) ((D)) ((D')) alf(D)- -DDD'
202
a42
(9)
20
202
202
where (.) denotes the inner product.
Theorem 3. Let J be defined as above, A be the Ingsensittvvty of and a be the standard deviation of Mc's 's noise. Let Gy be the trade-off off function of a μ-GDP real-valued GM. Then, if X A/o, Me preserves p-GDP if and only X p. Thus, to preserve p-GDP, must choose o² such that:
(10)
4

---

Under review as a conference paper at ICLR 2022
Proof (Sketch). We will show that T
T (N > G.-
((D))) 2
(11)
A
> Gajo Gp
-= J
(12)
which follows from the fact that [I((D) 2 < A and that G2 is strictly monotonically decreasing in
These findings allow for a seamless transfer of results which apply to real-valued functions to the complex domain. In particular, they yield the following insights:
1. The complex GM inherits all properties of the real-valued GM, such composition and sub-sampling amplification, as well as the tight analysis afforded by GDP.
2. The complex GM, like the raalevalued GM, fully characterised by the sensitivity and the magnitude of the noise a.
3. The GM "naturally fits" (-DP due to the aforementioned convenient properties of the cir- cularly symmetric complex-valle
Gaussian distribution. As an additional counterexam- ple, a complex-valued laplace random variable is naturally non-circular in the complex (and multivariate) case, even when constructed from independent distributions (Kotz et al.] [001I. Moreover, the utilisation of the Li-metric on the output space of f is disadvanta- geous, as even for scalar (complex) outputs, the L1 sensitivity can be higher than the Le sensitivity. Lastly, the utilisation of elliptical Laplace noise is inherently unable to satisfy (E, O)-D in any dimension > (Reimherr & Awan 2019). We thus leave the introduc- tion of alternative strategies for obtaining (e, DD in the complex-valued valued setting to future investigation.
We conclude this section by introducing a modification of the DP stochastic gradient descent (DP- SGD) algorithm, which will be employed in our experimental evaluation.
4.2 GDDP-SGD
The DP-SGD algorithm (Abadi et al., 2016) represents an application of the GM to the training of deep neural networks. Using the terminology above, each training step of the neural network (which, in this setting, represents the query) leads to the release of a privatised gradient. Evidently, the noise magnitude of the GM must be calibrated to the sensitivity of the loss function. However, most neural network loss functions have Lipschitz constant which is too high to preserve DP while maintaining acceptable utility (and generallly -the Lipschitz constant of neural networks is hard to compute (Scaman & Virmaux 2018)). Thus, DP-SGD (Abadi et al. 2016) artificially induces bounded sensitivity condition by clipping the L2-norm of the gradient to a pre-defined value. loss function with real-valuee outputs is required for minimisation (even when the function's 's arguments are complex-valee valued) as the complex plane -contrary to the real number line- does not admit a natu- ral ordering.
Our implementation of the algorithm makes use of Wirtinger (or CR-) calculus (Kreutz-Delgate gado 2009) for gradient computations similar previous works on complex-value deep learning (Virtue et al. 2017 Boeddeker et al. 2017). This technique, discussed in detail Appendix A.5 provides several benefits: It relaxes the requirement for component functions to be holomorphic (that is, dif- ferentiable in the complex sense), only requiring them to be individually differentiable with respect to their real and imaginary components (differentiable in the real sense). For holomorphic functions C C, CIR derivvivisssess nevertheless recover the correct derivative definition. Thus, CR-derivatives can also be used to compute the global sensitivity the C - C-case via the Lipschitz constant. More importantly, for functions C + R. they lead to correct gradient magnitude calculation, whereas expressing complex-valle -aauud functions as vector-valued functions in ? a technique often employed in complex- valued neural network training (Trabelsi et al. 2017). can incur undesirable multiplicative sensitivity increase which would diminish the utility of (-DP-SGD, We exemplify this phenomenon and the noise savings CR-calculus enable in Appendix A.5 and Section[5.1]
<-DP-SGD is presented in Algorithm | and relies on a modification of the gradient clipping step: clip the conjugate gradient, which represents the direction of steepest ascent for loss function
5

---

Under review as a conference paper at ICLR 2022
L with vealevalued outputs and complex- valued arguments:
ac
dc
VL ==
(13)
001
20n
where (f1.....8,,, ,,,) is the conjugate weight vector. We remark that, due to the aforementioned properties of the complex GM, the algorithm is compatible with other first-order optimisers (such as Adam), as well as other clipping techniques, such as per-layer clipping (McMahan et al. 2018) Moreover, newer methods for analysing mechanism composition, such the Fourier Accouniant Kookell et al. 2020) can be used.
Algorithm (-DP-SGD
Require: Database with samples { IN INJ E C", neural network with loss function L and weight vector 0 e C². Hyperparameters: learning rate 7), noise variance o², sampling probability R. gradient norm bound B, total steps T.
Initialize 0, randomly
for e [T] do
Draw lot L, with sampling probability R using Poisson or uniform sampling
Compute per-sample conjugate gradient
For each E Lt, compute (x(x,) - VZ(0,,,,)
Clip conjugate gradient
(x:) - ( (mxx)/mxxx max (,,مي
Apply the Complex Gaussian Mechanism and average
J: I (Σ, (m:) + Nc(00, o² o2 I,))
Descend
01+1 0, 7/gtt
end for
Output updated neural network weight vector 0, and compute the privacy cost.
EXPERIMENTAL EVALUATION
Throughout this section, we present results from the experimental evaluation of (-DP-SGD. De- tails dataset preparation and training can be found in Appendix A.6 All e-values are converted (losslessly) from GDP. RDP guarantees are slightly looser and provided in Appendix A.2] We also provide experimental evaluation of commonly used complex-valued activation functions for neu- ral network training with (-DP-SGD in Appendix [A.3 Lastly, we provide additional benchmark experiments demonstrating a complex-valued variant of the MNIST dataset Appendix A.4
5.1 BENCHMARKING Ç C-DP-SGD ON CIFAR-10
We begin by demonstrating that the utilisation of the complex Gaussian mechanism and (-DP-SGD provides increased model utility for the same privacy budget, even on tasks with real-valued inputs in which no additional information from the input's imaginary component is available. Moreover, empirically confirm that, for (-DP-SGD, leveraging the Wirtinger//CRRcaluuu putations with respect to complex-valued weights results in lower sensitivity penalty, and thus improved performance compared to "simulating" complex-valued neural networks with real-valued weights 2 similar to Trabelsi et al. (2017). Further theoretical and experimental evaluation of this phenomenon can be found in Appendix A.5)
for gradient com-
We selected the CIFAR-10 dataset (Krizhevsky et al., 2009). a challenging dataset in the (real-
valued) DP setting. We selected the model architecture recently reported by Papernot et al. (2020). which we equipped with either complex-valus weight matrix (for training in ""). We trained to the same values as Papernot et al. (2020) (in RDP), but modified each image such that the real component contained the image and the imaginary component was zero-filled filled (for training in C), or added an additional trailing zero-filled channel (for training in " The results were identical when the imaginary/second channel components were filled with the same real-valued image. Table summarises these results. Without DP, the increased
weights (for training C or with an additional

---

Under review as a conference paper at ICLR 2022
learning capacity of the complex- valued weights led to improved test performance compared to real-valued weights, with identical performance for training in C and R2" When DP was employed, the model trained with (-DP-SGD achieved the best performance, outperforming both the real-valued DP model and the R2n "simulated" complex-valued DP model. This result under- scores the aforementioned theoretical finding that the complex Gaussian mechanism (contrary to the naive application of a real-valued multivariate Gaussian mechanism in R²n) provides improved performance for the same privacy budget even in tasks with real-valued input data.
Table 1: Classification results for Sections 5.1
5.2 and 5.3
C: Complex- valued weights. DP:
(G-)DP-SGD. For the CIFAR- 10 dataset, 2 indicates a network architecture utilising two channels simulate complex inputs/ model weights. GDP: Gaussian DP accounting. not used. v: used.
Dataset
C
DP
Aceuracy
ROC-AUC
Fi-Score
Recall
GDP
80%
97%
80%
80%
0
V
X
82%
98%
82%
82%
00
0
CIFAR-I0
V
58%
91%
58%
59%
7.54
10-5
V
61%
92%
61%
62%
7.54
10-5
2n
58% V
90%
58%
57%
7.54
10-5
x 87.6%
91.9%
70.8%
81.0%
60
0
ECG
/
88.5%
92.1%
69.8%
71.4%
1.62
10-4
83.4%
98.0%
83.2%
83.3%
80
SpeechCommands
v
62.5%
93.7%
60.3%
62.6%
1.39
10-5
5.2 PRIVACY-PRESERVE ELECTROCARDIOGRAM ABNORMALITY DETECTION ON WEARABLE DEVICES
The advent of wearable devices incorporating electrocardiography (ECG) sensors has provided con- sumers the ability detect signs of an abnormal heart rhythm. In this section, we demonstrate the utilisation of a small neural network architecture suitable for deployment, e.g. to mobile device connected such biosensor, be trained on ECG data from the the China Physiological Signal Challenge (CPSC) 2018 challenge dataset (Liu et al. 2018). We selected the task of automated Left Bundle Branch Block (LBBB) detection, formulated normal (sinus) rhythm. This task is clinically relevant, as the sudden appearance of LBBB can herald acute coronary syndrome which requires urgent attention avert myocardial infarction. As ECG data constitutes personal health information, its protection is mandated both legally and ethi- cally. We utilised <-DP-SGD for training a complex-valued neural network on Fourier-transformeed ECG acquisitions. We adopt this strategy transform: ECG data can contain high-frequency noise which irrelevant for diagnosis and can be reduced using Fourier filtering. Concurrently, this technique compresses the signal, which drastically reduce the amount of data transferred. Table [ shows classification results and Figure] shows exemplary source data.
binary classification task against
can benefit from two key properties of the Fourier
A
B
Figure Exemplary ECG data used for classification. A shows an example of sinus rhythm (normal ECG) and B shows an example of an ECG exhibiting signs of LBBB. Observe also the high frequency noise around the baseline which can be filtered using the Fourier transform.

---

Under review as a conference paper at ICLR 2022
5.3 DUFFERENTIALLY PRIVATE SPEECH COMMAND CLASSIHICATION FOR VOICE ASSISTANT APPLICATIONS
In recent years, voice assistants have gained popularity in consumer applications such as home speakers, and rely heavily on MI.. Recordings collected from users for training speech processing algorithms can be used in impersonation attacks, resulting in successful identity theft (Sweet 2016) or in acoustic attacks, which trigger unintended behaviour in voice assistants (Yuan et al. 2018 Carlini et al. 2016). Protecting privacy in this setting is therefore paramount to increase trust and applicability, well as safeguard both users and systems from adversarial interference. Convo- lutional neural networks (CNNs) have been demonstrated to yield state-of- of-the-art performance on ppecrogram-transfoommee transformed audio data (Palanisamy et al.] 2020). However, this and other works (Zhou et al. 2021) typically discard the imaginary components. We here experimentally demonstrate the DP training of a 2-dimensional CNN directly the complex spectrogram data. We utilised subset of the SpeechCommands dataset (Warden 2018), specifically samples from the categories "Yes", "No", "Up". "Down", "Left", "Right", "On". "Off", "Stop", and "Go", summing up 8000 ex- amples. We transformed each waveform signal to complex-valuee 2-D spectrogram and used C-DP-SGD to train complex-valuee CNN. These results are summarised in Table T and Figure
spectrogram-
A
Figure 2: Exemplary waveform (A), real (B) and imaginary (C) spectrogram components from the utterance Stop. Spectrograms are log- magnitudee transformed for clarity.
5.4 MRI RECONSTRUCTION
MRI is an important medical imaging modality and has been studied extensively in the context of deep learning (Akçakaya et al. [2119] [ammmrnik et 2018 Küstner et al.) al.] 2020; Muckley et al.) 2020). MRI data is acquired in the so-called k-space. Sampling only subset of k-space data al- lows for considerable speed-up in acquisition time, benefiting patient comfort and costs, however, typically leads image artifacts, which reduce the diagnostic quality of the resulting MR images. Although neural networks have the ability to produce high-quality reconstructions, their usage for this task has been shown sometimes lead to the appearance of spurious image content from the fully-sampled reference images the models have been originally trained on (Hammernik et al. 202T is designed
Muckley et al. 2020 Shimron et al. 2021). DP could counteract such hallucination to limit the effect of individual training examples on model training. However, this positive effect of DP may be counterbalanced by an unacceptable decrease in the diagnostic suitability of the re- constructed images. In this section, we investigate the ramifications of DP on the quality of MRI reconstructions. For this purpose, we trained a complex-valuee U-Net model architecture on the task of reconstructing single-coil knee MRI images from the fastMRI dataset (Zbontar et al. 2018) using apeudo-randon k-space sampling at 4x acceleration. We observed a nearly equivalent performance in the non-DP and the CDDP-SGD settings, whereby the non-DP model enjoyed mance advantage in all metrics. Moreover, to assess the diagnostic suitability of the reconstructed images, we asked diagnostic radiologist who was blinded to whether not C-DP-SGD used,
2% perfor-
8

---

Under review as a conference paper at ICLR 2022
to compare the resulting scans. No differences in diagnostic suitability were observed by the expert in any the reconstructed images.
We thus conclude that -at least with respect to image quality- DP can indeed match the non-pri non-private training of MRI reconstruction models, even at e & 1; we intend to investigate its effect on pre- venting training data hallucination into reconstructed images in future work. Results from these experiments are summarised in Table and Figure
Table 2: Results on the MRI reconstruction task. NMSE: normalised mean squared error, PSNR: peak signal-to-noise ratio in dB, SSIM: structural similarity index metric. GDP: Gaussian DP.
NMSE
PSNR
SSIM
GDP
E
Non-DP
0.042
30.74
0.70
8
0
(-DP-SGD
0.043
30.57
0.69
0.16
10-5
A
B
C
Figure 3: Exemplary reconstruction from coronal proton-density weighted image of the knee. A: reference image, B: reconstruction model trained non-privately, C: reconstruction model trained with SDDP-SGD reconstruction.
CONCLUSION
Our work presents (-DP, an extension of DP to the complex domain and introduces key building blocks of DP model training, namely the complex Gaussian mechanism and (-DP-SGD. Our exper- iments demonstrate that the training of DP complex-valued neural networks is possible with high utility under tight privacy guarantees. Our theoretical analysis allows to leverage the increased learning capacity of complex-vee cemplex-weighted neural networks. We thus show improved learning perfor- mance compared scalar real-valued networks and to networks using two real-valued channels to approximate complex numbers, hence, ignoring the specific relationship between the real and imag- inary complex components. Lastly, Wirtinger calculus and Gaussian DP allow to derive the tight sensitivity estimates and privacy bounds for the complex Gaussian mechanism.
We acknowledge some limitations of our work: Both complex-valuee deep learning and DP incur considerable computational performance penalty. Despite steadily improving complex number sup- port, current deep learning frameworks have not yet implemented full palette of complex-valued layers and activation functions. Moreover, the software framework utilised to computationally re- alise <-DP-SGD in our work relies on multithreading, which suffers from considerable overhead compared to implementations utilising vector instructions and/or bespoke hardware. We discuss the topic of software implementation and provide computational performance benchmarks in Ap- pendices A.7 and A.8 Our investigation highlights requirement for mature software frameworks able to offer feature and performance parity with their real-valued counterparts, which we intend to develop and publish free-and-open-souuee software in the near future.
In conclusion, we contend that (-DP represents promising future research direction for complex- valued and even for purely real-valued tasks, and the improved privacy-utility trade-offss resulting from our work represent a worthwhile contribution to the implementation of DP to broad variety of relevant tasks, which can help increase the amount of data available for scientific studies.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work follows all applicable ethical research standards and laws. All experiments were conducted on publicly available datasets. No new data concerning human animal subjects was generated during our investigation.
REPRODUCIBILITY STATEMENT
We adhere ICLRs reproducibility standards and include all necessary information reproduce our experimental and theoretical results either in the main manuscript or in the Appendix. Theoret- ical results and proofs can be found in the main manuscript, Section 4 and additional information can be found in Appendix A..] Details of dataset preparation and analysis can be found in Appendix A.6 Specifically, it contains details about the used datasets, their number of samples, all training, validation and test splits, as well as preprocessing steps. Furthermore, we describe model architec- tures, employed optimisers, learning rates, and the number of epochs for which models were trained. Lastly, for all DP trainings provide the noise multipliers, L2 clipping norms and sampling rates, as well as the d-values at which the e-values were calculated. Software implementation details and computational resources used can be found in Appendices A.7 and
an7 A.8
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
John M Abowd. The US Census Bureau adopts differential privacy. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2867- 2867, 2018.
Mehmet Akçakaya, Steen Moeller, Sebastian Weingärtner, and Kåmil Uğurbil. Scan-specific robust for k-space interpolation (RAKI) reconstruction: Database-fres deep learning for fast imaging. Magnetic Resonance Medicine, 81(1):439 453, 2019.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128 PMLR, 2016.
Borja Balle and Yu-Xiang Wang. Improving the Gaussian mechanism for differential privacy: Ana- lytical calibration and optimal denoising. In International Conference Machine Learning, pp- 393-403. PMLR, 2018
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing in- terpretations and Rényi differential privacy. In International Conference Artificial Intelligence and Statistics, pp. 2496-2262 2506. PMLR, 2020.
Joshua Bassey, Lijun Qian, and Xianfang Li. survey of complex-valued neural networks. arXiv preprint arXiv:2101.1
2021.
Christoph Boeddeker, Patrick Hanebrink, Lukas Drude, Jahn Heymann, and Reinhold Haeb- Umbach. On the computation of complex-valued gradients with application to statistically op- timum beamforming. arXiv preprint arXiv: 1701.0039, 2017.
DH Brandwood. complex gradient operator and its application in adaptive array theory. In IEE Proceedings HH-Microwaves, Optics and Antennas, volume 130, pp. 11-16. IET, 1983.
Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wenchao Zhou. Hidden voice commands. 25th USENIX Security Symposium, pp. 513-530, 2016.
Soumick Chatterjee, Chompunuch Sarasaen, Alessandro Sciarra, Mario Breitkopf, Steffen Oeltze- Jafra, Andreas Nürnberger, and Oliver Speck. Going beyond the image space: undersampled MRI reconstruction directly in the k-space using a complex valued residual neural network. In 202/ ISMRM & SMRT Annual Meeting & Exhibition, pp. 1757, 2021.
10

---

Under review as a conference paper at ICLR 2022
Elizabeth K Cole, Joseph Y Cheng, John M Pauly, and Shreyas S Vasanawala. Analysis of deep complex-valuee convolutional neural networks for MRI reconstruction. arXiv preprint 00070, 2020.
Graham Cormode, Somesh Jha, Tejas Kulkarni, Ninghui Li, Divesh Srivastava, and Tianhao Wang. Privacy at scale: Local differential privacy in practice. In Proceedings of the 2018 International Conference on Management of Data, pp. 1655-1658, 2018.
Joseph Cruz and David Wishart. Applications machine learning in cancer prediction and prognosis. Cancer informatics, 2:59-77, 2006.
Jinshuo Dong, Aaron Roth, and Weijie Su. arXiv:1905.02383, 2019.
Gaussian differential privacy. arXiv preprint
Cynthia Dwork and Guy N Rothblum. Concentrated differential privacy. arXiv preprint 100.v.1603.01887, 2016.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-407 2014.
Fan and Xiong. Adaptively sharing real-time aggregate with differential privacy. IEEE Trans- actions on Knowledge and Data Engineering (TKDE), 26(9):0944-2066, 2013.
Olga Fink, Enrico Zio, and Ulrich Weidmann. Predicting component reliability and level of degra- dation with complex-values neural networks. Reliability Engineering & System Safety, 121:198- 206, 2014.
Ferdinando Fioretto, Terrence WK Mak, and Pascal Van Hentenryck. Differential privacy for power grid obfuscation. IEEE Transactions on Smart Grid, 1 11(2): 1356--36661666 2019.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting Gradients- How easy it to break privacy federated learning? arXiv preprint arXiv:2003. 2.t::2003./45.. /4053, 2020.
George M Georgiou and Cris Koutsougeras. Complex domain backpropagation. IEEE transactions Circuits and systems II: analog and digital signal processing, 39(5):330-334 1992.
Nitzan Guberman. On complex valued convolutional arXiv:1602.09046, 2016.
neural networks.
arXiv preprint
Kerstin Hammemik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine, 7965)3305--077,, 2018.
Kerstin Hammernik, Jo Schlemper, Chen Qin, Jinming Duan, Ronald M. Summers, and Daniel Rueckert. Systematic evaluation of iterative deep neural networks for fast parallel MRI recon- struction with sensitivity weighttd coil combination. Magnetic Resonance in Medicine, 86(4):
1859-1872, 2021.
Nicholas J Higham. Stability of a method for multiplying complex matrices with three real matrix multiplications. SIAM journal on matrix analysis and applications, 13(3):681-68)) -8777 1992.
Akira Hirose. Complex-valued neural networks, volume 400. Springer Science & Business Media, 2012.
Mary C Kociuba and Daniel B Rowe. Complex-valued time-series correlation increases sensitivity in FMRI analysis. Magnetic resonance imaging, 34(6):765-770, 2016.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint /610.005 2016.
Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics, pp. 2560-2569. PMLR, 2020.
11

---

Under review as a conference paper at ICLR 2022
Samuel Kotz, Tomaz J. Kozubowski, and Krzysztof Podgórski. The Laplace Distribution and Gen- eralizations. Birkhäuser Boston, 2001.
Ken Kreutz-Delgado. The complex gradient operator and the CR-calculus. arXiv preprint
arXiv:0906.4835,, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Thomas Küstner, Niccolo Fuin, Kerstin Hammernik, Aurelien Bustin, Haikun Qi, Reza Hajhosseiny, Pier Giorgio Masci, Radhouene Neji, Daniel Rueckert, René M Botnar, et al. CINENet: deep learning-bas 3D cardiac CINE MRI reconstruction with multi-coil complex- valued 4D spatio- temporal convolutions. Scientific reports, 10(1):1-13, 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. 2010.
Feifei Liu, Chengyu Liu, Lina Zhao, Xiangyu Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, Jianqing Li, and Eddie Ng Yin Kwee. An Open Access Database for Evaluating the Algorithms of Electrocardiogram Rhythm and Morphology Abnormality De- tection. Journal of Medical Imaging and Health Informatics, 8(7):1368-137, 2018.
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference Learning Representations, 2018.
lyya Mironov. Rényi Differential Privacy. 2017 IEEE 30th Computer Security Foundations Sympo- sium (CSF), 2017.
Matthew J Muckley, Bruno Riemenschneider, Alireza Radmanesh, Sunwoo Kim, Geunu Jeong. Jingyu Ko, Yohan Jun, Hyungseob Shin, Dosik Hwang, Mahmoud Mostapha, et al. State-of- the-art Machine Learning MRI reconstruction in 2020: Results of the second fastMRI challenge. arXiv preprint arXiv:2012.06318, 2020.
Ivan Nazarov and Evgeny Burnaev. Bayesian Sparsification of Deep C-valued Networks. In Inter- national Conference on Machine Learning, volume 119, pp. 7230-7242. PMLR, 2020. ISSN: 2640-3498.
Tohru Nitta. An extension of the back-propagation algorithm to complex numbers. Neural Networks, 10(1):139,11455, 1997.
M. Searcoid. Metric Spaces. Springer Undergraduate Mathematics Series. Springer London, 2006. ISBN 9781846286278.
Kamalesh Palanisamy, Dipika Singhania, and Angela Yao. Rethinking CNN models for audio clas- sification. arXiv preprint arXiv:2007.11154, 2020.
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Ülfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. arXiv preprint 11111:2007711191,, 2020.
Musa Peker. An efficient sleep scoring system based on EEG signal using complex-valued machine learning algorithms. Neurocomputing. 207:165-177, 2016.
Qing Rao and Jelena Frtunikj. Deep learning for self-driving cars: Chances and challenges. In Proceedings of the 1st International Workshop on Software Engineering for Al in Autonomous Systems, pp. 35-38, 2018
Sofya Raskhodnikova and Adam Smith. Lipschitz extensions for node-private graph statistics and the generalized exponential mechanism. 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 495-504. IEEE, 2016.
Theng-Shyuse Rau, Pao-Jen Kuo, Peng-Cher Chien, Chun- Ying Huang, Hsiao-Yus Hsieh, and Ching-Hus Hsich. Mortality prediction in patients with isolated moderate and severe traumatic brain injury using machine learning models. PloS one, 13(1 1)e00007192 2018.
Chun-Ying
12

---

Under review as a conference paper at ICLR 2022
Matthew Reimherr and Jordan Awan. Elliptical Perturbations for Differential Privacy. arXiv preprint arXiv:1905.09420, 2019.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation. arXiv preprint arXiv:1805. 10965, 2018.
Simone Scardapane, Steven Van Vaerenbergh, Amir Hussain, and Aurelio Uncini. Complex-valuee -valued neural networks with nonparametric activation functions. IEEE Transactions on Emerging Topics in Computational Intelligence, 100-100, 2018.
Efrat Shimron,
Jonathan I. Tamir, Ke Wang. and Michael Lustig. Subtle Inverse Crimes: Naively training machine learning algorithms could lead to overly-optimistic results. arXiv preprint arXiv:2109.08237, 2021.
Caroline E Sweet. The Hidden Scam: Why Consumers Should No Longer Be Forced to Shoulder the Burden of Liability for Mobile Cramming. Bus. & Tech. L. 11:69, 2016
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, Joao Fe- lipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. arXiv preprint 1705v/7705.0977 2017.
Tim Van Erven and Peter Harremos. Rényi divergence and Kullback-Leibler divergence. JEEE Transactions Information Theory, 60(7):3797-3820, 2014.
Patrick Virtue, Stella Yu, and Michael Lustig. Better than real: Complex-valued neural for MRI fingerprinting. In 2017 IEEE international conference on image processing (ICIP), pp. 3953- 3957. IEEE, 2017.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint Kxiv:1804.032000 2018.
W. Wirtinger. Zur formalen Theorie der Funktionen mehr komplexen Veränderlichen. Mathe- matische Annalen, 37133335733 1927.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through Gradients: Image Batch Recovery via GradInversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, and Carl Gunter. Commandersong: A systematic approach for practical adversarial voice recognition. In 27th USENIX Security Symposium, pp. 49 64,,,, 2018.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprinl arXiv: /8/ 088839, 2018
Quan Zhou, Jianhua Shan, Wenlong Ding, Chengyin Wang, Shi Yuan, Fuchun Sun, Haiyuan Li, and Bin Fang. Cough Recognition Based on Mel-Spectrogram and Convolutional Neural Network. Frontiers Robotics and AI, 8, 2021.
Alexander Ziller, Dmitrii Usynin, Rickmer Braren, Marcus Makowski, Daniel Rueckert, and Geor- gios Kaissis. Medical imaging deep learning with differential privacy. Scientific Reports, 11(1): 1-8, 2021.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOFS OF THEORETICAL RESULTS
Proof of Theorem I. The claim represents generalisation of the Analytic Gaussian Mechanism
(Balle& Wang 2018) to Mc. suffices show that the magnitude of the privacy-loss random
variable ! is bounded by with probability 1 - As shown in Dwork & Rothblum (2016) and Balle & Wang (2018), given some fixed output O Cd S2 is given by:
((Mc = 00
Ω = log ((Mc(((DD)) - 0)
(14)
where log is the natural logarithm, and is distributed as:
W ([(D)-ff -( [f(D) - f(')
(15)
20
o²
As [f(D) transpose, S2 has a real-valued mean and hence follows a real-valued normal distribution, even when 0 e C. From here, the proof proceeds identically the proof to Theorem 8 of (Balle & Wang 2018).
f(D))" e R, where H denotes the Hermitian
For proving Theorem 2, we will rely on the following fact about the Rényi divergence of order a between arbitrary distributions:
Corollary (Definition 2 in (Van Erven & Harremos 2014)). Let P and Q be arbitrary dis- Iributions defined on a measurable space (X, with densities p(x) and q(æ). Then, for a > 1;
0
Da (1 a- log p(x)"q(()) (
dd.
(16)
In particular, for two normal distributions with means and μ4l and common variance o²1:
Da (W (( a (<Mo where (-> denotes the inner product.
(17)
20
We can now prove Theorem
Proof Theorem 2. By Definition 7 and the additive property the Gaussian distribution, the den- sity functions of Mc f(D) and f(D") follow circularly symmetric complex-valus Gaussian distribution with means f(D) and f(D') and common covariance matrix I²1. By substituting in equation (7):
D. (NE(F(D), (2)) {((())
20
((D))) alf(()- - -fD''11
aA2
(18)
20
20
20
Hence, to preserve (a, pRDP, suffices to choose o² such that o² > ad2/2p.
Proof of Theorem 3. We recall that the trade-off function between Mc(f(D)) and Mc(f(D')
given by:
G = T (N(O, (() and is strictly monotonically decreasing μ such that μr 2 p2z then Gyn & Gay- To preserve μ-GDP, we require that:
(19)
T ( G₁.,
(20)
14

---

Under review as a conference paper at ICLR 2022
intuitively, that the outputs of Me are least as hard to distinguish the distributions N 1) and (u, 1) given a single draw. We have:
Φ ( o) [I(D)
((D)))
T (W
(21)
= -(D
(22)
We require
A
-(Dlaa 2 Gajo 3 Gp a
(23)
where we have used the fact that [[((D) S(D) & A and the strict monotonicity of G₁.- Thus, to preserve j-GDP, one must choose 6² such that 0² (A/u)²
A.2 RÉNYI DP ANALYSIS OF THE MAIN MANUSCRIPT EXPERIMENTS
Table an overview of the differences between the privacy bounds of RDP and GDP for the experiments in the main manuscript. We recall that GDP provides tight analysis of the Gaussian mechanism (Dong et al. 2019) and lossless conversion between GDP and (e, 6)-DP is possible, whereas RDP cannot be losslesly converted. Of note, are already using the improved conversion formula from RDP to (e, 6)DP shown Balle et al. (020) instead of the original formula proposed in Mironov (0177)
Table Comparison of privacy analyses using Rényi DP (RDP) and Gaussian DP (GDP).
Rényi DP
Gaussian DP
CIFAR-10
7.63
7.54
10-5
ECG
1.76
1.62
5 10
SpeechCommands
1.47
1.39
10
FastMRI
0.67
0.16
10-5
PhaseMNIST
0.53
0.51
10
A.3. BENCHMARKING COMPLEX-VALUED ACTIVATION FUNCTIONS FOR Ç-DP-SGD
A number of specialised activation functions designed for utilisation with complex-valued neural networks have been proposed literature. To guide practitioner choice our newly proposed setting of (-DP-GGD training. we here provide activation function benchmarks the SpeechCom- mands dataset used in Section 5.3 of the main manuscript. Table / summarises these results. We consistently found the inverted Gaussian (iGaussian) activation function to perform best in the Ç- DP-SGD setting. This may be in part due to its bounded magnitude, thereby recapitulating the effect Papernot al. (2020) discuss for real-valued networks, i.e. that bounded activation functions lead to improved performance in DP-SGD. We leave the further investigation of this finding to future work.
A.4 BENCHMARKING (-DP-SGD ON PHASEMNIST
The MNIST dataset (LeCun et 2010) is widely used as a benchmark dataset in real-valued DP- SGD literature. We here therefore show the experimental evaluation of an adapted, complex-valus version of MNIST, which we term PhaseMNIST. The details of how this dataset can be constructed as well as details on the used model architectures are provided in Appendix A.6 In brief, for each example the original MNIST dataset with label L9ı E (0, (0....,9); 9). we obtain the imaginary component by selecting
image with label L3 such that Lox Lg 9 resulting in input image
arrangement (0, 9). (1, 8) 8).... (9.0) 0). Only the label of the realevalued image is used. The results summarised in Table 5 where also provide baselines for real-valued MNIST training on the same architecture (with real-valued weights). For simplicity, we have included the RDP and GDP guarantees in the same table.
15

---

Under review as a conference paper at ICLR 2022
Table 4: ROC-AUC (TD) of complex-valued activation functions the SpeechCommand dataset trained with identical settings and the same network architecture over five repetitions with <-DP-SGD.
Activation function
Reference
ROC-AUC
Separable Sigmoid
Nitta 1997
52.9 + 0.02%
zReLU
Guberman (0166)
54.7 + 0.03%
Trainable Cardioid (per-feature bias)
Virtue a.. (2017)
80.2 + 0.02%
SigLog
Georgiou & Koutsougeras
1992
87.3 0.01%
Trainable ModReLU (per-feature bias)
Arjovsky al. (2016)
89.0 0.01%
Cardioid
Virtue et al. (2017)
89.2 0.01%
Trainable Cardioid (single bias)
Virtue al. (2017)
89.4 0.02%
ModReLU (single bias)
Arjovsky et al. (2016)
89.5 : 0.01%
cReL.U
Trabelsi et al.
2017
91.9 0.01%
iGaussian
Virtue et al. 2017
93.4 + 0.01%
Table Results for PhaseMNIST training in a private (C-DP-SGD and non non-private (non-DP) fash- ion. Results for real-valued MNIST are provided for approximate comparison using the same model architecture (but with real-valued weights) trained with identical settings.
Accuracy
ROC-AUC
Fi-score
Recall
RDP-s
GDP-s
PhaseMNIST (-DP-SGD
99.0%
100%
99.0%
99.0%
0.53
0.51
10-5
MNIST DP-SGD
95.7%
99.9%
95.6%
95.6%
0.53
0.51
10
5
PhaseMNIST non-DP
99.3%
100%
99.2%
99.2%
8
0
0
MNIST non-DP 97.4%
100%
97.4%
97.4%
8
00
As with the CIFAR-10 benchmarks in the main manuscript, we found that (-DP-SGD outperformed the real-valued network. In this case, additional information is available from the imaginary com- ponent of the image in addition to the higher entropic capacity of the network due to the complex- valued weights. similar phenomenon was observed by Scardapane et al. (2018).
A.5 WEWINNGER/CR-CALCCCUE
In this section, we present key results from Wirtinger (or CR-) calculus which are used in our work. For detailed treatment, we refer to KKreuzz-elgadee (209). Consider function f C. As for real-valued functions, the derivative of a point e can be defined as:
('()) lim f(2 + ((+
5(z) h)
. C.
(24)
h
this limit is defined for the (infinitely many) series approaching =, f is called complex differen- tiable (equivalently, differentiable in the complex sense). If, in addition, f'(z) exists everywhere in the neighbourhood iI of 2, f is called holomorphic. It is also possible to write = x + y and to then express f as two real-valued functions u and u of the variables æ and y::
[ yi) u(ær, w) o(x, () 1, R.
(25)
f' can then be written as:
df
Su dæ
20
(26)
de
d..
If this derivative exists at f called differentiable in the real sense. This interpretation represents C as R22 or, more generally, for vector vector-valued functions, C as 22 The Cauchy- Riemann equations state that, for ff to be holomorphic, it must satisfy:
tu dv and
du
(27)
dr
δy
dx
16

---

Under review as a conference paper at ICLR 2022
As discussed above, the complex plane does not admit natural ordering. Hence, the minimisation
of a complex-valued function is not defined. Therefore, for complex-valued deep learning, we only consider real-valued (loss-) functions f C R. By equation (25). v(2z, y) 0. Thus, by the Cauchy-Riemane equations, such a real-valued function is only holomorphic if:
du du δy
0.
(28)
dx
This means that any holomorphic real-valued function must be constant, which invalidates its use- fulness for optimisation. The VirtingerCCR-deriaatiiie provide alternative interpretation of the Cauchy-Riemann equations which allows us to consider holomorphicity and differentiability in the real sense separately. Thus, they recover the usefulness of interpreting C" as g²n while prevent- ing multiplicative penalties on the gradient norm as a consequence of following this interpreta- tion "too closely". We will motivate this somewhat informal notion with an example below. The Ciangeer/CIR-drivaattiie of f are defined as:
1-160 δx
2
0
-
and 8z =
+
(29)
02
2
dy
2
An immediate consequence of this definition is that the Cauchy- Riemann equations can be expressed as:
(30)
è corresponds to the derivative in the complex sense (that Therefore, if a function f is holomorphic, de is, ) while, if f is differentiable in the real sense, both 2 and 2 are valid (and are conjugates 72 र2
of each other). As stated above, it can be shown that the steepest ascent of f is aligned with -.- In this sense, dz fulfils the role of the V operator for real, scalar- valued loss functions. Evidently, compared to the actual gradient of f in the real sense, the following relationship holds:
a vf 2 30,
as
(31)
00
However, re-defining Vf= DE
is desirable (and correct, as shown by Brandwood (183)).
We will
motivate this requirement with an example: Let fc be a function such that:
fc(2) = 27 = (x + (= yi) = x² +
(32)
The Bartinger/RR-derivaatii of fc is Øʃ= z, whose L2-norm is ||=||2 = |2|| v 1//3. The 02
same output can be realised by interpreting ʃ as function of real-valued vector (5.
(a) aa = =x2 v
(33)
The gradient of fn is fI (2r.2y). whose Ln-norm is f 22 (2
+ [y| 2v 2|2 This undesirable multiplicative penalty, which would translate to a super- fluous multiplicative increase in the noise scale of the GM preserve DP, is a consequence of "ignoring" the connection between real and imaginary part inherent to complex numbers. but not to components of vectors. In fact, 23 is neither equivalent
2² (as would be the case if R where Va E R. = a), nor is it equivalent to aa, a E R22 as the complex inner product lacks the bilin- earity inherent to its real-valued counterpart. Both complications are avoided by the re-definition of the Wirtinger/CRR-derivatti as the gradient used for optimisation, which prompts its utilisation in our work.
To exemplify this finding in the training of neural networks, we conducted the following experiment: We generated one thousand points using the function f(2) = sin(2) + NN 1), e C. We then trained two separate neural networks to fit the sine function and measured the norms of the gradient vectors after 100 training steps (when gradient norms were empirically highest). One network in- cluded complex-valued weights (C), the other simulated complex-valued weights with an additional weight matrix per layer (R²"). Both networks utilised the Cardioid activation function and were
'The term derivative represents an abuse of terminology, as they are formal operators and not derivatives with respect to actual variables. However, the interpretation derivatives is intuitive, and we will thus retain
17

---

Under review as a conference paper at ICLR 2022
trained with the Adam optimiser with learning rate of 0.01 using set of identical random seeds over 100 repetitions for 1000 steps. We utilised the Mean Squared Error (MSE) loss function given by fly.8) Jlly $112. (y. C. For the C network, calculated CR-derivatives, while for the R network, standard gradients were obtained using the PyTorch automatic differentiation system.
Figure shows the gradient norms of the two networks. The 2 network had significantly higher average gradient norms (Student 1-test 0.0043) over the 100 repetitions.
6
L2-norms
4
2
CR calculus
22n
Figure 4: Gradient norms of the C network calculated using the CRRccaculus and of the R network. N = 100 repetitions. Significantly higher gradient norms are observed when the CR- calculus is not used (Student r-test p = 0.0043).
As a note to practitioners, certain deep learning frameworks silently re-scale the Wirtinger/CIRR gradient by 2x to avoid user confusion by lower effective learning rate. To ascertain correct implementation, we therefore recommend examining this behaviour by testing the gradient norm of known functions.
A.6 DATASET PREPARATION AND MODEL TRAINING
A.6.1 CIFAR-10
Dataset preparation We utilised the CIFAR-10 dataset described in Krizhevsky et al. (2009). For complex-valuee training C, we either zeroed or copied each image into the imaginary com- ponent, while for training in 2n two separate training channels in Table are for the zero imaginary/second channel component, but were identical (within < 1% for all metries) for the where the imaginary component was filled with the same image.
used for each tensor. Results
Model training For real-valued non-DP training, we used the model described in Papernot et al. (2020) (see Table 2) with the SGD optimiser at learning rate of 0.1 with a momentum term of 0.9 and cosine learning rate scheduler. The hyperbolic tangent (TanH) activation function was used. For complex-valued non-DP training, we utilised the same architecture and activation functions, albeit with complex weights, or with additional weight matrix per layer. For DP training, used an La clipping norm of 1.0, a noise multiplier of 0.61, a sampling rate of 0.005 and trained for 20 epochs. The SGD optimiser at learning rate of 0.1 with momentum term of 0.9 and a cosine learning rate scheduler was used. Interestingly, this arrangement led to a higher test set performance than reported by Papernot al. (2020) in the non-DP setting, however to reproduce the test set accuracy reported in the real-valued DP setting. For training in 2R we note that multivariate Gaussian noise used instead of circularly symmetric complex-valued Gaussian noise, and "standard" gradient computations were used instead of Wirtinger derivatives.
were unable
A.6.2 ECG DATASET
Dataset preparation We utilised the China Physiological Signal Challenge 2018 (Liu et al. 2018) dataset for this task. We used the normal and left bundle branch block classes and channel 3. The ECGs were loaded from the provided Matlab format using the SciPy library and trimmed or padded to length of 5000. The numpy Fast Fourier Transform implementation used whereby the
18

---

Under review as a conference paper at ICLR 2022
signal was pre-trimmed to length 512. The final dataset consisted of 1012 training examples and 113 testing examples.
Model training We implemented a complex-valueds fully- fully-comnected neural network architecture consisting of input/hidden layers with (512, 256, 128) (G-DP-SGD) units and a single output unit. The cReLU activation function was used both the non-DP and the (-DP-SGD setting. The out- put layer implemented the magnitude operation followed by a logistic sigmoid activation function. Models were trained using the SGD optimiser at a learning rate of 0.08 with an Lg regularisation of 5 x 10ST 3 for non-DP training and learning rate of 0.05 for (-DP-SGD training, respectively, A batch size of 64 was used for non-private training and a sampling rate of 0.063 at a noise multiplier of 5 and L2 clipping norm of 0.5 for (-DP-SGD. was calculated at a 5 of 1012 5 10-4 where 1012 is the number of training samples. Both models were trained for 100 epochs.
A.6.3 SPEECH COMMAND CLASSIFICATION DATASET
Dataset preparation We used a subset of the SpeechConmunds dataset (Warden 2018) as de- scribed above, consisting of 2000 samples each from the categories "Yes", "No", "Up". "Down", "Left", "Right", "On". "Off", "Stop". and "Go". Of these, 7200 examples were used as the training test and 800 as the testing set. The waveform data was decoded using the TensorFlow library and, where necessary, padded to length of 16000 samples. The TensorFlow implementation of the Short time Fourier Transform function was used with a frame length of 255 and a frame step of 128.
Model training For this task, we employed a complex-valuce 2D CNN consisting using filters of size 3 3 without pero-ppdding and stride of 1. The convolutional layers had (8, 16, 32, 64) output filters, whereby MaxPooling layer was used between the second layer and the third layer and an adaptive MaxPooling layer after the final convolutional layer. The convolutional block was followed by a fully connected layer with 64 units and an output layer of 8 units. Both employed the iGaussian activation function. The non-DP model was trained at batch size of 64 for 10 epochs at learning rate of 0.1 using the Stochastic Gradient Descent optimiser, whereas the (-DP-SGD network was trained using sampling rate of 0.009 for epochs with the same learning rate and optimiser, noise multiplier of and an L2 clipping norm of 2. We calculated E at 8-value of 10-5, 5
A.6.4 FASTMRI KNEE DATASET
Dataset preparation We utilised the single coil knee MRI dataset of the fastMRI challenge pro- posed by Zbontar et al. (018)). We used the reference implementation P and employed the default settings using an acceleration rate of 4x and 8% of densely sampled k-space center lines in the mask. Masks are sampled pseudo-randomly during training time. The dataset offers 34742 train and 7135 validation images.
Model training We changed the U-Net network to use complex-valued weights and accept complex-valued inputs instead of the magnitude image employed in the original example. We placed the original ReLU activation functions with CReLU. In the DP setting, we used a noise multiplier of 1.0, L2 clipping norm of 1.0 and sampling rate of 3 x 10-5 and calculated the e at a of 10-5, The learning rate was set to 0.001 using the RMSProp optimiser and a stepwise learning rate scheduler. We trained both in the non-private and the (-DP-SGD setting for 30 epochs and disabled the collection of running statistics in the Batch Normalisation layers to render them compatible with DP (in the utilised library, sample sizes of 1 are used, so Batch Normalisation corresponds to Instance Normalisation in this setting).
A.6.5 PHASEMNIST
Dataset construction As described in Appendix A.4 above, PhaseMNIST is intended as a bench- mark dataset for complex-valued computer vision tasks and contains images of handwritten digits from 0 to o The training set consists of 60000 images and the testing set of 10000 images. For each
https: :ttttss::/iiie wwtphh...oomaeaaeommmmemoeeeeeee httpbb/cco/acccccoreeeeeeeccooce
/ mmw///aeemmmmmm//eeeeeeme///aaet fastmri_
exampl es/ es/unet
"The version of the dataset used in this study will be made publicly available upon acceptance.
19

---

Under review as a conference paper at ICLR 2022
example of the original MNIST dataset, from which PhaseMMNIST is constructed, performed the following procedure: Let Log E (O..... 9] be the label corresponding to the real-valued image. We then constructed the imaginary component by (deterministically) sampling uniformly with replace- ment from the set of images whose label Lg satisfies Log + Lg = 9. We used the label of the real-valued image the label of the overall training example.
Model training We used a complex-valued model consisting of three fully connected layers with (784, 256, 128) units and an output layer of 10 units. The Cardioid activation function was used between layers and the Softmax activation function after the output layer. The model was trained with the Stochastic Gradient Descent optimiser at learning rate of 0.1 both for (-DP-SGD and for non-private training. The non-private model converged after 3 epochs, whereas the (-DP-SGI model required 10 epochs to achieve the same accuracy. The noise multiplier was set to 1.1 and the Le clipping norm to 1.0. The e value was calculated at a 6 = 10-5. A sampling rate of 0.001 was used for (-DP-SGD, and batch size of 64 for non-DP training.
A.7 SOFTWARE LIBRARIES AND COMPUTATIONAL RESOURCES USED
Implementations of the DP-SGD algorithm, and by extension- <-DP-SGD require access to per- example gradients. We utilised the deepee software library Ziller et al. 2021) to implement Ç- DP-SGD, as it is compatible with arbitrary neural network architectures, including such containing complex-valued weights. We report results using uniform without replacement sampling and using the RDP option provided by deepee. For complex-valued neural network components, the PyTorch Complex library (Chatterjee et al., 2021) with PyTorch 1.9 were used. Standard PyTorch was also used to create the 2²' model architectures for the experiments in Section 5.1 TensorFlow 2.4 4 was used for loading data and the Short Time Fourier Transforms discussed above, but no neural network components were used from this library. Experiments were carried out in Python 3.8.5 on a single workstation computer running Ubuntu Linux 20.04 and equipped with single NVidia Quadro RTX 8000 GPU, 12 CPU cores and 64 GB of RAM.
A.8 COMPUTATIONAL CONSIDERATIONS
We conclude by presenting a systematic evaluation of the computational considerations incurred by the utilisation of complex-valued neural networks and by the implementation of (-DP-SGD using the above-mentioned libraries. Two main sources of computational overhead arise between real- valued and complex-valuee neural networks. Complex numbers are internally represented as a pair of 32-bit floating point numbers. This affects inputs and neural network weights. Moreover, even though a complex-values architecture may contain the same number of parameters as its real-valued counterpart, an increased number of computational operations is required in C. For instance, scalar multiplication requires a single multiplication operation in TR. However, in it can require up to 4 multiplications (although this can be reduced to 3 multiplications (Higham 1992)). Perfor- mance moreover depends on whether vector hardware is used and whether complex floating point instructions implemented in the respective framework (e.g., cuDNN). Table A.8 shows results for individual matrix multiplication operations and convolutions with cal/complex-vallus inputs and weight matrices.
Table 6: Average computation times (100 repetitions) for a batched matrix multiplication with batch size 64 and matrix dimensions (512 512) (Linear) and convolution operation with input dimen- sions (64 x 3 x 224 x 224) (batch, channel, height, width) and kernel dimensions (3, 16, 3) (in, out, kernel size) (Conv.). R: Real-valued input and weight matrix and C: complex-valuee input and weight matrix. Times are given on CPU and GPU.
Linear
Conv.
R
C
R
CPU
68.6 js
1.21
31.8 ms
214 ms
GPU
49.6 js
238 ju8
1.38 ms
12.5 ms
20

---

Under review as a conference paper at ICLR 2022
(-DP-SGD carries additional overhead it requires per-sample gradients. In the utilised deepee framework, this is realised through dispatching one computation thread per example in the minibatch (more precisely, lot) to perform a forward and backward pass, which incurs substantial overhead compared to pure vectorisation. These results are shown in Table A.8] Of note, for the non-private models, the computation time includes the forward pass, backward pass, loss gradient calculation (Mean Squared Error against a vector of dimensions (64, 1)) and weight update (Stochastic Gradient Descent). For the (-DP-SGD model, the following additional steps occur between the loss gradient calculation and the weight update: gradient clipping, averaging of per-sample gradients, noise ap- plication. Moreover, the deepee framework requires an additional step between the weight update and the subsequent batch.
Table 7: Average computation times for a model consisting of a 2D convolutional layer with 3 input channels, 32 output channels and a kernel shape of 3 x 3 followed by linear layer with matrix dimensions of (28800, 1) executed on an input of dimensionality (64,3, 3, 32, 32) (batch, channel, height, width) for 100 repetitions.
Non-DP
(G-)DP-SGD
R
C
CPU
156 ms
1.45 s
27.6 s
1.08 min
GPU
187 ms
588 ms
8.52
17.1 s
21


Reviews:
[{"review_id": "z9acWr0GoT2", "reviewer": "Reviewer_kQQG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper to be well written and the problem of  \"complex DP\" well-motivated.\nThe idea of implementing DP-SGD with the complex Wirtinger notion of derivative is particularly elegant and appealing.\n\nHowever, I would like to contend that there the introduced ideas $\\zeta$-DP and complex Gaussian mechanism are not inherently complex.\nIndeed, by considering the identification $\\mathbb{C}^n = \\mathbb{R}^{2n}$. The complex Gaussian mechanism reduces to the standard Gaussian mechanism when the dimension is doubled. The same is also true for $\\zeta$-DP and 'vanilla' DP in real spaces. This is reflected, for example, in the proof of Theorem 1, which is essentially identical to the proof in Balle & Wang (2018) (except the dimension has doubled).\nI would be very happy to be proven wrong on this point and I expect the authors' response.\n\nIn light of the above though, it seems that the main contribution of the paper is to suggest an alternative to DP-SGD, at least when the dimension is even. Since the functions to optimize are necessarily real-valued, otherwise, it's not even clear what is being optimized, the authors suggest using the conjugate of the complex derivative. For real-valued functions, the conjugate is aligned with the regular gradient. Again, I don't find this setting particularly 'complex-valued'. Other than making the gradient smaller by a constant what are the benefits of using the Wirtinger derivative? \nI would at least expect to see some experiments which show that $\\zeta$-DP-SGD on $\\mathbb{C}^n$ outperforms 'vanilla' DP-SGD on $\\mathbb{R}^{2n}$, when optimizing the same function, with the same privacy budget.\nCurrently, the experiments only demonstrate that $\\zeta$-DP-SGD well on a variety of tasks. As the authors note this can be due to the extra information contained in the additional dimensions.\n\nOne final point, which is a matter of personal taste. The title of the paper \"COMPLEX-VALUED  DEEP  LEARNING...\" is not appropriate, and even misleading, in my eyes. Nothing in the suggested framework is specific to deep learning. While the authors do use neural networks (none of which are particularly deep) in their experiments, it is not the main focus of the paper.\n\n\n\n", "summary_of_the_paper": "The paper proposes a new framework, which extends differential privacy to complex-valued functions. The authors name this framework $\\zeta$-DP and introduce their main privacy mechanism, the complex Gaussian mechanism. The authors also show how to adapt the private gradient descent algorithm into a private algorithm to minimize real-valued functions, defined on the complex plane.\nIt is then shown, experimentally, that the new algorithms and notions perform very well on a variety of tasks related to signal processing (and hence to complex-valued functions).\n", "summary_of_the_review": "The authors suggest a new framework for differential privacy in complex spaces, which works well with complex notions of derivatives.\nHowever, as far as I have gathered, this is not particularly different than the existing notion of differential privacy in real spaces and I was not convinced there is an actual benefit to using the complex derivatives.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "SIw6E85v0MU", "reviewer": "Reviewer_SGmD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is very well-written and insightful. The contents are complete, covering all important aspects of DP deep learning. Especially the complex-valued Gaussian distribution, the choice of L2 over L1 sensitivity, the seamless connection between complex-valued privacy accountant and existing ones, the choice of Wirtinger calculus, etc. are very clear and exciting.\n\nThe definition of $\\xi$-DP is exactly $(\\epsilon,\\delta)$-DP but extended to the complex domain. So this is moderately interesting. The true highlights of this paper are the new Gaussian mechanism (as well as the new DP optimizers) and good motivation of complex-valued deep learning.\n\nAs for the weakness, I think there are 2 points:\n(1) this paper under-sells its strength in generalizability. Although the author discussed other privacy accounting like f-DP, I believe more other accounting like Gaussian DP, Fourier accountant and others should be mentioned. Another key part is the optimizer. If there is no fundamental difficulty, the authors should discuss $\\xi$-DP-Adam/Momentum/Adagrad and new clipping method like global clipping. If there is a challenge to design new complex-valued DP optimizers, the authors should be state clearly the reasons.\n(2) the experiments are exciting but can be improved. First of all, MNIST is too simple even for DP learning. I would expect to understand the performance on CIFAR10. Notice that while DP learning can get 99% on MNIST, it at most gets 70% on CIFAR10 without transfer learning. Also Table 2,3,5 look incomplete to me. Why MNIST has accuracy but these tables have ROC? Please include accuracy and possibly other metrics in all tables.", "summary_of_the_paper": "This paper proposes a novel and comprehensive direction in private deep learning, namely the complex-valued private learning. A new notion of privacy--$\\xi$-DP, is defined and carefully analyzed. For this new privacy, a new Gaussian mechanism allows novel DP optimizers to be developed. Extensive experiments demonstrate good performance by combining differential privacy and complex-valued deep learning.", "summary_of_the_review": "The paper is very well-written and insightful. It provides new DP notion, new Gaussian mechanism, new DP optimizer and exciting experiments. Minor weakness exists: the extensions are not well-discussed and experiments can be enhanced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2lwkqkqrM9", "reviewer": "Reviewer_n9FY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: This paper is well structured and its motivation is clear. The experimental results show that the proposed algorithm is possible to achieve high utility under tight privacy guarantees.  \n\nCons: More significant results are needed. My main concerns are as follows:\n\n- On the theory side, the idea seems too natural. It would be better if authors can provide a tighter bound for the privacy loss.\n- On the experiment side, to make the results more convincing, please provide more comprehensive experiment results, such as accuracy for differen", "summary_of_the_paper": "This paper proposes a complex-valued DP-SGD mechanism, named \\zeta-DP-SGD, to train privacy-preserved neural networks on complex-valued data.  In particular, the authors introduce a complex Gaussian mechanism with DP and Renyi-DP properties, \\zeta-DP, to extend DP to the complex domain. \\zeta-DP allows the re-use of prior theoretical results and software implementations. ", "summary_of_the_review": "Although the privacy-preserved complex-valued DL is an interesting problem, I’m not sure this paper reports enough contribution for pushing the development of this research field. The theory for complex Gaussian Mechanism seems to re-use the prior theoretical results. The used gradient computation method based on Wirtinger calculus is similar to the previous works on complex-valued deep learning.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers in general agree that the proposed complex valued DP method is interesting and novel. However, there are two key concerns due to which the paper might not be ready for publication at ICLR: a. the key technical contribution of the work is not clear, as the methods seem relatively straightforward extension of real valued DP methods to complex valued domains. 
b. More importantly, the experimental results (and hence the motivating applications) are not convincing and do not strongly support the claims of i) complex data provides more flexibility and hence provide better model, ii) proposed method is accurate. 
For example, the accuracy numbers for SpeechCommands even without DP seem quite low. For example, standard methods like matchboxnet for keyword detection have accuracy numbers in the range of 97%. While the work considers a subset of keywords, but it would be important to show how the standard methods work on this dataset. If the gap is this large, then the case for using complex valued datasets itself is weak. 
Similarly, on CIFAR10 it seems like that the considered architecture is quite poor as the accuracy is just ~80% while most standard architectures get >93% on the dataset. So the experiment claims of the paper might not hold for practically relevant architectures.
Decision: Reject