The Second AAAI Workshop on Privacy-Preserving Artificial Intelligence (PPAI-21)

Dopamine: Differentially Private Federated Learning on Medical Data
Mohammad Malekzadeh, Burak Hasircioglu, Nitish Mital, Kunal Katarya,
Mehmet Emre Ozfatura, Deniz Gündüz*
Department of Electrical and Electronic Engineering, Imperial College London.
{m.malekzadeh, b.hasircioglu18, n.mital, kunal.katarya15, m.ozfatura, d.gunduz}@imperial.ac.uk

Abstract
While rich medical datasets are hosted in hospitals distributed
across the world, concerns on patients’ privacy is a barrier
against using such data to train deep neural networks (DNNs)
for medical diagnostics. We propose Dopamine, a system totrain DNNs on distributed datasets, which employs federated
learning (FL) with differentially-private stochastic gradientdescent (DPSGD), and, in combination with secure aggregation, can establish a better trade-off between differential
privacy (DP) guarantee and DNN’s accuracy than other approaches. Results on a diabetic retinopathy (DR) task showthat Dopamine provides a DP guarantee close to the centralized training counterpart, while achieving a better classification accuracy than FL with parallel DP where DPSGD
is applied without coordination. Code is available at https://github.com/ipc-lab/private-ml-for-health.

1

Introduction

Deep neural networks facilitate disease recognition frommedical data, particularly for patients without immediate access to doctors. Medical images are processed with DNNs
for faster diagnosis of skin disease (Esteva et al. 2017),
lung cancer (Dunnmon et al. 2019), or diabetic retinopathy (DR) (Gulshan et al. 2016). However, the memorization capacity of DNNs can be exploited by adversaries
for reconstruction of a patient’s data (Carlini et al. 2019),
or the inference of a patient’s participation in the training
dataset (Shokri et al. 2017; Dwork et al. 2017). Due to suchprivacy risks and legal restrictions, medical data can rarely
be found in one centralized dataset; thus, there has been asurge of interest in privacy and utility preserving training on
distributed medical datasets (Kaissis et al. 2020).
Federated learning (FL) (McMahan et al. 2017) trainsa DNN where hospitals collaborate with a central serverin training a global model on their local datasets. At eachround, the server sends the current model to each hospital, then hospitals update the model on their private datasets
and send the model back to the server. The hospitals’ updates are susceptible to information leakage about the patients’ data due to model over-fitting to training data (Carlini et al. 2019). Differential privacy (DP) (Dwork and Roth2014) limits an adversary’s certainty in inferring a patient’s* This work was funded by the European Research Council (ERC) through Start-

ing Grant BEACON (no. 677854) and by the UK EPSRC (grant no. EP/T023600/1).

presence in the training dataset. Before optimizing the DNN,
Gaussian random noise is added to the computed gradients on the patients’ data to achieve differentially-private
stochastic gradient descent (DPSGD) (Abadi et al. 2016).
We propose Dopamine, a customization of DPSGD for
FL, which, in combination with secure aggregation by homomorphic encryption, can establish a better privacy-utility
trade-off than the existing approaches, as elaborated in Section 3. Experimental results on the DR dataset (Choi et al.
2017), using SqueezeNet (Iandola et al. 2016) as a benchmark DNN, show that Dopamine can achieve a DP boundclose to the centralized training counterpart, while achieving better classification accuracy than FL with parallel DP
where the hospitals apply typical DPSGD on their sides
without any specific coordination. We provide theoreticalanalysis for the guaranteed privacy by Dopamine, and discuss the differences between Dopamine and the seminal
work proposed by (Truex et al. 2019): our solution allows
to properly keep track of the privacy loss at each round as
well as taking advantage of the momentum (Qian 1999) inFL-based DPSGD, which improves the DNN’s accuracy.
The main contribution of this paper is the design andimplementation of FL on medical images while satisfying record-level DP. While previous works on medical
datasets (as discussed in Appendix, Section B) either do notguarantee a formal notion of privacy, e.g. (Li et al. 2020),
or apply weaker notion of DP, e.g. parameter-level DP (Liet al. 2019), to the best of our knowledge, Dopamine is thefirst system that implements FL-based DPSGD that guarantees record-level DP for a dataset of medical images. Finally,
we publish a simulation environment to facilitate further research on privacy-preserving training on distributed medical
image datasets.

2

Dopamine’s Methodology

Problem Formulation1 Let each hospital k∈{1, . . . , K} own a dataset, Dk , with an unknown numberof patients, where each patient i participates with a labeleddata (xi , yi ). The global server owns a validation dataset
1

As notation, we use lower-case italic, e.g. x, for variables; upper-case italic, e.g.
X, for constants; bold font, e.g. X, for vectors, matrices, and tensors; blackboard font,e.g. X, for sets; and calligraphic font, e.g. X , for functions and algorithms.

DG , and we consider a DNN’s utility as its prediction accuracy on DG . The goal is to collaboratively train a DNN
while satisfying record-level DP. We assume a threat model
where patients only trust their local hospital, and hospitals
are non-malicious and non-colluding. The global server ishonest but curious. Finally, hospitals and the server do nottrust any other third parties. We assume the patient’s privacy,
defined by (, δ), is a bound on the record-level DP loss.
The hospitals aim to ensure their patients a computational
DP against the server during training, and an informationtheoretical DP against the server and any other third parties
after training. It is called computational DP as a cryptosystem is only robust against computationally-bounded adversaries. Dopamine assumes that adversaries are computationally bounded during training, which is a typical assumption,
but after training they can be computationally unbounded2 .

Dopamine’s Training
The training procedure is given in Algorithm 1, where we
perform federated SGD among K hospitals. At each roundt, each hospital k samples a batch of samples from its local dataset, Dtk ⊂ Dk , where each local sample is chosen
independently and with probability q. Due to independent
sampling, the batch size is not fixed, and is a binomial random variable with parameters q and |Dk |. Let |Dtk | and η
denote the batch size and the learning rate, respectively. Let
C denote the maximum value of the L2 -norm of per-sample
gradients. If a per-sample gradient has a norm greater thanthis, then its norm is clipped to C (Abadi et al. 2016).
As hospitals do not trust the server, a potential solutionis, for each hospital, to add a large amount of noise to themodel updates, wkt , to keep them differentially private fromthe server. However, adding a large amount of noise has theundesirable effect of decreasing the accuracy of the trainedmodel. A better solution is to employ secure aggregation of
the model updates, which prevents the server from discovering the hospital’s model updates. Since the model updates
are now hidden from the server, the hospitals can add lessamount of noise to keep their model updates differentiallyprivate from the server.
Lemma 1. In Algorithm
1, if each hospital k adds

 Gaussian
2 ln (1.25/δ)C 2
2
noise nk ∼ N µ = 0, σ =to the aver2 |Dt |2 Kkt
age of (clipped)wG
is (, δ)-DP against thepgradients, then
server, and  K/K − 1, δ -DP against any hospital.

Proof. We compute the effect of the presence or the absencet−1tof a single sample at hospital k on wG. Let wkt −wG
be the(noiseless) model update of hospital k at round t. Due to secure aggregation, the server and other hospitals receive (unPK
t−1
t−1
t−1
1
t
t
encrypted) wG−wG=K
k=1 wk −wG . Since wGis already known to all, it is sufficient to only consider wkt .
tt
We define w0 k as the local model if D0 k is used by hospitaltk instead of Dtk , where Dtk and D0 k only differ in one sample. Since L2 -norm of each per-sample gradient is bounded
by C and the hospital k averages all per-sample gradients of2

Background materials are provided in Appendix, Section B.

Algorithm 1 Dopamine’s Training
1: Input: K: number of hospitals, D: distributed dataset,

w: model’s trainable parameters, L(·, ·): loss function,
q: sampling probability, σ: noise scale, C: gradient norm
bound, η: learning rate, β: momentum, T : number ofrounds, (, δ): bounds on record-level DP loss.
2: Output: wG : optimized global model.
0
3: wG= random initialization.
4: ˆ = 0
5: for t : 1, . . . , T do
6:
for k : 1, . . . , K do
7:
Dtk = Sampling(Dk ) // by uniformly samplingeach item in Dk independently with probability q.
8:for xi ∈ Dtk do
t−1
9:
gt (xi ) = ∇w L(wG
, xi )
ti )||2tt
10:ḡ (xi ) = g (xi )/ max 1, ||g (xC11:
end forP
σ 2 ·C 2 ·Itekt = |D1t |12:g)xi ∈Dt ḡ (xi ) + N (0,Kkk

ekt + βĝkt−1 //ĝk0 = 0ĝkt = gt−1
wkt = wG
− ηĝkt
end forˆ = CalculateP rivacyLoss(δ, q, σ, t) // by Moments Accountant (Abadi et al. 2016)17:
if ˆ >  thent−1
18:
return wG
19:
end ifP
1t=K
SecureAggregation( k wkt )
20:
wG
t
)
21:
Broadcast(wG
22: end for
13:
14:
15:
16:

|Dtk | samples, we havet

max wkt − w0 k

Dtk ,D0 tk

2= ηC/|Dtk |,for all k ∈ [K]. As adding or removing one sample at hospital k only changes the gradients of that hospital, we have
1
ηC
t
t
t
max
wG
− w0 G =max
wkt − w0 k = t
,
t
t
0t
0t
K Dk ,D k|Dk |K
2
2
Dk ,D ktt

for all k ∈ [K], where w0 G is the global model when D0 k
is used. Thus, to guarantee a record-level (, δ)-DP, the variance of effective noise added to each model update must be
2 ln(1.25/δ)η 2 C 2.
2 K 2 |Dtk |2
If hospitals use noise nk in Lemma 1, then the effectivet−1tnoise added to the global model update, wG
− wG
, is
P
2
K
ησ
η
k=1 nk , with a variance K . When the expression forK
σ 2 in Lemma 1 is put in place, the variance of the ef2 2C2fective noise is 2 ln(1.25/δ)η
= σeffective, which proves2 |Dtk |2 K 2(, δ)-DP against the server. From a hospital’s perspective,
since it knows its share in the effective noise, the variance of2 2
C (K−1)the noise after canceling its share is 2 ln(1.25/δ)η
,2 |Dtk |2 K 3pwhich results in ( K/(K − 1), δ)-DP against hospitals.
2
σeffective=

Single vs. Multiple Local Updates
An algorithm, similar to our Algorithm 1, is previously proposed by (Truex et al. 2019) (Section 5.2), where they allow each party to carry out multiple local SGD steps before
sharing the updated model with the server. We argue that
this approach has an important drawback: it may violate thecritical assumption in the moments accountant (MA) procedure (Abadi et al. 2016) for tracking the privacy loss ateach round, and for the same reason, it does not allow using momentum for SGD (line 13 in our Algorithm 1), whichhelps to improve the model’s accuracy. The MA is introduced by (Abadi et al. 2016) for keeping track of a boundon the moments of the privacy loss random variable, inthe sense of DP, that depends on the random noise added
to the algorithm’s output. In the proof provided by (Abadiet al. 2016), the MA is updated by sequentially applying
DP mechanism M. Particularly, (Abadi et al. 2016) models the system by letting the mechanism at round t, Mt ,to have access to the output of all the previous mechanisms Mt−1 , Mt−2 , ..., M1 , as the auxiliary input. Hence,
to properly calculate the privacy loss at round t, one needs
to make sure that the amount of privacy loss in the previous t − 1 rounds are properly bounded by adding the properamount of noise.
In Algorithm 1, we allow each hospital to add a Gaussian noise of variance σ 2 /K, at each round, and then sharethe updated model to securely perform FedSGD (McMahan
et al. 2017). Thus, before running any other computation on
the DNN, we add the required amount of effective noisethat is needed by MA to calculate the privacy loss  (seeLemma 1). Moreover, the effect of each record on the local
momentum is the same as its effect on the aggregated model,
thus, due to the post-processing property, using momentums
does not incur a privacy cost (see Appendix C). However,(Truex et al. 2019) allow the local model to be updated for
multiple (e.g. 100) local iterations while at each iteration the
DNN model that is used as the input is the output of the previous local iterations, and not the aggregated model. Thus,
the sequential property of the proof in MA may be violated.
Moreover, when running multiple local iterations, it is notclear how one can ensure a proper sampling similar to whatis needed in the central DPSGD.
Fixing these issues, our algorithm allows to take advantage of both privacy loss tracking with the MA and using
SGD with momentums. We emphasize that although our algorithm needs more communications due to sharing model
updates after every iteration, this cost can be tolerated considering the importance of the other two aspects in medical
data processing: model accuracy and privacy guarantee.

3

Evaluation

DatasetDiabetic retinopathy is an eye condition that can cause vision loss in diabetic patients. It is diagnosed by examining scans of the blood vessels of a patient’s retina thereby making this an image classification task. A DR
dataset (Choi et al. 2017), available at https://www.kaggle.
com/c/aptos2019-blindness-detection/data, exists to solve

this task. The problem is to classify the images of patients’retina into five categories: No DR, Mild DR, Moderate DR,
Severe DR, and Proliferative DR. The dataset consists of
2931 variable-sized images for training, and 731 imgaes fortesting. To deal with the variation in image dimensions, allimages were resized into the same dimensions of 224 × 224.

DNN Architecture
We use SqueezeNet (Iandola et al. 2016): a convolutional
neural network that has 50x fewer parameters than the famous AlexNet, but is shown to achieve the same level ofaccuracy on the ImageNet dataset. The size of the model is avery important factor for both training and inference, which
is one of the main motivations for using SqueezeNet. Moreover, SqueezeNet can achieve a classification accuracy of
about 80% on the DR dataset while the current best accuracy reported on the Kaggle’s competition3 on this dataset isabout 83% using a much larger DNN, EfficientNet-B3, thathas 18x more parameters than SqueezeNet4 .

Baselines
There are several approaches for training a DNN that provide us different trade-offs between utility, privacy, and complexity. We compare Dopamine against the following wellknown approaches:
Non-Private Centralized Training (C). Every hospitalshares their data with the server, such that a centralized
dataset D is available for training. In this method the problem is reduced to architecture search and optimization procedure for training a DNN on D. This method provides thebest utility, a moderate complexity, but no privacy. It results
in a single point of failure in the case of data breach. Moreover, in situations where patients are from different countries or private organizations, is almost impossible reachingan agreement on a single trusted curator.
Centralized Training with DP ( CDP). When a trusted
curator is available, one can trade some utility in themethod C to guarantee central DP for the patients byusing a differentially private training mechanism such as
DPSGD (Abadi et al. 2016). The complexity of CDP issimilar to C, except for the fact that existing DP mechanisms
make the training of DNNs slower and less accurate, as theyrequire performing additional tasks, such as per-sample gradient clipping and noise addition.
Non-Private FL ( F). Considering that a trusted curator is
not realistic in many scenarios due to legal, security, andbusiness constraints, hospitals can collaborate via FL, instead of directly sharing their patients’ sensitive data. In F,
every hospital has its own dataset that may differ in sizeand quality. We assume the server runs FedAvg (McMahan
et al. 2017). While F introduces some serious communication complexities, it removes the need for trusted curator
and enables an important layer of privacy protection. Basically, after sharing the data in the methods C and CDP,
34https://www.kaggle.com/c/aptos2019-blindness-detection/notebooks

The Pytorch implementation of is available at https://pytorch.org/docs/stable/
modules/torchvision/models/squeezenet.html

hospitals have no more control on the type and amount of
information that can be extracted from their patients’ data.
However, in F, hospitals do not share the original data, butsome aggregated statistics or the results of a computation
over the dataset. Even more, hospitals can decide to stop releasing information at any time, thus having more control
over the type and amount of information sharing. Although
it seems that the utility of F might be lower than that of C,
there are studies (Bonawitz et al. 2019) showing that witha more sophisticated algorithms, one can achieve the same
utility as C.
FL with Parallel DP (FPDP). Similarly to CDP, one canapply a DP mechanism to the local training procedure at thehospital side. Thus, every hospital locally provides a central DP guarantee. Notice that this is different form a local
DP guarantee (Dwork and Roth 2014), as each hospital isa trusted curator. In FPDP, a much better privacy can beprovided than the previous methods. However, as the size ofeach local dataset Di is much smaller than the size of thewhole dataset D, more noise should be added to achieve thesame level of privacy, which results in a utility loss.

Results
Dopamine proposes FL with a Central DP guarantee thatis similar to F, except the hospitals add noise to their gradients before running a secure aggregation procedure to aggregate the models at the server. Dopamine is similar to FPDP
in the sense that the hospitals add noise to their gradients,
but thanks to secure aggregation, the variance of the noise
hospitals add is less than FPDP by an order of K.
Figure 1 compares the classification accuracy (top plot)
and DP bound (bottom plot) of all the introduced methods for SqueezeNet trained on the DR dataset. We show
the mean and standard deviation of these 5 runs for eachmethod. To be fair, for each method we tuned the hyperparameters to achieve the best accuracy of that method. For
FL scenarios, we have divided the dataset equally among 10
hospitals (K = 10) in an independent and identically distributed (iid) manner, thus each hospital owns 293 images.
For F and FPDP we perform FedAvg (McMahan et al.
2017), where each hospital performs 5 local epochs in between each global epoch, and at each round half of the hospitals are randomly chosen to participate. The details of thehyper-parameters used are provided at https://github.com/
ipc-lab/private-ml-for-health/tree/main/private training/
We see that Dopamine’s bound is very close to that of
the centralized training counterpart CDP, and even better
than CDP at early epochs. Dopamine achieves a better classification accuracy than FPDP and more importantly muchbetter record-level privacy guarantee. In FPDP, each hospital needs to add noise with a variance K times larger to
get the same privacy level of CDP. In our experiments, wefound that this amount of noise makes learning impossible.
Finally, we see that F, compared to C, has a competitive
performance, but there still is a considerable accuracy gap
between F and Dopamine due to privacy protection provided. Such a gap can be mitigated if we could get accessFigure 1: Comparison of (top) classification accuracy (bigger better) vs. (bottom) DP bound (smaller better) on DR
dataset. Dopamine achieves a DP bound very close to thecentralized training counterpart, while it also achieves a better classification accuracy than FPDP. For all DP methods,
δ = 10−4 . For Dopamine, each epoch is equal to one round(see Figure 2 in Appendix for a more details)
to a larger dataset and proportionally reduce the amount ofnoise without weakening the privacy protection.

4

Conclusion and Future Work

We proposed Dopamine, a system for collaboratively training DNNs on private medical images distributed across several hospitals. Experiments on a benchmark dataset of DR
images show that Dopamine can outperform existing baselines by providing a better utility-privacy trade-off.
The important future directions are: (i) we aim to combine
the proposed secure aggregation functionality (discussed inAppendix D) to the Dopamine’s training pipeline. (ii) Current privacy-analysis only allows us to have one local iteration. We observed that allowing local hospitals to perform more than one iteration can improve the accuracy by
about 3%. We will carry out the privacy analysis for thiscase in the future. (iii) Existing DP libraries do not allow
using arbitrary DNN architectures. They are either not compatible with more recent architectures, like EfficientNet, orthey need too much resources. Making DP training faster isstill an open area of research and engineering. (iv) Finally,
we aim to evaluate Dopamine on other medical datasets, andinvestigate novel methods for accuracy improvement while
retaining the privacy protection provided.

References
Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H. B.;
Mironov, I.; Talwar, K.; and Zhang, L. 2016. Deep learningwith differential privacy. In Proceedings of the 2016 ACM
SIGSAC Conference on Computer and Communications Security, 308–318.
Bittau, A.; Erlingsson, U.; Maniatis, P.; Mironov, I.; Raghunathan, A.; Lie, D.; Rudominer, M.; Kode, U.; Tinnes, J.;
and Seefeld, B. 2017. Prochlo: Strong privacy for analytics in the crowd. In Proceedings of the 26th Symposium on
Operating Systems Principles, 441–459. ACM.
Bonawitz, K.; Eichner, H.; Grieskamp, W.; Huba, D.; Ingerman, A.; Ivanov, V.; Kiddon, C.; Konecny, J.; Mazzocchi,
S.; McMahan, H. B.; Van Overveldt, T.; Petrou, D.; Ramage, D.; and Roselander, J. 2019. Towards federated learningat scale: System design. In Proceedings of the 2nd SysML
Conference, Palo Alto, CA, USA.
Carlini, N.; Liu, C.; Erlingsson, Ú.; Kos, J.; and Song, D.
2019. The secret sharer: Evaluating and testing unintended
memorization in neural networks. In 28th USENIX Security
Symposium (USENIX Security 19), 267–284.
Choi, J. Y.; Yoo, T. K.; Seo, J. G.; Kwak, J.; Um, T. T.;
and Rim, T. H. 2017. Multi-categorical deep learning neuralnetwork to classify retinal images: A pilot study employing
small database. PLOS ONE 12: 1–16.
Choudhury, O.; Gkoulalas-Divanis, A.; Salonidis, T.; Sylla,
I.; Park, Y.; Hsu, G.; and Das, A. 2020. Anonymizing Data
for Privacy-Preserving Federated Learning. arXiv preprint
arXiv:2002.09096 .

Dwork, C.; Smith, A.; Steinke, T.; and Ullman, J. 2017. Exposed! a survey of attacks on private data .
Erlingsson, Ú.; Feldman, V.; Mironov, I.; Raghunathan,A.; Talwar, K.; and Thakurta, A. 2019.
Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, 2468–
2479. SIAM.
Erlingsson, Ú.; Pihur, V.; and Korolova, A. 2014. Rappor: Randomized aggregatable privacy-preserving ordinal
response. In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, 1054–
1067.
Esteva, A.; Kuprel, B.; Novoa, R. A.; Ko, J.; Swetter, S. M.;
Blau, H. M.; and Thrun, S. 2017. Dermatologist-level classification of skin cancer with deep neural networks. nature542(7639): 115–118.
Fan, J.; and Vercauteren, F. 2012. Somewhat Practical Fully
Homomorphic Encryption. Cryptology ePrint Archive, Report 2012/144.
Gao, D.; Ju, C.; Wei, X.; Liu, Y.; Chen, T.; and Yang, Q.
2019. HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalography. arXiv preprint
arXiv:1909.05784 .
Gentry, C. 2009. A Fully Homomorphic Encryption Scheme.
Ph.D. thesis, Stanford, CA, USA.

Cormode, G.; Jha, S.; Kulkarni, T.; Li, N.; Srivastava, D.;
and Wang, T. 2018. Privacy at scale: Local differential privacy in practice. In Proceedings of the 2018 International
Conference on Management of Data, 1655–1658.

Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M. C.; Wu, D.;
Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams,T.; Cuadros, J.; Kim, R.; Raman, R.; Nelson, P. C.; Mega,
J. L.; and Webster, D. R. 2016. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic
Retinopathy in Retinal Fundus Photographs. JAMA 316(22):2402–2410.

Corrigan-Gibbs, H.; and Boneh, D. 2017. Prio: Private, robust, and scalable computation of aggregate statistics. In
14th USENIX Symposium on Networked Systems Design and
Implementation (NSDI’17), 259–282.

Iandola, F. N.; Han, S.; Moskewicz, M. W.; Ashraf, K.;
Dally, W. J.; and Keutzer, K. 2016. SqueezeNet: AlexNetlevel accuracy with 50x fewer parameters and¡ 0.5 MBmodel size. arXiv preprint arXiv:1602.07360 .

Ding, B.; Kulkarni, J.; and Yekhanin, S. 2017. Collecting
telemetry data privately. In Advances in Neural Information
Processing Systems, 3571–3580.

Kaissis, G. A.; Makowski, M. R.; Rückert, D.; and Braren,
R. F. 2020. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence 1–7.

Doröz, Y.; Çetin, G. S.; and Sunar, B. 2016. On-the-fly Homomorphic Batching/Unbatching. In Financial Cryptography Workshops.
Dunnmon, J. A.; Yi, D.; Langlotz, C. P.; Ré, C.; Rubin,
D. L.; and Lungren, M. P. 2019. Assessment of convolutional neural networks for automated classification of chest
radiographs. Radiology 290(2): 537–544.
Dwork, C.; McSherry, F.; Nissim, K.; and Smith, A. 2006.
Calibrating noise to sensitivity in private data analysis. In
Theory of cryptography conference, 265–284. Springer.
Dwork, C.; and Roth, A. 2014. The algorithmic foundationsof differential privacy. Foundations and Trends in Theoretical Computer Science 9(3-4): 211–407.

Kifer, D.; Messing, S.; Roth, A.; Thakurta, A.; and Zhang,
D. 2020. Guidelines for Implementing and Auditing Differentially Private Systems. arXiv preprint arXiv:2002.04049.
Li, W.; Milletarı̀, F.; Xu, D.; Rieke, N.; Hancox, J.; Zhu, W.;
Baust, M.; Cheng, Y.; Ourselin, S.; Cardoso, M. J.; et al.
2019. Privacy-preserving federated brain tumour segmentation. In International Workshop on Machine Learning in
Medical Imaging, 133–141. Springer.
Li, X.; Gu, Y.; Dvornek, N.; Staib, L.; Ventola, P.; and Duncan, J. S. 2020. Multi-site fmri analysis using privacypreserving federated learning and domain adaptation: Abide
results. arXiv preprint arXiv:2001.05647 .

Lyubashevsky, V.; Peikert, C.; and Regev, O. 2013. On Ideal
Lattices and Learning with Errors over Rings. J. ACM 60(6).
ISSN 0004-5411.
McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; andy Arcas, B. A. 2017. Communication-efficient learning ofdeep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282. PMLR.
Pfohl, S. R.; Dai, A. M.; and Heller, K. 2019. Federated
and Differentially Private Learning for Electronic Health
Records. arXiv preprint arXiv:1911.05861 .
Qian, N. 1999. On the momentum term in gradient descentlearning algorithms. Neural networks 12(1): 145–151.
Regev, O. 2009. On Lattices, Learning with Errors, RandomLinear Codes, and Cryptography. J. ACM 56(6). ISSN 00045411.
SEAL. 2020. Microsoft SEAL (release 3.5). https://github.
com/Microsoft/SEAL. Microsoft Research, Redmond, WA.
Sheller, M. J.; Reina, G. A.; Edwards, B.; Martin, J.; andBakas, S. 2018. Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain
tumor segmentation. In International MICCAI Brainlesion
Workshop, 92–104. Springer.
Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017.
Membership inference attacks against machine learningmodels. In 2017 IEEE Symposium on Security and Privacy(SP), 3–18. IEEE.
Titus, A. J.; Kishore, S.; Stavish, T.; Rogers, S. M.; and Ni,
K. 2018. PySEAL: A Python wrapper implementation of the
SEAL homomorphic encryption library.
Truex, S.; Baracaldo, N.; Anwar, A.; Steinke, T.; Ludwig,
H.; Zhang, R.; and Zhou, Y. 2019. A Hybrid Approach to
Privacy-Preserving Federated Learning. In Proceedings ofthe 12th ACM Workshop on Artificial Intelligence and Security, AISec’19, 1–11. New York, NY, USA: Association for
Computing Machinery. ISBN 9781450368339.
Wang, T.; Blocki, J.; Li, N.; and Jha, S. 2017. Locally differentially private protocols for frequency estimation. In 26th
USENIX Security Symposium (USENIX Security 17), 729–
745.

A

Related Work

(Sheller et al. 2018) apply FL, without any privacy guarantee, for the segmentation of brain tumor images using a
deep convolutional neural network, known as U-Net. They
compare FL with a baseline collaborative learning scenario
where hospitals iteratively train a shared model in turn, andshow that FL outperforms this baseline.
(Pfohl, Dai, and Heller 2019) employ FL with FedAvg
and DPSGD algorithms to train logistic regression and feedforward networks with one hidden layer for in-hospital mortality and prolonged length of stay prediction using the patients electronic health records. Comparing FL with a centralized approach, when DP is used in both cases, their experimental results show that, while centralized training canachieve a good DP bound with  ≈ 1, FL with DP performs
poorly in terms of both accuracy and .
Considering the differences in type of the device, numberof sensors, and the placement of the sensors that is used ineach hospital for collecting electroencephalography (EEG)
data, (Gao et al. 2019) propose an application of FL for training a classifier over heterogeneous data; however, they do
not consider any formal privacy guarantee. (Li et al. 2020)
use FL among hospitals for training an fMRI image classifier
and add some noise to the trained parameters before sharing
them with the server which does not follow the requirementof a valid DP mechanism, thus it does not provide a formal
privacy guarantee.
(Choudhury et al. 2020) offer a k-anonymity based privacy guarantee to generate a syntactic tabular dataset, instead of the original dataset, and use it for participating in
FL. (Li et al. 2019) propose an FL system for brain tumour segmentation in a method where each hospital owns
a dataset of MRI scans. However, their implementation does
not satisfy record-level DP, but parameter-level DP, which
for neural networks is not a meaningful privacy protection.

B

Background

Differential Privacy (DP)When we want to publish the results of a computation F(D),over a private dataset D, DP helps us to bound the ability ofan adversary in distinguishing whether any specific sample
data was included in the dataset D or not. DP has a worstcase threat model, assuming that all-but-one can collude andadversaries can have access to any source of side-channel
information.
Definition 1 (Differential Privacy). Given , δ ≥ 0, a
mechanism (i.e. algorithm) M satisfies (, δ)−differential
privacy if for all pairs of neighboring datasets D and D0 differing in only one sample, and for all Y ⊂ Range(M), wehavePr M(D) ∈ Y ≤ e Pr M(D0 ) ∈ Y + δ,(1)
where  is the parameter that specifies the privacy loss (i.e.
e ), δ is the probability of failure in assuring the upperbound on the privacy loss, and the probability distribution
is over the internal randomness of the mechanism M, holding the dataset fixed (Dwork et al. 2006; Dwork and Roth2014). When δ = 0, it is called pure DP, which is stronger
but less flexible in terms of the mechanism design.
For approximating a deterministic function F, such as themean or sum, a typical example of M in Equation (1) isa zero-mean Laplacian or Gaussian noise addition mechanism (Dwork and Roth 2014), where the variance is chosen
according to the sensitivity of F that is the maximum of theabsolute distance |F(D) − F(D0 )| among all pairs of neighboring datasets D and D0 (Dwork et al. 2006). In general, tobe able to calculate the sensitivity of F, we need to know,or to bound, the Range(F); otherwise we cannot design areliable DP mechanism.
Definition 2 (Gaussian Mechanism). Gaussian mechanism adds a Gaussian noise to a function F(x) so that theresponse to be released is (, δ)-differentially private. It isdefined as
2 ln (1.25/δ)(∆2 F)2 ,
M(x) = F(x) + N µ = 0, σ 2 =2where ∆2 F is the L2 sensitivity of F(x) to the neighboringdatasets (Dwork and Roth 2014).
DP mechanisms are in two types: central DP and local
DP. In central DP, we consider a model where there is atrusted server, thus the patients original data is shared withthat server. The trusted server then run M on the collected
data before sharing the result with other parties. In local DP,
every patient randomly transforms data at the patient’s side
without needing to trust the server or other parties (Erlingsson, Pihur, and Korolova 2014; Wang et al. 2017; Erlingsson et al. 2019). Thus, in terms of privacy guarantee, local DP ensures that the probability of discovering the true
value of the user’s shared data is limited to a mathematically defined upper-bound. However, central DP gives suchan upper-bound for the probability of discovering whether a
specific user has shared their data or not.
DP mechanisms are mostly used for computing aggregated statistics, such as the sum, mode, mean, most frequent,or histogram of the dataset (Bittau et al. 2017; Ding, Kulkarni, and Yekhanin 2017; Cormode et al. 2018). DP mechanisms can also be used for training machine learning models
on sensitive datasets while providing DP guarantees for thepeople that are included in the training dataset (Abadi et al.
2016; Bonawitz et al. 2019; Kifer et al. 2020).

Federated Learning (FL)
Many parameterized machine learning problems can bemodeled as a stochastic optimization problem
min := ED∼D L(w, D),

w∈Rd

(2)

where w denotes the model parameters, D is a random
dataset sampled from the true, but unknown, data distribution D, and L is a task-specific empirical loss function.
The core idea of the FL is to solve the stochastic optimization problem in (2) in a distributed manner such thatthe participating users do not share their local dataset witheach other but the model parameters to seek a consensus.
Accordingly, the objective of K users that collaborate tosolve Equation (2) is reformulated as the sum of K loss
functions defined over different sample datasets, where eachcorresponds to a different user:K

min L(w) =w

1 X
EDk ∼Dk L(w, Dk ),
K

(3)

k=1

where Dk denotes the portion of the dataset allocated to userk. SGD is a common approach to solve machine learning
problems that are represented in the form of Equation (2).
FL considers the problem in Equation (3), and aims to solve
it using SGD in a decentralized manner.

Homomorphic Encryption (HE)
Fully Homomorphic Encryption (FHE) allows computationof arbitrary functions F on encrypted data, thus enablinga plethora of applications like private computation offloading. Gentry (Gentry 2009) was the first to show that FHE
is possible with a method that includes the following steps:(i) construct a somewhat homomorphic encryption (SHE)
scheme that can evaluate functions of low degree, (ii) simplify the decryption circuit, and (iii) evaluate the decryption
circuit on the encrypted ciphertexts homomorphically to obtain new ciphertexts with a reduced inherent noise. The third
step is called bootstrapping, and this allows computations offunctions of arbitrary degree. Here, the degree of a functionis the number of operations that must be performed in sequence to compute the function. For example, to computex2 , one multiplication is necessary, therefore its degree is 1,
while to compute x3 , 2 multiplications (x ∗ x and x ∗ x2 ) arenecessary, and therefore its degree is 2.
The security of homomorphic encryption schemes isbased on the learning with errors problem (Regev 2009),
or its ring variant called ring learning with errors (Lyubashevsky, Peikert, and Regev 2013), the hardness of which
can be shown to be equivalent to that of classical hard problems on ideal lattices, which are the basis for a lot of postquantum cryptography. All FHE schemes add a small noise
component during encryption of a message with a public
key, which makes the decryption very hard unless one hasaccess to a secret key. The decrypted message obtained consists of the message corrupted by a small additive noise,
which can be removed if the noise is “small enough”. Each
computation on ciphertexts increases the noise, which ultimately grows large enough to fail the decryption. The bootstrapping approach is used to lower the noise in the ciphertext to a fixed level. Addition of ciphertexts increases thenoise level very slowly, while multiplication of ciphertexts
increases the noise very fast. When FHE is only used for aggregation, the bootstrapping step, which is a computationally expensive procedure, is not usually necessary since thenoise never grows to a large enough size.
While machine learning models compute on floating point
parameters, encryption schemes work only on fixed point
parameters, or equivalently, integer parameters, which canbe obtained by appropriately scaling and rounding fixed
point values. Given an integer u to encode an integer baseb, a base-b expansion of u is computed, and represented as apolynomial with the same coefficients. The expansion usesa ‘balanced’ representation of integers modulo b as the coefficients, that is, hwhen b is odd, ithe coefficients are chosen tolie in the range −(b−1), (b−1), and when b is even, the co2
2
h
i
(b−1)
efficients are chosen to lie in the range −b,, except22
if b = 2, when the coefficients are chosen to lie in the set{0, 1}. For example, if b = 2, the integer 26 = 24 +23 +21 is
encoded as the polynomial x4 +x3 +x. If b = 3, 26 = 33 −30
is encoded as the polynomial x3 − 1. Decoding is done bycomputing the polynomial representation at x = b. To map
an integer to a polynomial ring R = Z[x]/f (x), where f (x)is a monic polynomial of degree d, the polynomial represen-tation of the integer is viewed as being a plaintext polynomial in R. The base b is called the plaintext modulus. The
most popular choice of f (x) is xd + 1, with d = 2n called
the polynomial modulus. Let q > 1 be a positive integer,then we denote the set of integers [− 2q , 2q ] by Zq , and theset of polynomials in R with coefficients in Zq by Rq . The
integer q is called the coefficient modulus, and is usually set
much larger than the plaintext modulus.
For multi-dimensional vectors, the single-instructionmultiple-data technique, also known as batching (Doröz,
Çetin, and Sunar 2016), encodes a given array of multiple
integers into a single plaintext polynomial in Rq . A single
computation instruction on such a plaintext polynomial isequivalent to simultaneously executing that instruction on
all the integers in that array, thus speeding up the computation by many orders of magnitude. The plaintext modulus isassumed to be larger than any integer in the given array. The
length of the array is assumed to be equal to the degree of thepolynomial modulus d, and in case that its size is less thand, then it is padded with zeros to make its length equal to d.
Lagrange interpolation is performed over the array of integers to obtain a polynomial of degree d − 1, thus obtaining
the plaintext polynomial in Rq encoding d integers.

C

On the Effect of Momentums

Consider the momentum term (Algorithm 1, line 13) atround t. For user k, we haveĝkt = g̃kt + βĝkt−1 =t−1X

β i g̃kt−i ,

(4)

i=0

where g̃kt ’s are the noisy gradients (Algorithm 1, line 12).
After every round, ĝkt ’s are securely averaged at theserver, such that the averaged gradient at round t becomes
K
K t−1
t−1
K
1 X t X X i t−i
1 X i X t−i
ĝ =
ĝk =
β g̃k =
β
g̃k .
K
K i=0
k=1
k=1 i=0k=1(5)PK
Since the term k=1 g̃kt−i is computed securely, and thenoise of each g̃kt−i adds up, as it is shown in Lemma 1,
t−i∀i ∈ {0,is (, δ)-DP against the server
p 1, . . . , t − 1}, g̃and ( K/(K − 1), δ)-DP against other hospitals, whereˆ ≤  is the DP-bound at round i.
Thanks to the post-processing property (Dwork and Roth2014) of DP, use of the same noisy gradients repetitively
does not incur any additional privacy costs. Thus, use of momentum in our setting does not incur additional privacy costeither.
t

D

Secure Aggregation

Secure aggregation helps in providing a computational DP
guarantee, even during the training procedure, so that thehospitals do not need to trust the server. We use theBrakerski-Fan-Vercauteren (BFV) scheme (Fan and Vercauteren 2012) for homomorphic encryption. For security
parameter λ, random element s ∈ Rq , ∆ = b qb c, and a discrete probability distribution χ = χ(λ), the BFV scheme isimplemented as follows:• SecretKeyGen (1λ ): sample s ← χ, and output sk = s.
• PublicKeyGen (sk): sample a ← Rq , e ← χ, and output
pk = [−(as + e)]q , a .
(6)• Encrypt(pk, m): To encrypt a message m ∈ Rb , let
p0 = pk[0] and p1 = pk[1], and sample u, e1 , e2 ← χ,and output
h
i h
i 
ct = p0 u + e1 + ∆.m , p1 u + e2 . (7)qq• Decrypt(sk, ct): Set s = sk, c0 = ct[0], and c1 = ct[1],and computehh b.[c + c s] ii
0
1 q
.
(8)
q
b
We use the open source PySEAL library (Titus et al.
2018), which provides a python wrapper API for the original SEAL library published by Microsoft (SEAL). We simulate the FL environment over the message passing interface (MPI), with the rank 0 process modeling the server, andthe rank 1, . . . , K processes model the hospitals that participate in the training. The plaintext modulus b is chosen to
be 40961, while the polynomial modulus d is chosen to be
x4096 + 1. Therefore, each ciphertext can pack d = 4096
integers with magnitude less than 40961 for SIMD operations, thus speeding up the communication and computation
on encrypted data by 4096 times. The methodology is as follows:
1. Key generation and distribution: A public key and secretkey are generated by the rank 1 process (hospital 1). The
public key is broadcasted to all the involved parties, while
the secret key is shared with the hospitals but not theserver.
2. The server initializes the model to be trained, and sends
the model to the hospitals in the first round.
3. Each hospital performs an iteration on its local model
based on Algorithm 1, at the end of which each hospital
has the updated model.
4. Each active hospital:• flattens the tensor of model parameters into a 1D arrayw,• converts the floating point values in the array to 3 digit
integers by multiplying each value by 103 , and thenrounding it to the nearest integer,• partitions the array w into chunks of d = 4096 parameters, denoted by w1 , . . . , wl , where l = d length(w)4096 e,• encrypts the chunks w1 , . . . , wl with the public key,
into ciphertexts ct1 , . . . , ctl , respectively, using thebatching technique, and sends the list of ciphertexts tothe server.
5. The server adds the received encrypted updates and sends
the encrypted aggregated model to the hospitals for the
next round.

Figure 2: The top plot in Figure 1, but starting from epoch 1.
6. The hospitals decrypt the received aggregate array w,
scale and convert each integer value to a floating pointvalue of correct precision, divide each value by the number of hospitals that were active in the previous round,and finally reshape w back to the structure of the originalmodel.
7. Go to step 3.
At the end of the above procedure, the hospitals send the final unencrypted updated models to the server, which aggregates them and updates the global model for the final time toobtain the final trained model. Notice that we assume hospitals are non-malicious, otherwise one needs to use othertechniques, e.g. (Corrigan-Gibbs and Boneh 2017). However, methods such as (Corrigan-Gibbs and Boneh 2017) use
secure multi-party computation, and not homomorphic encryption. The method used in (Corrigan-Gibbs and Boneh2017) requires multiple non-colluding aggregators (at least
one honest aggregator). That is different from our setting,where we consider only one global aggregator. Another alternative is for the hospitals to send secret shares to the otherhospitals acting as aggregators. However, that would require
the hospitals to be able to communicate with each other,which is not the best solution in dynamic settings.