Generative Models for Effective ML on Private, Decentralized Datasets
Authors: Sean Augenstein
H. Brendan McMahan
Daniel Ramage
Swaroop Ramaswamy
Peter Kairouz
Mingqing Chen
Rajiv Mathews
Blaise Aguera y Arcas
Venue:  (2020)
Keywords: generative models
federated learning
decentralized learning
differential privacy
privacy
security
GAN

Abstract: To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data—of representative samples, of outliers, of misclassifications—is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses,
and c) assigning or refining human-provided labels. However, manual data inspection is risky for privacy-sensitive datasets, such as those representing the behavior of real-world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where raw examples are stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models—trained using federated methods and with formal differential privacy guarantees—can be used effectively to debug data issues even
when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using a novel algorithm for differentially private federated GANs.

Published as conference paper ICLR 2020
GENERATIVE MODELS FOR EFFECTIVE ML ON PRIVATE, DECENTRALIZED DATASETS
Sean Augenstein
H. Brendan McMahan
Daniel Ramage Google Inc.
Google Inc.
Google Inc. mcmahan
saugenst wowpnnstttoogll..o.oooooo com
google. mcmahan@google.cco
dramage@googlee.com
Swaroop Ramaswamy
Peter Kairouz
Mingqing Chen
Google Inc. swarogpram@ggoggee.com com
Google Inc. kairouz@google..o com
Google Inc.
mingeingggooogle.comm com
Rajiv Mathews Google Inc. ma mathews@google..om com
Blaise Aguera y Arcas
Google Inc.
blaisea@google.com com
ABSTRACT
To improve real-world applications machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data-of representative samples, of outliers, of misclasiificatios- an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-pre proviiddd labels. However, manual data inspection is problematic for privacy-sensitivg datasets, such those representing the behavior of real- world individuals. Furthermore, manual data inspection is impossible in the increasingly important setting of federated learning, where examples stored at the edge and the modeler may only access aggregated outputs such as metrics or model parameters. This paper demonstrates that generative models-trainge formal differential privacy guarantees- can be used effectively to debug many commonly occurring data issues even when the data cannot be directly inspected. We explore these methods in applications to text with differentially private federated RNNs and to images using novel algorithm for differentially private federated GANs.
using federated methods and with
INTRODUCTION
Real-world systems increasingly depend on machine learning (ML) to make decisions, detect anomalies, and power products. Applications ranging from fraud detection to mobile phone keyboards use models trained on data that may be privacy sensitive. The data may contain financial, medical, or behavioral informa- tion about individuals. Institutions responsible for these applications of ML must balance data stewardship minimizing the risks data loss, theft, or abuse-wil with the practical needs of the "modeler" whose job develop and improve the machine learned models.
Modelers working with privacy-sensitive data face significant challenges in model development and debug- ging. Often, modeler's first step would be to inspect individual examples in order to discover bugs, generate hypotheses, improve labeling, or similar. But direct inspection data may be audited or disallowed by pol- icy in some settings. In other settings settings-includingg including federated learning (FL) (MeMahan & Ramage 2017]

---

Published as a conference paper ICLR 2020
McMahan al. 2017), which is the motivation and focus of this work-the data cannot be inspected. In FL., raw data remains distributed across a fleet of devices, such as mobile phones, while an orchestrating server coordinates training of a shared global model. Only the final model parameters and statistics are gathered and made available to the modeler.
How can modeler effectively debug when training data privacy sensitive or decentralized? This paper demonstrates that the novel application of auxiliary models, namely privacy-preserving generative models, can stand in for direct data examination during the process of debugging data errors during inference or training. By combining ideas from deep generative models, FL, and user-level differential privacy (DP), we show how some needs traditionally met with data inspection can instead be met by generating synthetic federated generative model. These examples could be representative of
examples from a all or subset of the ion-inspece data, while at the same time preserving the privacy of individuals. Our contributions include:
Identifying key challenges in implementing end-to-end workflows with non-inspectable data, e.g.. for debugging 'primary' ML model used mobile application.
Proposing methodology that allows (sufficiently powerful) 'auxiliary' generative models to solve these challenges.
Demonstrating how privacy preserving federated generative models RNNs for text and GANs for images can be trained to high enough fidelity to discover introduced data errors matching those encountered in real world scenarios. This requires a novel adaption of generative adversarial networks (GANs) to the federated setting with user-level DP guarantees.
2 CHALLENGES OF ML ON NON-INSPECTABLE DATA
A key aspect of deep learning is the utilization of large datasets. The behavior of a deep network is tightly coupled the data trains on. As (Scullyy et al. 2015) puts it, "MI. is required in exactly those cases when the desired behavior cannot be effectively expressed in software logic without dependency on external data." The centrality of data to ML. performance recognized the attention paid to data analysis, curation, and debugging. Textbooks devote chapters to methodologies for "Determining Whether to Gather More Data" (Goodtellow et al., 2016)) The literature conveys practical lessons learned on ML system anti-patterns that increase the chance of data processing bugs (Sculley et al., 2015). In general, an assumption of unfettered access to training or inference data is made (Riley] [016]] Zinkevich] 2017) Chakarov et al.) 2016). What to do if direct access is precluded, e.g., if working with private and decentralized data, largely unaddressed.
Below, we describe six common tasks (11-66, summarized in Table T where a modeler would typically use direct data access. Our choice of these tasks is validated by Humbatova al. (2019), a recent survey provid- ing a taxonomy of faults in deep learning systems. Two of the largest classes of faults are in 'Preprocessing of Training Data" and 'Training Data Quality"; T1-T6 are manners of addressing these faults.
Sanity checking and model debugging (T1-T4) Experienced modelers will often inspect some random examples and observe their properties before training model (T1); if this step skipped, it is often a first step in debugging. Are the size, data types, and value ranges as expected? For text data, are words or word pieces being properly separated and tokenized? A modeler is often working to identify mistakes in upstream processing, affecting all or most data, and readily apparent by looking at one or few input samples.
When a model misbehaving, looking at a particular subset of the input data natural. For example, in a classification task, modeler might inspect misclassified examples to look for issues in the features or
Despite these limitations, FL systems (Bonawitz et al. 2019 have proven effective for training and deploying real-world models, for example see Hard al. (2018) Yang et al. (2018: Chen al. (2019).
2

---

Published as a conference paper ICLR 2020
ML modeler tasks typically accomplished via data inspection. In Section 2.1 we observe that selection criteria can be applied programmatically to train generative models able to address these tasks.
Table 1:
Task
Selection criteria for data to inspect
T1
Sanity checking data
Random training examples
T2
Debugging mistakes
Misclassified examples (by the primary classifier)
T3
Debugging unknown labels/classes, e.g. out-of-vocabulary words
Examples of the unknown labels/classes
T4 Debugging poor performance on certain classes/Slices/users
Examples from the low-accuracy classes/slices/users
T5
Human labeling of examples Detecting bias in the training data
Unlabeled examples from the training distribution
T6
Examples with high density in the serving distribution but low density in the training distribution.
labels (T2). For tasks where the full of possible labels is too large, e.g. a language model with fixed vocabulary, the modeler might examine a sample of out-of-vocabulary words (T3). For production ML applications, is often important monitor not only overall accuracy, but accuracy over finer grained slices of the data for example, by country, by class label, by time of day, etc 2 If low accuracy is observed on such a slice, is natural examine examples selected from that segment of the data (T4). For example, if training on data which can be grouped by user, it is natural (but potentially problematic from privacy perspective) to look data from users with low overall accuracy.
Data labeling (T5) Supervised learning problems require labeled data. Often in FL., labels can be inferred naturally from context on device-for example, the next-word predictor in a keyboard app (e.g., as in Hard et al. (2018) is problem where the input sequence of words and the label is the subsequent word. But some supervised learning problems are better suited to human raters inspecting and manually assigning class labels (e.g., visual object recognition datasets as Deng al. (2009)). Because FL systems do not allow users' data to be sent to the cloud for labeling, it would be desirable to synthesize realistic, representative examples from the decentralized data that could be labeled by human raters. Doing so would require private mechanism of generating high-fidelin fidelity training examples. This would expand the set of ML. problems that can be afforded FL's privacy advantages.
Detecting bias in training data (T6) Another common use of human raters in supervised learning to assign labels to ppriaccy-enssitii training examples curated from a public dataset or collected via donation. These examples might be the only ones a classifier is trained on, even if a much larger unlabeled dataset exists (e.g. on phones in FL). The distributional difference between the datasets is a source of bias. For example, a labeled training dataset for a face detector might not be representative of the diversity of racial/ethnic/genderla
classes that the face detector sees through a phone's 's camera (i.e., at serving time). These classes can be unrelated to the model's task but serve as a confounding factor in prediction the model has erroneously fit an assumption about them during training (c.g.. smiling face detector that judges smiling vs. non-smiling based on gender or gender expression (Denton et al.) 2019)). When entire groups are un- be biased predictions that induce quite insidious social effects
underrepresented in training, the result (Angwin et al.) 2016). Modelers' goals of building unbiased classifiers could be met by inspecting examples of classes missing at training but present at prediction time. These examples-eve even low fidelity- indicate whether a model is fair and point to where additional training data collection is required.
'E.g.. g.. McMahan et al. 2013 Sec. 5.2) notes 100s of slices of relevance for ad-click-through-et prediction.
3

---

Published as a conference paper at ICLR 2020
2.1 USING GENERATIVE MODELS INSTEAD OF DATA INSPECTION
The general methodology for using generative models in place of data inspection as follows: in a situation where the modeler would otherwise inspect examples based on particular criteria (as shown in Table [.. this criteria is expressed as programmatic data selection procedure used to construct a training dataset for a generative model. In the case of FL specifically. this might involve both selecting only subset of devices to train the model, as well as filtering the local dataset held on each device.
We conduct experiments for three such tasks in this paper. Section 5 considers T3, and shows how the methodology proposed here can be extended when the primary model is itself a generative model. We con- duct experiments for T2 and T4 in Section where both approaches provide alternative ways of discovering the same bug; results for the T4 approach are presented in the main body, with T2 results appearing in the Appendix. We chose these scenarios being representative of large class of data-related ML bugs, based on real-world experience in both industry and academic settings. Independently, Humbatova et al. (2019) identifies 'text segmentation" and 'pixel encoding' as being very common sources of bugs. By showing that privacy-preserving federated generative models can debug these scenarios, we demonstrate the applicability of the approach. One of the reasons we select these scenarios is that high-fidelity generation not necessary to detect these bugs, and pushing the state-of-the-art in generative models is not the aim of this work.
contrast, for some classes of bugs, as well for generating examples used for labeling and training (T5), high-fidelity private generative models are necessary. We hope that this work will serve as a call-to-action for the generative modeling community, as our work strongly suggests that these models have a fundamental ML workflows.
role to play in enabling the widespread use of
Task T6 requires a more subtle approach in defining the selection criteria. We briefly describe possible approaches in the spirit of providing a more complete description of the uses of generative models in private ML workflows, but leave details and experiments as promising directions for future work. Suppose we have a public, centralized distribution Ppubicc as well as a private, decentralized distribution Ppevate- Informally,
we might wish to generate examples according to x ~ Punnse()))
Poais or perhaps x ~ Pprivace (# PPpulli (x) S e).
If we have family of generative models for the domain of x that explicitly provide likelihood (e.g.. the language models we consider in Section 5), then we can first train such a generative model on Ppablic and use this to filter or re-weight the decentralized samples drawn from Prrivate train second generative model which will be used to generate samples representative of P Privare but not publlic In the case of architectures that do not provide an explicit likelihood (c.g.. GANs), alternative techiques are needed, for example a secondary discriminator used to differentiate samples from P Private from those from Puublic.
Of course, if any of the generative models used here simply reproduce the private examples of users, nothing has been gained. Hence, we next turn to training these models with strong privacy guarantees.
3 DIFFERENTIALLY PRIVATE FEDERATED GENERATIVE MODELS
To implement the approach of Section we propose combining three technologies: generative models, federated learning (FL), and differential privacy (DP). Deep generative models can synthesize novel exam- ples, FL can train and evaluate against distributed data, and FL and DP both afford user privacy protections.
Critically, none of the challenges presented require the inspection of any particular user's data. As with MI,, the goal is to discover something broadly meaningful. Thus, it is natural to consider the use of suitable syn- thetic examples in licu of real user data. For this, we can leverage generative models based on deep neural networks (or 'deep generative models'). In contrast to discriminative models, generative models learn joint distribution p(æ, y) and can be applied to data synthesis. Such neural networks can either approximate a like- lihood function or serve as mechanism for drawing samples (i.e., an implicit distribution). Deep generative
4

---

Published as a conference paper ICLR 2020
Goodfellow al. 2014 Kumar et al. 2019 Radford et al. 2019) have garnered a considerable amount of research interest, particularly for high dimensional spaces where explicit modeling intractable. Application domains include text, audio, and imagery.
models of various forms (Kingma & Welling 2013
To work with decentralized data, we train these generative models via FL, ensuring raw user data never leaves the edge device. Instead, a randomly chosen subset of devices download the current model, and each locally computes a model update based on their own data. Ephemeral model updates are then sent back to the coordinating server, where they are aggregated and used to update the global model. This process repeats over many rounds until the model converges (for more information, see McMahan et al. (2017) and Bonawitz jtt al. (2019). FL is borne of the mobile domain, where privacy is paramount and data is decentralized. Privacy for the mobile domain is major motivator for this work, and we'll use FL extensively in this paper.
As with other ML models, deep generative models have a tendency to memorize unique training examples, leading to valid concern that they may leak personal information. Differential privacy is powerful tool for preventing such memorization. In particular, in this paper we'll emphasize the use of user-level DP, obtained in the FL context via combination of per-user update clipping and post-aggregation Gaussian noising, following McMahan et al.] (2018); Appendix [ reviews user-level DP and FL. We assume this mechanism is implemented by the FL infrastructure, before the modeler has access the trained models. This approach bounds the privacy loss to both the modeler and the devices that participate in FL rounds. In our experiments, we quantify the privacy obtained using (c. 8) upper bounds.
4 RELATED WORK
Prior work has looked training GANs with DP in centralized setting (Torkzadehmahani et al. 2019 Xie
let al. 2018 Zhang et al. 2018 Frigerio et al. 2019 Beaulieu-Jones et al. 2018 Esteban et al.) 2017). The focus is how privacy protections like DP can be leveraged to quantifiably guarantee that GAN will not synthesis examples too similar to real data. However, this work does not apply to training on decentralized data with FL, nor does it consider user-level DP guarantees, both of which are critical for our applications.
Other work studies generative models for decentralized data problems, but without privacy protections, An example is training GANs (Hardy et al. 2018) on decentralized data across multiple data centers. We con- sider this conceptually distant from the topic of this paper, where user- privacy protection is paramount.
Two recent papers focus on what could be considered as specific instances of generative FL-based tasks, without connecting those instances to a much larger class of workflows on decentralized data in need of a general solution (i.e., all of T1-T6 from Table First, the FedGP algorithm Triasttnyn & Faltings| (2019) is an approach to building a federated data synthesis engine based on GANs, to address specific task re- sembling T5. Where we use DP as our means of measuring and protecting against data release (providing a worst-case bound on privacy loss), FedGP adopts a weaker, empirical measure of privacy (specifically, dif- ferential average-case privacy) which has not yet been broadly accepted by the privacy community. Second, recent language modeling work in Chen et al. (2019) uses character-level RNN trained via FL to perform the T3 task of generating common out-of-vocabulary (OOV) words (i.e., words that fall outside a finite vo- cabulary set). OOVs are interesting the language modeler as they reflect natural temporal changes in word usage frequency (c.g.. new slang terms, names in the news, etc.) that should be tracked. Their work does not apply DP. We adopt this OOV model for debugging purposes (and add DP) in experiments in Section
5 AN APPLICATION TO DEBUGGING DURING TRAINING WITH RNNS
DP Federated RNNs for Generating Natural Language Data Recurrent Neural Networks (RNNs) a ubiquitous form of deep network, used to learn sequential content (e.g., language modeling). An interesting property of RNNs is that they embody both discriminative and generative behaviors in single model. With
5

---

Published as conference paper ICLR 2020
Table 2: Top generated OOV words by joint character probability (computed using Equation [ with and without the bug. Number accompanying is joint probability. The model trained with case-insensitive tokenization.
Without bug With bug
generated are
Without bug (prob 10
3)
With 10% bug (prob 10 3)
regex
5.50
i have
8.08
isfiddle
3.90
am
5.60
xcode
3.12
regex
4,45
divs
2.75
you can
3.71
stackoverflow
2.75
if you
2.81
listview
2.74
this is
2.77
textbox
2.73
here is 2.73
Position in sentence
foreach
2.34
jsfiddle 2.70
asyne
2.27
want 2.39
Figure 1: Percentage of samples generated from the word-LM that are OOV by posi- tion in the sentence, with and without bug.
iis
2.21
textbox 2.28
regards to the former, an RNN learns a conditional distribution p(1++1 Ti..... Zo; D). Given a sequence of tokens -0,--...2 Ty and their successor 1i+1, the RNN reports the probability of xi+1 conditioned on its predecessors (based on observing dataset D). Typically, zp special 'beginning-of-seeee of-ssuuenceeee token.
But an RNN is also generative model: the successor token probability distribution can be sampled from to generate a next token, and this process can repeated to generate additional tokens the sequence. The repeated application of the RNN provides sample from the following joint distribution:
plEn: ................................ 32, 21, D) II p(z Fi.... roz D) D) i=0
(I)
In this way, RNN word language models (word-LMs) can generate sentences and RNN character language models (char-LMs) can generate words. Training an RNN under FL and DP can be performed via the DP-FedAvg algorithm McMahan al. (2018), restated Appendix A Algorithm
Experiment Consider mobile keyboard app that based on previous words. The app takes as input raw text, performs preprocessing (c.g. tokenization and normalization), and then feeds the processed list of words as imput to the word-LM. The word-LM has fixed vocabulary of words V. Any words in the input outside of this vocabulary are marked as 'out-of-vocabulary (OOV). Such word-LMs have been trained on-device via FL. (Hard et al., 2018). We demonstrate how we can detect bug in the training pipeline for the word-LM using DP federated RNNs.
word-LM offer next-word predictions to
Suppose bug introduced in tokenization which incorrectly concatenates the first two tokens in some sen- tences into one token. E.g., 'Good day to you. is tokenized into list of words as ('Good day' 'to',... instead of [ 'God' 'day', "to" ..... J. Such issues are not uncommon in production natural language processing (NLP) systems. Because these concatenated tokens don't 't match words in vocabulary V. they will marked as OOVs. Under normal circumstances OOVs occur at relatively consistent rate, this bug will be noticed via spike in OOV frequency, metric NLP modeler would typically track (see Appendix B). Were this the cloud, some of the O0V tokens could be inspected, and the erroneous concatenation realized. But here the dataset is private and decentralized, inspection is precluded.
6

---

Published as a conference paper at ICLR 2020
We simulate the scenario using a dataset derived from Stack Overflow questions and answers (hosted by TensorFlow Federated (Ingerman & Ostrowski 2019)). It provides realistic proxy for federated data, since we can associate all posts by an author as a particular user (see Appendix [. for details). We artificially introduce the bug in the dataset by concatenating the first two tokens in 10% of all sentences, across users.
Two federated generative models are used in complementary fashion for debugging. The first is the 'primary' model, the DP word-I.M that is being trained via FL. for use in the next word prediction pipeline. While the bug is inhibiting its training for production use, we can still harness its generative properties for insight into the nature of the bug. The second is 'auxiliary' model for T3, DP char-LM trained only on OOV words in the dataset; this model is taken from Chen et al. (2019). but here our goal is debugging, not learning new words. We assume we have access to both the word-LM and char-LM trained before the bug was introduced, as well as both models trained on buggy data. Continuous training of auxiliary models is beneficial in a production workflow, to contrast output of models trained at different times or different selection criteria.
Both the DP word- and char-LMs are trained independently via simulated FL. (following the approach of McMahan al. (2018) the bug-augmented and bug-free Stack Overflow dataset, producing four trained models in total. These models are now used to synthesize useful and complementary information about the pipeline's 's input data. From Figure [ we notice that with the bug the sentences generated by the word-LM have a larger than normal OOV rate for the first position. In Table we see that with the bug most of the top OOV words generated from the char-LM have spaces in them, indicating a tokenization bug in the pipeline. This bug can then be fixed by the modeler. See Appendix B for further details and expanded results.
The dataset contains 342,477 unique users. Each round, 5,000 random users are selected The models trained for 2,000 communication rounds with DP hyperparameters as in Appendix [B.]] Table 5 As an example, this gives = 9.22 with d = 2.92 x 10-6, A larger number of users would lead to smaller
Privacy
AN APPLICATION TO DEBUGGING DURING INFERENCE WITH GANS
DP Federated GANs for Generating Image Data Generative Adversarial Networks (GANs) (Goodfel- low et al., 2014) are ofatc-of-the-ar form of deep generative model, with numerous recent successes in particular in the image domain (Isola et al. 2016: Zhu et al. 2017 Karras et al., 2018 2019 Brock et al. 2019). GANs work by alternttly training two networks. One is generator which maps random input vector in a low-dimensional latent space into a rich, high-dimensional generative output like an image. The other is a discriminator, which judges whether an input image is 'real' (originating from a dataset of ac- tual images) or 'fake' (created by the generator). Each network tries to defeat the other; the generator's ss training objective is to create content that the discriminator unable to discern from real content, and the discriminator's 's training objective is to improve its ability to discern real content from generated content.
We can take the GAN training framework and adapt it to FL and DP, analogous to RNN training under FL and DP in McMMann et al. (2018) via the DP- DP-FedAvg algorithm. The difference here is that two sets of model parameters are updated, by alternating minimization of two loss functions. One key insight that only the discriminator's 's training step involves the use of real user data (private and restricted to the user's device); the generator training step does not require real user data, and thus can be computed at the coordinating server via traditional federatd)))))) gradient update. A second insight is that the generator's 's loss is a function of the discriminator. As observed by earlier works involving DP GANs, if the discriminator is trained under DP and the generator is trained only via the discriminator, then the generator has the same level of privacy the discriminator, via the post-processing property of DP (Dwork & Roth 2014). No additional computational steps (e.g., clipping, noising) are necessary the application of the generator gradient update.
TThesc experiments used sampling with replacement instead of without replacement due to technical limitation of the simulation framework; however, this difference should have negligible impact on the utility of the models trained.
7

---

Published as a conference paper ICLR 2020
Server-orchestrated training loop:
UserDiscUpdate(k; &D. Ba):
parameters: round participation fraction g (0, 1], total number of N N, total number of rounds T N, noise scale E R*,, clip param- eter 5 E IR"
parameters: number of steps B N, disc. learning ter E R gen, input size ny E N. gen. function G(U;0G) disc. loss function Ep (0 Breal, blake)
N. batch size
di RR, cllp parame-
Initialize generator 0% discriminator B. privacy accountant MM
0n 03
x- (k's data split into n size B batches) for each batch brcal E B do
Set a= 25 q.V
U + (sample B random vectors of dim. no)
blke + G(U; 0G) || generated data
for each round from 0 T do
Op 0D 7/1 ( brea, brake)
C' <- (sample of qıV distinct users) for each user & E C in parallel do
A = 0D 08
return update A A min (1. II Clip
A + UserDiseUpdate(( 0b, 0%)
GenUpdate(OD,,
A
Σ ++
parameters: number steps n N. batch size B E N, gen. learning rate ÿa E IRt, gen. input size nu E N, gen. loss function &a (9c;
kECe
0 0b -+1 ((0,100))
Og 02
for each generator training step from to n do U + (sample B random vectors of dim. ny) buae + G(U;6c) 11 generated data
0/²' < GenUpdaate(?)"1.00 0%)
print p.t...gge..rrivcc.seee
0c 8G na VoaVlafe: brace, 0p)
return AG
Algorithm DDD-FeedAv- based on De-FedAvg (App.
but accounts for training both GAN models.
Algorithm [ ('DD-FedAvg- GAN') describes how to train a GAN under FL and DP. The discriminator update resembles closely the update in standard De-FedAvg, and then each round concludes with a generator update the server. The discriminator explicitly trained under DP. During training the generator only exposed to the discriminator, and never directly to real user data, so it has the same DP guarantees as the discriminator.
Experiment Our second experiment shows how auxiliary DP federated GANs can be used to monitor an on-device handwriting classification network being used for inference, Once trained, the federated GANs synthesize privacy-preserving samples that identify the nature of a bug in on-device image preprocessing.
Consider the scenario of banking app that uses the mobile phone's camera scan checks for deposit. Internally, this app takes raw images of handwriting, does some pixel processing, and feeds the processed images to a pre-trained on-device Convolutional Neural Network (CNN) to infer labels for the handwritten characters. This CNN is the 'primary' model. The modeler monitor its performance via metrics like user correction rate (i.c., how often do users manually correct letters/digits inferred by the primary model), to get coarse feedback on accuracy. To simulate users' processed data, (Caldas et al. 2018). Figure 2a shows example images; see Appendix [.T for further details on the data.
the Federated EMNIST dataset
Suppose new software update introduces bug that incorrectly flips pixel intensities during preprocessing. inverting the images presented to the primary model (Figure 2b) 4 This change to the primary model's input data causes incorrectly classify most handwriting. As the update rolls an increasing percentage of
*Such data issues are in fact common in practice. For example, the Federated EMNIST datasct encodes pixel intensities with the opposite sign of the MNIST dataset distributed by TensorFlow (TFF documentation 2019).
8

---

Published as a conference paper ICLR 2020
7
X
V
$
4
0
7
Q
J
9
r
9
M
&
F
3
M
7
P
9
8
7
3
1
7
6
S
N
3
1
T
2
E
6
r
T
3
4
h
9
d
3
4
3
(a) Expected. (Without Bug)
(b) Inverted. (Bug)
(a) Trained on
high-
(b) Trained accuracy users.
low-
accuracy users.
Figure 2: Examples of primary model CNN input, Figure 3: DP federated GAN generator output given from EMNIST with letters and digits (62 classes).
an inversion bug on 50% of devices.
the app's ss user population, the user correction rate metric spikes, indicating to the app developers problem exists, but nothing about the nature of the problem. Critically, the images in Figure are never viewed by human eyes, but are fed to the primary model on device. If the data were public and the inference being performed on the app's server, some of the misclassified handwriting images could potentially inspected, and the pixel inversion bug realized. But this cannot be done when the data is private and decentralized.
Instead, we train two DP federated GANs via Algorithm[] [ one on a subset of the processed image data that tends to perform best when passed to the primary model, and one a subset of processed image data that tends to perform worst. (Specifically, we do this by forming two distinct subpopulations of users who exhibit high and low classification accuracy, respectively, and training GAN on each user subpopulation, as in T4; full details in Appendix CC.3 and an alternative selection criteria explored in Appendix C.4] By con- trasting images synthesized by the two GANs, hope understand what is causing degraded classification accuracy for some app users. In our simulation of this scenario, we train for 1,000 communication rounds, with 10 users (drawn random) participating each round. See Appendix C.2 for training details. Figure shows images generated by the two trained GANs, when the bug is present on 50% of users' phones.
Comparing the GANs' output, the presence of pixel inversion bug is clear, enabling the modeler to locate and fix the issue 5 natural follow-on is to repeat training the two GANs after the fix is made; results shown in Figure (Appendix [C3.I]). Comparing these new images verifies the pixel inversion bug fixed. also potentially reveals new insights. Perhaps it identifies another, more subtle bug, or a particular characteristic of handwriting common to users with many misclassified examples (indicating a missing user class the primary model's training data, i.c. instance of T6). We propose it is useful continually train federated generative models like the GANs in this experiment, as a tool to monitor and debug.
Privacy at Scale We follow an approach in McMahan et al. (018)) to determine the level of privacy at realistic population sizes. We simulate with 10 users per round and small total population size (3,400) order to validate the utility of our models and also determine noise scale multiplier 2 that does not degrade the quality of the GAN's generated images. This simulation scenario will not afford good privacy protections (as indicated by the resulting very large bound). But it does indicate the factor necessary to scale up to 1.0; this is the factor we'd have to apply to per-round participation count to achieve a real-world scenario that affords good privacy protections (as measured by and d) while maintaining the quality of the GAN's generated images. As the experiment shows good utility is achieved with a z of 0.01, the factor to apply
'The relatively low quality of generated characters is irrelevant for this task; more training might result in higher quality, but these results sufficient diagnose the bug. Training be stopped early for better DP guarantee.
9

---

Published as a conference paper ICLR 2020
Table 3: Privacy parameters for different scenarios. N is size of user subpopulation that meets selection criteria (not overall population size). Simulations with overall population of 3,400, and realistic scenarios are with overall population of 2,000,000. All experiments use clip parameter S of 0.1 and 1 1,000 rounds.
g-V
N
e
6
simulation (Figure
3a
10
425
0.01
9.99 x 10
2.35 x 10 3
realistic scenario
1,000
250, 000
1.00
2.38
4.00 10-8
simulation (Figure 3b
10
2, 125
0.01
9.99 10
4.71 10
realistic scenario
1, 000
1, 250, 000
1.00
1.48
8.00 10-9
simulation (Figure
10
850
0.01
9.99 10
1.18x 10-3
realistic scenario
1000
500, 000
1.00
1.79
2.00 x 10-8
100: a real-world use case would involve 1,000 users per round, out of total population of e.g. million mobile app users. This reasonable for a real-world system, e.g. Bonawitz al. (2019).
Table 3 gives DP hyperparameters and resulting (c, 8) values, both for the simulation and the corresponding scenario with scaled-up, realistic population. The calculation of 8) is fully described in Appendix A.1 Note that three sets of numbers are presented, as (c.0) depend on the number of users meeting the selection criteria for a given use case. The cases are: (i) high-accuracy users when the bug exists (12.5% of overall population), (ii) low-accuracy users when the bug exists (62.5% of overall population), and (iii) either group of users when the bug gone (each 25% of overall population). In all cases, for the realistic population we achieve single digit (between 1,48 and 2.38), indicating reasonable amount of user privacy.
7 CONCLUSION
In this work we described ML modeler workflows that rely data inspection and are hence precluded when direct inspection is impossible (i.e., in a private, decentralized data paradigm). We proposed a tool to overcome this limitation: differentially private, federated generative models that synthesize examples representative of the private data. We demonstrated application of two example model classes (DP federated RNNs and GANs), showing how they enable a modeler to debug natural language and image problems. In the GAN case, we presented a novel algorithm for training under FL and DP. Section 2 discusses several other modeler workflows (e.g.. debiasing, labeling) that rely on data inspection and are inhibited in private, decentralized paradigm. Experiments applying DP federated generative models to these workflows is a promising direction for future work, especially since applications like data labeling will likely require higher- fidelity generative models than the ones considered here.
To convert the concept proposed in this paper into a practical data science tool, variety of further research is necessary. For federated generative models to be useful and broadly applicable, they should require minimal tuning: excessive experimentation needed to achieve convergence both destroys the value proposition the modeler and destroys the privacy budget under DP analysis. They must be capable of synthesizing relevant content in scenarios where the "signal-to-noise" ratio low, e.g., when a bug is only present a small amount of data (in absolute relative terms). How to achieve this in the case of GANs, which are powerful but can suffer from mode collapse, of special interest. Advances in these dimensions will greatly benefit the private, docentralized data science community. As initial step to stimulate research in this area, provide an open-source implementation of our DP federated GAN code (used to generate the results in Section 6). We share additional thoughts on open problems in Appendix
10

---

Published as a conference paper ICLR 2020
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan MeMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In 23rd ACM Conference on Computer and Communications Security (ACM CCS), 2016.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's 's software used across the country to predict future criminals. and it's biased against blacks. ProPublica, 2016.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algo- rithms and tight error bounds. In Proceedings of the 2014 IEEE 55th Annual Symposiun on Foundations of Computer Science, FOCS "14, pp. 464-473, Washington, DC, USA, 2014. IEEE Computer Society, URL http: doi org/
ISBN 978-1-4799-6517-55 doi: 10. 10.1109/1 10.1109/FOCS.2044.
FOCS.2014:
Brett K. Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev Bhavnami, James Brian Byrd, and Casey S. Greene. Privacy-preserving generative deep neural networks support clinical data sharing. bioRxiv, 2018. doi: 10.1101/159756. URL https://ww..bio: http:/www.brrxxxx. org/ org/cotttt///////ooooooooo
2arly/2018/12/20///
Keith Bonawitz, Hubert Hichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMAhan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In SysML2019 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://
openreview.nee net. faret//ooums (errummidd-BBxsqj09999
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint 1 2018
N. Carlini, C. Liu, J. Kos, Ú. Erlingsson, and D. Song. The Secret Sharer: Mcasuring Unintended Neural Network Memorization & Extracting Secrets. ArXiv e-prints, February 2018. URL https://arxiv:
brq/abs/1882..00 08232 08232
Aleksandar Chakarov, Aditya Nori, Sriram K. Rajamani, Shayak Sen, and Deepak Vijaykeerthy, Debug- ging machine learning tasks. CoRR, 100.09903,, 2016. /ttp wttp:////aaaaa.....///// 07222
Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk mini- mization. J. Mach. Learn. Res., 12, July 2011.
Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Françoise Beaufays. Federated learning of out-of- vocabulary words. CoRR, 106355. 2019. URL http:://aaaxxiv.crr/a////0...
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with generative counterfactual face attribute augmentation. CoRR, htss/1966.6639,, 2019. URL httl:/ttpp//axi...oo abs/ 1906 06439
11

---

Published as a conference paper ICLR 2020
Cynthia Dwork. A firm foundation for private data analysis. Commun. ACM, 54(1):66-95, January 2011. ISSN 0001-0782. doi:
18666788. URLhttp:///aa acm wt...org/1111100.
1866739. 1866758
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science. Now Publishers, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pp. 265-284. Springer, 2006.
Cristóbal Esteban, Stephanie Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans, 2017. httt:://axxiv.ooottt /abs/ 1atpp///1111.38311
Lorenzo Frigerio, Anderson Santana de Oliveira, Laurent Gomez, and Patrick Duverger. Differentially private generative adversarial networks for time series, continuous, and discrete open data. CoRR, abs/1901.024777, 2019. URL
abs 1901. 02477
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL http://papers. nips. cc/pppr/ wata--generaliieedvvveerrrraeaee
pdd
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. Book in preparation for MIT Press, 2016.
Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. CoRR, abs/1503.0406t 2015. URL http:://aa......rr/aa///////
04069
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved train- ing of wasserstein gans. Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, NIPS' 17, pp. 5769-5767777 5779, USA. 2017, Curran Associates Inc. ISBN 978-1-5108 108-6609644 URL. http:/tt:://11.... a cm. org/ tat on www.cccmm...mm....m....m wwt.mmfm==33999222223333 32953277
Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811 1.11 180.00304000 2018.
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. MD-GAN: multi-diseriminator geriirrrimiinaaogggnettiiiiee adversar- ial networks for distributed datasets. CoRR,
2018. URL http:tt/////xii.rrgg// abs//
1811.0381 03850
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan:
Membership inference
attacks against generative models. Proceedings Privacy Enhancing Technologies, 1019(1) 2019.
Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Reconstruction and membership inference at- tacks against generative models. arXiv preprint arXiv: wexiv:/906...0 /906.03006, 2019.
Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. Taxonomy of real faults in deep learning systems, 2019.
Alex Ingerman and Krzys Ostrowski. Introducing tensorflow federated, 2019. URL http:: /medium
12

---

Published as a conference paper ICLR 2020
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. USENIX Security Symposium, Aug 2019. URL httpttttp:///rr.af.... wtt..a/f..o//bbbl/////9999999990 http:///w////////////a/////////a
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy.
IEEE Transactions on Information Theory, 63(6):4037-4449, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL
htttps:/apenreviiee net/ forum? 2idmmmmmddddeeeeeeee
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and
Durk Kingma. Videoflow: flow-based generative model for video. CoRR, abs/1903.014344 2019. URL
http:/ www.aaiiii.gaaaaaaaaaaaaaaaaaaaa abs 1903...... 01434
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication--efiee learning of deep networks from decentralized data. In Artificial Intelligence and Stutistics, pp. 1273-1282, 2017.
H Brendan McMahan and Daniel Ramage. Federated learning: Collaborative machine learning with- out centralized training data, April 2017. URL https: //ap:://ai.. googlebloo......ooooooooooooooo. wwwc 2017/04/
html Google Al Blog.
H. Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan
Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat Chikkerur, Dan Liu, Martin Wattenberg, Amar Mar Hrafnkelsson, Tom Boulos, and Jeremy Kubica. Ad click prediction: view from the trenches. In KDD, 2013.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR), 2018
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson,, Ian Goodfellow, and Kunal Talwar. knowledge transfer for deep learning from private training data. In Proceedings of the International Conference on Learning Representations, 2017. httttt://////siit wttp:////wwwwww./o///a/ao//// wwtp://ooggaa////.e/.osssss
Semi semi-supervised
05755
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In OpenAl Technicul report, 2019.
Patrick Riley. The unofficial google data science blog: Practical advice for analysis of large, complex data sets, October 2016.
Hasim Sak, Andrew W. Senior, and Françoise Beaufays. Long short-term memory based recurrent neural
network architectures for large vocabulary speech recognition. CoRR, abs/1402.1 1228, 2014. URLhttt:
//atxiv. wwwpx/ww.rrg/abs/100.....
13

---

Published as a conference paper ICLR 2020
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2234 2242. Curran Associates, Inc., 2016. URL 6125- improved
for peerraiiingggnss.....e pdf
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recogni- tion and clustering. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. Hidden tech- nical debt in machine learning systems. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances Neural Information Processing Systems 28, pp. 2503-2511. Curran Associates, Inc., 2015. URL Learning systemss.sssss......o...... .
mips. cpssccc//pprrr
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models, 2019.
TFF documentation. 2019. URL http:///www..eenoo.oo..ooo/oeeee http://///ppp..oooo//p//oo////// s htw.r//nn/aaaasstseemmsssss//s//
http://ww.....tt..oo.oo.ooottnna documentation,
t
data
Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Aleksei Triastcyn and Boi Faltings. Federated generative privacy. Proceedings of the IJCAI Workshop on Federated Machine Learning for User Privacy and Data Confidentiality (FMI. 2019), 2019. URL
http:/ fict:://infot http://nnnnnciioocoooo/ooooooocc www.fosciinncc.ccm ence.epf......e cchh/eccord/229922
Yu-Xiang Wang, Borja Balle, and Shiva Kasiviswanathan. Subsampled rl'enyi differential privacy and analytical moments accountant. arXiv preprint arXiv:1808.000 2018.
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey F. Naughton. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of SIGMOD, 2017.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversar- ial network. CoRR, abs/1802.06739, 2018.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint 18/2.02903, 2018.
Xinyang Zhang, Shouling Ji, and Ting Wang. Differentially private releasing via deep generative model. CoRR, 000/18011019994 2018 URL http:////aari org/ wwa.b/////1...... 01594
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
Martin Zinkevich. Rules of machine learning: Best practices for ml engineering., 2017. URL http: / /ma rt itin. n. nkevichhhhc wwwevvichhrrgrrlleeeeeeeee ruw.s _o woc.ooommmmrleeeeeeeeee _ftess_oooff__ee _f__m_ .pdf
14

---

Published as a conference paper ICLR 2020
A USER-LEVEL DIFFERENTIAL PRIVACY AND FEDERATED LEARNING
Differential privacy (DP) (Dwork et 2006 Dwork 2011 Dwork & Roth 2014 provides a well-tested formalization for the release of information derived from private data. Applied to ML, a differentially private training mechanism allows the public release of model parameters with strong guarantee: adversaries are severely limited in what they can learn about the original training data based on analyzing the parameters, even when they have access to arbitrary side information. Formally, it says:
Definition Differential Privacy: A randomized mechanism M: D - R. with a domain 'D (e.g. possible training datasets) and range RR (e.g. all possible trained models) satisfies (c, difffeerrntlaa privacy if for any two adjacent datasets d, d' E D and for any subset of outputs S S R it holds that Pr[M(d)= E S e Pr[MM(d') E + 6.
The definition above leaves open the definition of adjacent datasets which will depend on the application. Most prior work on differentially private ML (e.g. Chaudhuri et al. (2011); Bassily et al. (2014); (Abadi et al. (2016); Wu et al. (2017); Papernot al. (2017) deals with example- level privacy: two datasets d and d' are level
defined to be adjacent if d' can be formed by adding or removing single training example from d.
However, recent work has presented algorithm (DP-FedAvg, McMahan et al. (2018) for differentially private ML with user-level privacy. Intuitively speaking, a model trained under De-FedAvg should not change too much when one user's data added, removed, or changed arbitrarily. Learning takes place via FL; since FL processes all of one user's data together, such guarantees are relatively easier to obtain than in the centralized setting.
We adopt an approach (Algorithm 2) for performing differentially private FL that follows closely from the DP-FedAvg algorithm. Similarly, our focus is on user-level differential privacy, where the privacy guarantee extends to all of user's data (rather than e.g. single training example).
Specifically, following DP-FedAvg, we:
enforce clipping of per-user updates the total update has bounded Le norm.
use bounded-sensitivitg estimator for computing the weighted average update to the global model.
add Gaussian noise to the weighted average update.
The sensitivity of query is directly proportional to the amount of clipping S and inversely proportional the number of participants in a training round g.N. Algorithm@ Sisows that noise scale parameter relates the amount of standard deviation in the added Gaussian noise this query sensitivity. As described in McMahan et al.) (2018), acceptable DP bounds are typically achieved when 2 2 1.0. So method of determining privacy hyperparameters that afford acceptable utility and privacy to start with a small number of per-round participants qN,, determine values of S and a that provide good utility, and then increase the number of per-round participants so that = Z 1.0.
A slight difference between our approach and that presented in DP-FedAvg is how we select devices to participate in a given federated round of computation. DP-FedAvg uses randomly-sized federated rounds, where users are selected independently with probability q. In this paper, instead use fixed-size federated rounds, where the number of users selected to participate in the round, qNN, is a constant (and N is the total number of mobile devices participating in training). Lee. g is the round participation fraction. This method of sampling has minor but non-trivial ramifications for the calculation of the overall privacy bound, which we discuss next.
15

---

Published as conference paper ICLR 2020
Server-orchestrated training loop:
UserUpdate(A ))):
parameters
parameters
round participation fraction o E (0, 1] user population N E N noise scale 2 E R clip parameter S E K
number of local epochs E N batch size learning rate E RR' clip parameter $ E IR+ parameter functions loss function {(0;b)
Initialize model 0%, privacy accountant M
Set o
25 q.V
0 + 80
for each local epoch from to E do t- (k's data split into size B batches) for each batch E do
for each round 0, ..... do
C <- (sample without replacement gN users from
population)
A = 0 4 1)7
for each user & E C' in parallel do A+ UserUpdate(k;,
80
S. return update A = A min (1,
// Clip
A41 4+
Σ
RECE
ge+1 Hf + A+ (O0 Toa)
M.accum.pra wwt.:pppvvviiiieee spending)())
print w.t......ppiivccsseeee
Algorithm DP-FedAvg with fixed-size federated rounds, used train word- and char-LMs in Section
A.1 PRIVACY BOUNDS
To obtain precise DP guarantees, we use the analytical moments accountant of subsampled Rényi differential privacy (RDP) method developed in Wang et al. (2018). particular, this work provides tight upper bound on the RDP parameters for an algorithm that: (a) uniformly subsamples records from an underlying dataset, and then (b) applies randomized mechanism the subsample (Gaussian perturbation in our case). This precisely what happens in one round of Algorithm Mo [
To track the overall privacy budget, we multiply the RDP orders of the uniformly subsampled Gaussian mechanism by the number of rounds we execute Algorithm or 2 and convert the resulting RDP orders to an (e, d) pair using Proposition of Mironov
(2017)
A.2 ADDITIONAL CONSIDERATIONS
Note that 0e DP upper bounds are often loose, and so can be complemented by direct measurement of memorization, as in Carlini et al. (2018); (Song & Shmatikov (2019). Further, while often unrealistic as a threat model, membership inference attacks memorization (Hayes et al.) 2019 Hilprecht et al. 2019). Finally, we note that since the modeler has access not only the auxiliary generative models, but also potentially other models trained the same data, if an (c. 8) guarantee is desired, the total privacy loss should be quantified across these models, e.g. using the strong composition theorem for DP (Mironov 2017
GANs can also be used tool to empirically quantify
Kairouz et al. 2017)
16

---

Published as conference paper ICLR 2020
B DP FEDERATED RNNs: EXPERIMENTAL DETAILS
B.1 STACK OVERFLOW DATA
The Stack Overflow dataset contains questions and answers from the Stack Overflow forum grouped by user ids. The data consists of the text body of all questions and answers. The bodies were parsed into sentences; any user with fewer than 100 sentences was expunged from the data. Minimal preprocessing was performed to fix capitalization, remove URLs etc. The corpus is divided into train, held-out, and test parts. Table summarizes the statistics on number of users and number of sentences for each partition. The dataset available via the Tensorflow Federated open-source software framework (Ingerman & Ostrowski 2019).
Table 4: Number of users and sentences in the Stack Overflow dataset.
Train
Held-out
Test
# Users
342K
38.8K
204K
Users with both questions and
297K
34.3K
99.3K
# Sentences
136M
16.5M
16.6M
Sentences that questions
57.8M
7.17M
7.52M
Sentences are answers
78.0M
9.33M
9.07M
B.2 RNN MODEL ARCHITECTURES AND TRAINING
DP Word-LM The DP word-LM model is based coupled input forget gate CIFG-LSTM (Greff et al. 2013) with projection layer (Sak et al. 2014). uses a 10K vocabulary, which is composed of the most frequent 10K words from the training corpus. The model architecture is based 670 and an input embedding dimension of 96. The model is trained for 2,000 rounds. A server learning rate of 1.0, on-device learning rate of 0.5, and Nesterov momentum of 0.99 used.
RNN with hidden size
DP Char-LM The model architecture used to train the DP char-LM is based on "FL in Chen et al. (201)) ihich also CIFG-LSTM with projection using 3 layers. 256 RNN hidden units and 128 projected dimensions. The input embedding dimension is also 1 128. The character vocabulary has a size of 258, based on UTF-8 encoding with 256 categories from a single byte and additional start-of-word and end-of- word tokens. All other training and optimization parameters
the same the word-LM model. Similar to Chen et al. 2019), once the char-LM trained, Monte Carlo sampling performed to generate OOV words.
Privacy Hyperparameters Table gives the privacy hyperparameters used with DP-FedAvg (Algorithm 2 to yield the trained Word-LM and char-LM models that generated the text results presented here.
Table 5: Privacy hyperparameters for DP RNN experiments.
Le clip parameter S
participation count g.N
total users
noise scale
number of
N
2
rounds T
DP word-LM
0.2
5,000
342,777
1.0
2,000
DP char-LM
0.1
5,000
342,777
1.0
2,000
17

---

Published as conference paper ICLR 2020
B.3 EXPANDED RESULTS
To demonstrate the effectiveness of using the OOV model monitor the introduced system bug, the ex- periments are done in four comparison settings: (1) without bug; (2) 1 % of sentences have first two words concatenated; (3) 10% sentences; (4) 100% sentences. In each setting the model trains for 2,000 commu- We observed in all four settings that
nication rounds, to meet the privacy guarantees stated in Section 5 models were converged after 2,000 rounds.
DP Word-LM Figure 4 shows percentage of OOV words by their position in the sentence. It clearly shows that as the percentage of sentences affected increases, the word-LM reflects this in generated output.
Table 6 shows the overall OOV rate observed in four experiment settings. It can be seen that overall OOV rate goes up as the percentage of sentences affected by the concatenation bug increases. Thus overall OOV rate can be used as an early abnormality indicator of the corpus.
Table 7 shows samples of phrases generated from the word-LM different experiment settings. We can see that the phrases have more OOVs (marked as UNK) as the percentage of sentences affected by the concatenation bug increases.
DP Char-LM Table 8 shows the 20 most frequently occurring OOV words generated by the char-LM, for the four different experiment cases. These generated results provide actual information on the tokenized words being fed to the model from the decentralized dataset. Similar to the word-LM, as the ratio of the bug increases, the chance of generating content reflecting the bug (in this case, concatenated words) becomes higher. In the 100% setting, all the top 20 words are concatenated words; the word-level joint probabilities of the concatenated words are significantly higher than their non-concatenated counterparts.
% of generated words that OOV
Without bug 1% with bug
10% with bug
100% with bug
60
40-
20
2
5
Position in sentence
Figure 4: Percentage of samples generated from the word-LM that are OOV by position in the sentence (computed over 100,000 samples).
Table 6: Overall OOV rate observed (during training) in different word-LM experiment settings.
without bug
1%
10%
100%
6.52%
6.63%
7.60%
17.89%
18

---

Published as conference paper ICLR 2020
Table 7: Sample phrases generated from word-LMs when 0%, 1%, 10%, and 100% of sentences affected.
without bug
1 %
For UNK metrics to work fine ago 1 think your question is the correct format question 1 UNK 7 java URX the selector of the type of module basically i do bunch of articles UNR data UNK UNK UNK 8 43 UNK this is blog post for huge
this tries to load the texthox by get any keer who was originally have an which one la differen: at the same time UNK currently building my ios app to integrate the only thing can think of is / home UNK jmeter /
the solution with application the current there is another solution in their UNk feed source properties in your worst
one possibility is to send diffarent limits to this is missing after that UNK start the above then
functions UNK index
your client
thank you
10%
100%
locks like it are pretty weather when i suppose
JNK lg for this is computing 3 as it is that will stop UNR of UNX you need to call uNK UNK might this
is executed when you freeze
delete disadvantage of the files and prediction system
UNK folder created please note that the button would mount would displaying UNK just all the UNK files auch UNK motter why note
UNK in the choice of zort of dragging pl
field
JuK UNK editing the push controllers
JuK must dapend
way
simply at UNK UNK but put it i started many debug Irom the jboss
JRK do it static UNK stop
UNK tzying give more html privileges ?
19

---

Published as conference paper ICLR 2020
Table 8: Top 20 char-LM --geenerated OOV words by joint character probability, for 0% (without bug), 1%.
10%, and 100% of examples affected by bug. Value alongside is joint probability (computed via Equation T
without bug (prob 10 10
1% (prob 10 3)
10% (prob 10-3)
100% (prob 10 10-2)
regex
5.50
regex
5.15
have 8.08
have 27.47
jsfiddle
3,90
jsfiddle
3.73
1 am
5.60
i am
16.50
xcede
3.12
xcode
2.86
regex
4.45
this is
9,92
divs
2.75
1istview
2.64
you can if you
3.71
you can
9.04
stackoverflow
2.75
textbox
2.56
2.81
here is
8.30
listview
2.74
divs
2.45
this is
2.77
if you 8.29
textbox
2.73
stackoverflow
2.44
here is
2.73
want 8.17
foreach
2.34
1is
2.27
jsfiddle
2.70
is there 7.51
async
2.27
foreach
2.24
i want
2.39
how can
5.78
iis
2.21
ling
2.20
textbox
2.28
when i
5.33
onclick
2.07
async
2.17
xcode
2.23
however
5,25
1ing
2.02
onclick
2.05
listview
2.10
would if i 4.22
4.72
params
2.00
htaccess
1.90
stackoverflox
2.10
htaccess
1.90
params
1.89
is there
2.07
for example
4.20
arraylist
1,90
mongodb
1,86
divs
1.93
tried 4.14
mongodb
1.83
при
1.77
async
1.79
edit : 4.07
xaml
1.79
xaml
1.74
linq
1.72
i think 3.86
nрm
1,66
arraylist
1.67
foreach
1.71
the problem
3.77
dataframe 1.61
enum
1.58
iis
1.70
don't
3.61
nginx nx
1.55
nginx
1.55
however
1.69
need
3.49
C DP FEDERATED GAN: EXPERIMENTAL DETAILS
C.I FEDERATED EMNIST DATA
We use the Federated EMNIST dataset (Caldas et al 2018 to represent the app's users' processed data our GAN experiments in Section 6 This dataset contains 28x28 gray-scale images (Figure 2a) of handwritten letters and numbers, grouped by writer. This has the realistic property that all data grouped to given user exhibits the same personally-identifying handwriting properties; solution should generate examples while obfuscating these properties. The dataset contains 3,400 users. It is available via the Tensorflow Federated open-source software framework (Ingerman & Ostrowski 2019).
C.2 FEDERATED GAN ARCHITECTURE AND TRAINING
Models and Losses Our generator and discriminator network architectures are borrowed from a popular GAN code tutorial oor MNIST. For the generator and discriminator loss functions, lG() and (p(), we used the Improved Wasserstein loss formulation presented in Gulrajani et al. (2017). We used gradient penalty coefficient of 10.0.
Hyperparameters Table 9 gives the various hyperparameters used with the DP-FedAvg-GAr algorithm to yield generators that produced the images displayed in Figures 3 6 and
hhtps//githhh.cm//eessffwwwya
20

---

Published as a conference paper ICLR 2020
Table 9: DP federated GAN experimental hyperparameters.
generator training steps n 6
generator batch size B
generator learning rate 1JG
generator
input size ne
32
0.005
128
discriminator training steps
discriminator batch size B
discriminator
L2 clip
learning rate Ijn
parameter S
6
32
0.0005
0.1
participation count qNV
total users
noise scale
number of rounds
N
2
T
10
see Table
17
0.01
1,000
Network Update Cadence For simplicity, Algorithm I shows a 1:1 ratio of ing updates. This is not required. The discriminator could go through multiple federated updates before an update to the generator is calculated, just as is the case with non-federated GANs (Goodfellow et al. (014). In these experiments used 1:1 ratio of discriminntor:geeeraaco training updates, as this update cadence empirically was determined to work well.
train-
C.3 SELECTION CRITERIA: FILTERING BY USER
The modeler ideally wishes to train one GAN on the exact subset of data affected by the bug and the other GAN on the exact subset of data unaffected by the bug, in order to best discern the difference and identify the bug. Of course, to achieve this perfectly they would need to know what the actual bug is a priori. As they don't, the modeler must settle for focusing the two GANs to train on the subsets of the data with the highest and lowest likelihood (respectively) being related to the bug.
A question is how to define these two subsets of the overall decentralized dataset. In the experiments in Section 6 we filter these two subsets at the level of the user (i.c., T4 in Table We define criteria for a user's app experience to be considered as 'poor', and if a user meets this criteria then their device belongs to the 'poorly performing' subpopulation. When their device is selected to participate in a round, their examples will be used calculate updates to the GAN discriminator for this subpopulation. (The equivalent is true for identifying members of the 'strongly performing' subpopulation.)
The criteria we use is user's classification accuracy, i.e., the percentage of examples in their local dataset that were correctly classified by the primary model (the CNN). F a histogram of classification accuracy for the app's users, and Table I0 gives the 25th and 75th percentiles when no bug exists. We set accuracy thresholds of 'poor' (and 'strong") performance based on the 25th (and 75th) percentile, i.c.. when no bug exists we aim to capture the 25% of users who experience the worst (and best) classification accuracy.
When a bug exists and performance of the primary model drops, the 'poorly performing' subpopulation swells and the 'strongly performing' subpopulation shrinks (note the shift in histogram in Figure with and without the bug). That is, subpopulation size varies depending on the degree of impact of the bug. Table gives the subpopulation sizes that resulted for the experiments we ran.
Eaah user in the Federated EMNIST set has different number of examples. Some users have less than 6 x 32 examples total. When such a user is selected for a round by the coordinating server, the discriminator update will use all its examples train, but the batch size of the final batch could be less than 32 and number of steps taken could be less than 6.
21

---

Published as conference paper ICLR 2020
1800
No bug
1600
Bug 50% of user devices
1400
1200
Users (#)
1000
800
400
200
0
10
95100
Accuracy (%)
Figure 5: User accuracy histogram.
Table 10: User accuracy percentiles (without bug).
Accuracy
75th Percentile Users
93.9%
25th Percentile of Users
88.2%
Table 11: Subpopulation sizes for simulations in Section
Total population size is 3,400 user devices.
Without Bug
Bug in 50% user devices
High-Accuracy Users
850
425
Low-Accuracy Users
850
2,125
22

---

Published as a conference paper at ICLR 2020
Of course, this not perfect manner for identifying which data contains the bug (although as the bug unknown, no manner could be perfect priori). The poorly performing subpopulation will always contain the 25% of the overall population that doesn't 't classify well for reasons unrelated to any bug (e.g.. users with irregular handwriting, etc.), and their data will be training the discriminator along with the data actually affected by the bug. The assumption is that either the generative model has the capacity to model both modes of data, or the unaffected data will be dwarfed by the affected data and the generative model will focus on the larger mode of data. This assumption held for this experiment, but further research on the "sensitivity" of selection criteria is necessary (see Section 7 and Appendix D)
An alternative selection criteria, where data is split up by classified and misclassified examples (as opposed by high accuracy and low accuracy users), is presented in Appendix C.4
C.3.1 ADDITIONAL GENERATED RESULTS
Figure the output of two DP federated GANs trained on the best and worst-classffyngg devices, once the bug described in Section 6 is fixed. These synthetic images are of high-enough quality to verify to the modeler that the pixel inversion bug they observed in Figure 3 is gone.
9
7
7
8
9
2
L
3
7
4
4
7
T
9
7
1
17
q
7
0
3
5
1
2
A
4
4
7
A
3
7
8
R
8
(a) Trained on high-accuracy users.
(b) Trained on low-accuracy users.
Figure 6: DP federated GAN generator output after the inversion bug is fixed.
C.4 ALTERNATIVE SELECTION CRITERIA: FILTERING BY EXAMPLE
As Section 2 describes, there are number of selection criteria that an ML modeler might employ in the course of their modeling tasks. The experiment described in Section 6 and Appendix C.3 involved data selection that filtered "by user', based on a user's classification accuracy (i.e., T4 in Table D. Here we demonstrate an alternative selection criteria: filtering 'by example', based on whether a data example was correctly classified by the primary model (i.e., T2 in Table TT. We show that we can equivalently use this alternative data selection and train DP federated GANs to debug the primary model and uncover the source of the drop in accuracy observed in Figure 5
Informally, one can consider the difference between filter "by user' and filter bby example' as "train on all examples, for some users" "train on some examples, for all users". In the former, the amount of users contributing to training varied quite a bit based on whether the bug was present or absent and whether we were training the high-accuracy or low-accuracy subpopulation (Table IT). With the latter, the amount of
23

---

Published as conference paper ICLR 2020
Table 12: Subpopulation sizes for filter "by example' simulations, when excluding a user's participation in subpopulation if containing < 5 examples of classification result. Total population size is 3,400 users.
Without Bug
Bug in 50% of user devices
Correctly Classified Examples Misclassified Examples
3,400
2,905
3,282
3,340
Some variation in subpopulation sizes is still present, due to our requiring a user device to contain a minimum number of examples of a classification result (either 'correctly classified' or "misclassified" in order for it participate in training for that particular subpopulation. This minimum number of examples threshold is applied to mitigate effect of outliers. In the experiment we used a threshold 5.
users in training subpopulation varies to much smaller extent (Table 12
Experiment As in Section 6 we train two distinct DP federated GANs via Algorithm 1 in order to infer the nature of the bug from the contrast in synthesized content between the two. One GAN trains on the examples each device that were correctly classified by the primary model, and the other trains examples that were misclassified. We again trained each GAN for 1,000 communication rounds involving 10 users per round.
Figures and on 50% of users' mobile phones. As with the 'by user' experiment, the 'by example' results here show that DP federated GANs can be trained to high enough fidelity to clearly distinguish that some private data has black and white reversed. Figures 8a and [8b] show examples of images generated by the two GANs when trained with the bug present on no phones (c.g., after the bug is fixed).
and 7b] show examples of images generated by the two GANs when trained with the bug present
}
4
A
6
1
(a) Trained on correctly classified examples (for all users).
(b) Trained on misclassified examples (for all
users).
Figure 7: Filter 'by example' simulation: generator output with the inversion bug present on 50% of devices.
Privacy at Scale Privacy bounds for these 'by example' experiments are determined in analogous fashion to the 'by user' experiments in Section In simulation we ran with 10 users per round and total population size of 3,400, validate good utility the presence of clipping and noising. While the and bounds
24

---

Published as conference paper ICLR 2020
J.
/
4
3
4
17
9
7
5
7
5
0
7
-
7
3
4
S
(a) Trained conrectly classified examples (for users).
(b) Trained misclassified examples (for all users).
Figure 8: Filter "by example' simulation: generator output after the inversion bug is fixed.
Table 13: Privacy parameters for different filter 'by example' (T2) scenarios. N size of user subpopulation that meets selection criteria and realistic scenarios are with overall population of 2,000,000. All experiments use clip parameter of 0.1 and 1,000 rounds.
(not overall population size). Simulations are with overall population of 3,400,
q/V
8
simulation (Figure
7a
10
2, 905
0.01
9.99 x 106
3.44 10
realistic scenario
1,000
1, 708,
824
1.00
1.47
5.85 x 10
simulation (Figure
7b
10
3,340
0.01
9.99
106
2.99 10-4
realistic scenario
1,000
1, 964,, 706
1.00
1.40
5.09 x 10 -9
simulation (Figure
8al
10
3,400
0.01
9.99
106
2.94 10 -4
realistic scenario
1,000
2, 000, 000
1.00
1.39
5.00 10-9
simulation (Figure 8b)
10
3, 282
0.01
9.99
106
3.05 x 10-4
realistic scenario
1,000
1, 930, 588
1.00
1.40
5.18x 10
do not guarantee meaningful privacy protection this small scale participants, the amount of clipping and noising are such that if we to scale up a larger realistic scenario, we have meaningful DP guarantees. Table 13 summarizes the privacy bounds, and shows we achieve single digit bounds for realistic scenarios involving 1,000 users per round and total population size of 2,000,000 user devices. (For each realistic scenario, we estimated the number of devices N the user subpopulation by taking the same proportion of out of the overall population occurred in the corresponding simulation scenario.) Reiterating from Section this scale of orchestration is feasible for real-world production FL systems, e.g. [Boawwtz et al. (2019)
25

---

Published as a conference paper ICLR 2020
D OPEN PROBLEMS
The intent of this paper is to highlight the gaps in a modeler's data inspection toolbox when undertaking private, decentralized learning, and to present what appears to be a promising solution direction involving generative models. To say that open work remains is an understatement, and there are many interesting topics to consider in this new space.
Alternative Notions of Privacy In this paper implemented user-level privacy via the mechanism of (e, 8)-differential privacy (DP), and considered acceptable privacy to be achieved via single-digit bounds on e. Other useful notions of privacy exist, e.g. Carlini al. (2018) defines concept called exposure, a more empirical measure of data disclosure that takes into account the network training process. Research like Carlini et al. (018)) Jayaaaaa & Evans (2019) shows that for many problems a large gap exists between empirical measures and DP's measure of privacy risk. While DP is tighter, empirical concepts like exposure may prove to be more practically useful for a given application; even if the e bound is larger than single- digits, exposure can indicate that data memorization is not taking place and privacy protected. Exploring how alternative concepts of privacy affect the utility of federated generative models could improve their usefulness. (This topic connects with the FedGP work in Triasteyn & Faltings (2099) exploring differential average-case privacy for federated GAN training.)
Sensitivity The GAN experiment in Section 6 demonstrated debugging when 50% of user devices have been affected. The RNN experiment in Section 5 demonstrated debugging when 10% of examples have been affected. In Appendix B we present RNN results with differing percentages of affected examples, but in general question remains: how sensitive to the presence of bug are such federated generative models? there a minimum percentage of users or examples that's required that the model learns to generate examples from that part of the distribution? This is of particular interest in the case of GANs, which are known to suffer from problems of mode collapse (Salimans et al. 2016).
composite data distribution such
Federated GAN Training We chose GANs for this paper as they had not (to our knowledge) previously been trained with FL and DP, and developing an algorithm to do so was an open question. Apart from the DP-RedAvg-GAN algorithm
present, the GAN used is relatively simplistic (see details in Appendix C.2 Exploring the space of GAN techniques to study which losses and model architectures are most amenable privatization (c.g., scaling and noising for DP) and federation would be immensely useful.
Other Classes of Generative Models Other types of generative models exist, with interesting questions on how to federate them. For example, Variational Autoencoders (VAEs) involve an encoder mapping high- dimensional data examples to lower-dimensional latent space, and a decoder mapping from latent space back to nigh-dimensiioal space. With decentralized user-private data, is a useful paradigm to have each user have a personalized, private encoder (specific to the particulars of their local data and how it maps the latent space), with a common decoder serving as synthesis engine mapping latent space to examples?
Federating Contrastive or Triplet Losses A class of ML losses involve distinguishing examples (or users) from each other. An example is the triplet loss used for FaceNet in Schrolf' et al. (2015). The loss is the sum of reward based on similarity to positive example and penalty based on similarity to negative example. For such loss calculations, a compute node requires access to both positive and negative examples.
FL protects not only the privacy of users from the coordinating server, but also the privacy of users from each other; each user's data resides in a distinct silo. Computing a triplet loss a user's device requires access to negative examples of data (i.e., from other users), which is not directly possible. DP federated generative models offer a potential solution. If one can be trained to embody a population, it can then be shipped to users' devices to serve as an engine providing synthetic negative examples for the triplet loss calculation.
26


Reviews:
[{"review_id": "SkeZlzEDqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Goals\nThe paper identifies a key challenge in a large class of real world federate learning problems where we also have to ensure user level data privacy. In these settings the modeler can not inspect the raw data samples from the user (due to privacy concerns) and hence all modeling tasks (from data wrangling to hypothesis generation to labeling to model class selection to validation) become far more challenging. The paper proposes that in these circumstances one may use a generative model that learns the data distribution using federated learning methods with provable differentiable privacy guarantees. The generative model can then produce data (unconditional, or conditional on some features or class labels) which can be inspected by the modeler without compromising user privacy. \n\nExperiments\nThe authors illustrate the approach using existing federated DP RNN learning methods, and using a slightly novel GAN learning algorithm for images (largely similar to other algorithms). They use these methods to provide two examples: 1) learning a language model from text (word sequences) where there is a bug in pre-processing steps (tokenization); 2) learning a GAN for images of handwriting on checks where there is a pre-processing bug that inverts the grayscale of images. These examples illustrate the potential for such methods to possibly be useful to modelers. While one may quibble about some details (see section below) the experimental set up is reasonable to illustrate the need and some of the challenges modelers are likely to face in the real world (\n\nEvaluation & Questions\n\nI'm really torn because I really enjoyed the paper very much overall but I have some strong concerns as well. \n\nPositives: the paper is well motivated and very well written (it is really a pleasure to read, and it is very clear about the details -- especially after they release the code it should be possible to reproduce the results too). The authors shine a spot light on a problem that is very important & widespread (eg while learning from condifential data on cell phones). The proposed solution is fairly simple, intuitive, and quite high level (lets use a generative model that creates phantom data that can be inspected)\n\nNegatives: I am not entirely sold on this being a realistic approach in the long term -- ie that some of the key problems will ever be solvable (I'm quite ok even if they are not solved now in the first paperr). The authors do a very good job of being transparent about several potential issues (see eg last paragraphs of main paper and appendix D). My biggest concerns are below:\n* the phantom samples generated from the model need to be very realistic in order to be useful. In other words, we need to have excellent, high fidelity generate models. Even to create proper hypothesis, create proper model classes, assess convergence, or assess whether the generative model is good enough one needs to be able to inspect the raw data -- which can not be done in the first place. This can not be entirely automated eliminating need for human inspection -- and the problem is much worse in generative models (which need to encode more information) than in discriminative models which need to encode less information (bits) almost by definition. Thus one has simply traded the problem of needing to inspect data to model the final algorithm (whcih could be discriminative) and has to deal with the problem of needing data to inspect the intermediate, generative model (which is also learned using federates, DP guaranteeing ways). It is not at all clear what one accomplished by doing this. \n* GANs are notoriously hard to train with mode collapse etc.  Setting hyper parameters of any generative model also needs access to original data and impacts the privacy guarantees. \n\n***NOTE added after author response***\nThe rebuttal has sufficiently address the quibbles I raised below. I'm leaving it here to allow traceability. I'm not fully convinced about the response to the main issue I raised above (ie if the generative model is not very representative, high-fidelity, then one cant know whether a potential bug discerned by inspecting its samples is an artefact of the generative model or whether it is truly a fundamental bug upstream -- and training a high quality generative model also requires one to inspect the raw data in the first place so the problem has simply been swept under the carpet). Nevertheless for a first paper on the topic I think the contributions and intuition provided here are quite valuable so I am ok leaving this for for future work. \n\n* quibble#1: theoretical DP bounds are not very tight. For example, in table 2 they may want to use realistic estimates instead of epsilon even to prove their high level point. I'm not sure I can buy their argument even on this illustrative problem as it stands. \n* quibble#2: You may want to at least make an effort to compare against the nearest possible methods in your experimental setup even if they are not a great match to the problem. I'm not intimately familiar with the recent literature but you mention Triascyn & Faltings (2019) so perhaps you could also use that and expand a bit more on the novelty here", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg96ATCtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a differentially private federated learning method to learn GAN with application to data bugging situations where privacy protection is needed. The proposed method tries to leave the data at the user-end to train the discriminators, and learn the generator at the centralised server. To support the debugging data related issues as claimed, two specific examples related to text and image modeling were presented. It is the generator which is DP-protected (as the discriminators are DP-protected) makes it possible where the generated data can hint the potential bugs. \n\nThe scenario being considered is interesting and two real examples have used to illustrate the idea. However, this paper falls short in the following ways: \n- It adopts what being proposed in McMahan et al. (2018) with some modifications to achieve the goal. The novelty is more related to the proposed application which allows debugging data issues to be possible when the data is private and decentralised.\n- The two debugging illustrations are very specific in term of the errors introduced and the ways to achieve the debugging goal. It is not sure how they can be further generalized to other types of bugs.\n- The paper is well written. However, the readers should have reasonable background on DP, GAN, federated learning, and generative models, or it will be hard to read through. Having said that, the authors do provide quite comprehensive literature review on related topics. But, then not much space is left for providing the necessary background and details for  the proposed federated learning for GAN with DP (other than referring to Algorithm 1). The experiment section is good.\n\nSpecific questions:\n- Other than the tokenisation bug and the image insertion bug, can more possible examples be described?\n- Can the examples be generalised into some methodologies? And, what are the limits? Will there be data inspection needs which cannot be achieved by this approach? What are they?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl4cLB15S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a method for using generative models to gain insight into sensitive user data, while maintaining guarantees about the privacy of that data via differential privacy (DP) techniques. This scheme takes place in the federated learning (FL) setting, where the data in question remains on a local device and only aggregate updates are sent to a centralized server. The intended application here is to use the trained generative models as a substitute for direct inspection of user data, thus providing more tools for debugging and troubleshooting deployed models in a privacy conscious manner.\n\nPros:\nGiven the growing computational power of mobile devices and the importance of privacy for large-scale deployment of machine learning, this work is a timely contribution that could augment the ML pipeline for at-scale applications dealing with sensitive data. The authors do a good job of fleshing out the intended use cases of their training scheme, and present a pair of experiments that are well-chosen for illustrating the utility of generative models when dealing with private data.\n\nCons:\nAlthough likely of practical use, the work seems to be lacking in novelty in several respects. First, the techniques developed here represent a fairly straightforward merger of DP and FL tools without much in the way of qualitatively new offerings. While the authors do develop a new GAN training scheme that works in the FL setting, this adaptation is also pretty straightforward, and mostly follows the approach laid out in [1] for training recurrent neural nets.\n\nSecondly, this paper comes in the midst of many other works aiming to integrate different combinations of generative models, privacy, and distributed training (as pointed out in the related work section). While the particular combination of techniques here differ from those in previous work, the authors don't attempt to justify why their training scheme should be preferred over these prior methods. And although their experiments are useful for understanding the general utility of generative models trained in a private and decentralized setting, they unfortunately don't permit any direct comparison with the experiments used in these previous papers.\n\nVerdict:\nFor the reasons given above, I cannot recommend acceptance of this work.\n\n[1] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang, Learning differentially private recurrent language models, ICLR 2018\n\n*** Follow-up after authors' rebuttal ***\n\nI'd like to thank the authors for their rebuttal, and for the significant addition to the paper in the form of an expanded Section 2. This material has helped me gain a bit better perspective on the use cases for their work, and convinced me of the potential for their methodology within real-world development of deep learning tools and services. In addition, this additional context helps to motivate the two experiments described here as fair representatives of actual debugging problems, and not simply issues that were hand-chosen to prove the authors' point.\n\nI still hold that the paper offers very little in the way of new conceptual or technical contributions, but in light of the potential utility of this privacy-conscious generative pipeline for the broader deep learning community, I have changed my score from a weak reject to a weak accept.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides methods for training generative models by combining federated learning techniques with differentiable privacy. The paper also provides two concrete applications for the problem of debugging models. Even though the method in the paper seems to be a standard combination of DP deep learning and federated learning, the paper is well-written and presents interesting use cases.
Decision: Accept (Poster)