Robbing the Fed:  Directly Obtaining Private Data in Federated Learning with Modified Models
Authors: Liam H Fowl
Jonas Geiping
Wojciech Czaja
Micah Goldblum
Tom Goldstein
Venue: ICLR 2022 Poster (2022)
Keywords: Privacy
Federated Learning
Gradient Inversion

Abstract: Federated learning has quickly gained popularity with its promises of increased user privacy and efficiency.  Previous works have shown that federated gradient updates contain information that can be used to approximately recover user data in some situations.  These previous attacks on user privacy have been limited in scope and do not scale to gradient updates aggregated over even a handful of  data  points,  leaving  some  to  conclude  that  data  privacy  is  still  intact  for realistic training regimes.  In this work, we introduce a new threat model based on minimal but malicious modifications of the shared model architecture which enable the server to directly obtain a verbatim copy of user data from gradient updates without solving difficult inverse problems.  Even user data aggregated over large batches – where previous methods fail to extract meaningful content – can be reconstructed by these minimally modified models.


Published as a conference paper at ICLR 2022
ROBBING THE FED: DIRECTLY OBTAINING PRIVATE DATA IN FEDERATED LEARNING WITH MODIFIED MODELS
Liam Fowl'
Jonas Geiping"
Department of Mathematics University of Maryland
Department of Computer Science University of Maryland
Wojtek Czaja
Mícah Goldblum Center for Data Science
Tom Goldstein
Department of Mathematics
Department of Computer Science University of Maryland
University of Maryland
New York University
ABSTRACT
Federated learning has quickly gained popularity with its promises of increased user privacy and efficiency. Previous works have shown that federated gradient updates contain information that can be used to approximately recover user data in some situations. These previous attacks on user privacy have been limited scope and do not scale to gradient updates aggregated over even handful of data points, leaving some to conclude that data privacy is still intact for realistic training regimes. In this work, we introduce malicious modifications of the shared model architecture which enable the server directly obtain a verbatim copy of user data from gradient updates without solving difficult inverse problems. Even user data aggregated over large batches where previous methods fail extract meaningful content can be reconstructed by these minimally modified models.
threat model based minimal but
INTRODUCTION
Federated learning KKnnčný et al., 2015). also known as collaborative learning Shokri & Shmatikov 2015), is mechanism for training machine learning models in distributed fashion on multiple user devices. In the simplest setting, a central server sends out model states to a group of users, who compute an update the model based on their local data. These updates are then returned to the server, aggregated, and used to train the model. Over multiple rounds, this protocol can train a machine learning model, distributed over all users, without exchanging local data only model updates are exchanged. Two central goals of federated learning are to improve training effi- ciency by decreasing communication overhead and to side-step issues of user-level privacy and data access rights that have become focus of public attention in recent years (Veale et al. 2018).
Accordingly, many organizations, ranging from large tech companies (MeMahan Ramage 2017) to medical institutions with especially strict privacy laws, such as hospitals (Jochems et al. 2016), have utilized federated learning to train machine learning models. However, in practice, data pri- vacy is not guaranteed in general, but is dependent and design choices specific to each federated learning system. In this work, we focus on the user perspective of privacy, and study federated learning systems in which the central server is not able to directly view user data.
large number of interdependent settings
The key privacy concern for users is whether model updates reveal much about the data which they were calculated. Although Kairouz et al. (202T) discuss that "model updates are more focused on the learning task at hand than the raw data (i.e. they contain strictly no additional information about the user, and typically significantly less, compared to the raw data)", scenarios can be constructed in which the model updates themselves can be inverted to recover their input user
Authors contrihuted equally. Order chosen alphahetically.

---

Published as a conference paper at ICLR 2022
information (Wang al., 2018 Melis al. 2018). Simple knowledge of the shared model state and model update can be sufficient for such an attack (Zhu et al. 2019 Geiping et al. 2020). These inversion attacks are particularly fruitful if a user's model update is based only a small batch. Accordingly, a strong defense against these attacks is aggregation. The user only reports model updates aggregated over significant number of local data points, and data from multiple users can be combined with secure aggregation protocols (Bonawitz et al. (017) before being passed to the server. This ability aggregate user updates while maintaining their utility is thought to be the main source of security in federated learning. Averaging raw local data in similar amounts would make it unusable for training, but model updates can be effectively aggregated.
single data point or
Previous inversion attacks typically focus on a threat model in which the server (server here is a stand-in for any party with root access to the server or its incoming and outgoing communication) is interested in uncovering user information by examining updates, but without modifying the feder- ated learning protocol, a behavior also referred to as coneestbut-ccure 2009). In our case, where the party intending to recover user data is the server, this "honest" scenario appears contrived, as the server can modify its behavior to obtain private information. In this work, are thus interested in explicitly malicious servers that may modify the model architecture and model parameters sent to the user.
or semi-honest (Goldreich)
We focus scenarios in which agent obtains data without making suspicious changes to the client code or learning behavior. One scenario which enables this threat model involves recently in- troduced APIs that allow organizations to train their models using established federated learning protocols (Cason 2020). In this environment, a malicious API participant can change their model's architecture and parameters but cannot force unsuspecting edge devices send user data directly.
We introduce minimal changes to model architectures that enable servers to breach user privacy, even in the face of large aggregations that have been previously deemed secure. These changes induce structured pattern in the model update, where parts of the update contain information only about a fixed subset of data points. The constituent data points can then be recovered exactly, while evading existing aggregation defenses. For architectures that already contain large linear layers, the attack even works directly, modifying only the parameters of these layers.
LIMITATIONS OF EXISTING ATTACK STRATEGIES
A range of possible attacks against privacy in federated learning have been proposed in recent liter- ature. In the simplest case of analytic attacks, Phhng et al. (2017)) were among the first to discuss that the input to a learnable affine function can be directly computed from the gradient of its weights and bias, and additional analysis of this case can be found in Qian & Hansen (2020); Fan et al.) (2020), and in Section 3.2 However, analytic recovery of this kind only succeeds for a single data point. For multiple data points, only the average of their inputs can be recovered, leading the attack to fail in most realistic scenarios.
Recursive attacks as proposed in Zhu & Blaschko (021)) can extend analytic attacks to models with more than only linear layers- construction also mentioned in Fan et al. 2020). However, these attacks still recover only the average of inputs in the best case. Improvements in Pan et al. (2020) transform linear layers with ReLU activations into systems of linear equations that allow for degree of recovery for batched inputs to these linear layers, although preceding convolutional layers still have to be deconvolved by recursion or numerical inversion techniques.
Surprisingly, optimization-busel attacks turn out to be highly effective in inverting model updates. Wang et al.. (2018) propose the direct recovery of input information in a setting where the users' model update is the model parameter gradient averaged over local data. In a supervised learning setting, we define this update by g and the loss function over this model by L with model parameters 0 and data points (z, y) E [0, 1]" x R". The server can then attempt recovery by solving the gradient matching problem of
min |V g/l
[=
and solve this optimization objective using first-order methods or any nonlinear equation solver. Subsequent work in Zhu et al. (2019): Zhao et al. (2020) and Wainakh et al. (2021) proposes lutions that also handle recovery of targets y and variants of this objective are solved for example
in Geiping al. (2020) with cosine similarity and improved optimization and in Jeon et al. (2021)
2

---

Published as a conference paper at ICLR 2022
with additional generative image priors. Reconstruction of input images can be further boosted by
additional regularizers as in Yin et al. (2021) and Qian et al. (2021).
Most attacks in the literature focus the described fedSGD setting (Konečný et al. 2015) in which the users return gradient information to the server, but numerical attacks can also be performed against local updates with multiple local steps Geiping et al. (2020), for example against fedAVG (McMahan et al. 2017). In this work, we will discuss both update schemes, noting that gradient aggregation in "time", " with multiple local update steps, not fundamentally more secure than ag- gregation over multiple data points. Previous attacks also focus significantly on learning scenarios where the user data comprised of images. This is an advantage the attacker, given that im- age data is highly structured, and a multitude of image priors are known and can be employed to improve reconstruction. In contrast, data types with weaker structure, such as tabular data, do not lend themselves to regularization based on strong priors, and we will show that our approach, on the other hand, does not rely on such tricks, and is therefore more data-agnostic.
The central limitation of these attack mechanisms is the degradation of attack success when user data is aggregated over even moderately large batches of data (either by the user themselves or by secure aggregation). State-of-the-arr attacks such as Yin et al. (2021) recover only 28% of the user data (given a charitable measure of recovery) on a batch size of 48 for ResNet-50 (He et al. 2015) model on ImageNet (ILSVRC2012 (Russakovsky et al., 2015)) with unlikely label collisions. The rate of images that can be successfully recovered drops drastically with increased batch sizes. Even without label collisions, large networks such as a ResNet-32-10 (Zagoruyko & Komodakis, 2016) leak only few samples for a batch size of 128 in Geiping et al. (2020). These attacks further reconstruct only approximations to the actual user data which fail recover parts of the user data or replace it with likely but unrelated information in the case of strong image priors.
Further, although all of the previous works nominally operate under an honest-but-en but-curious server model, they do often contain model adaptations on which reconstruction works especially well. such as large vision models with large gradient vectors, models with many features (Wang et al. 2018
Zhu & Blaschko 2021), special activation functions (Zhu et al. 2019: Zhu & Blaschko, 2021 wide models (Geiping et al.) 2020), or models trained with representation learning (Yin al. [0021] Chen et al., 2020). These may be seen malicious models with architectural choices that breach
user privacy. the same vein, we ask, what the worst-case (but small) modification that can be applied to neural network to break privacy?
MODEL MODIFICATIONS
In this section, we detail an example of a small model modification that has a major effect on user privacy, even allowing for the direct recovery of verbatim user data from model updates.
3.1 THREAT MODEL
We define two parties: the server S and the users 4. The server could be tech company, third party app using a federated learning framework on mobile platform, or an organization like hospital. The server S defines a model architecture and distributes parameters for this architecture to the users, who compute local updates and return them to the server. The server cannot deviate from standard federated learning protocol in ways beyond changes to model architecture (within limits imposed by common MI. frameworks) and model parameters. We measure the strength of malicious modification of the architecture using the number of additional parameters inserted into the model. While it is clear that models with more parameters can leak more information, we will see that clever attacks can have a disproportionate effect on attack success, compared to more benign increases in parameter count, such as when model width is increased.
3.2 A SIMPLE EXAMPLE
To motivate the introduction malicious modifications, we start with the simple case of fully connected layer. A forward pass on this layer is written as y W where W is weight matrix, b is bias, and x is the layer's input. As seen in Phong et al. (2017a); Qian Hansen (2020); Fan et al. (2020), when the parameters of the network are updated according some objective 2, the 30
3

---

Published as a conference paper at ICLR 2022
row of the update to W:
ac
0C V Vw.y
V Ww L Oyř
Oy' I,
where we use the shorthand C = [(x; W, b). Similarly,
0L Oy Ob Qy Ob
ac
dL Oy²
So as long as there exists some index with ac
+ 0, the single input x is recovered perfectly as:
20
dc
x Ww
(2)
where 2 denotes entry-wise division.
However, for batched input x. all derivatives are summed over the batch dimension and the same computation can only recover Ez= W. @ Ei=1 ac, from each where Σi= 8C. # O, abg
"B6""
which is merely proportional to linear combination of the xz's. However, data points IG only appear in the average 8C. is non-zero, a phenomenon also discussed in Sun et al. (2021). L has sparse gradient, e.g. in a multinomial logistic regression, then this structured gradient weakens the notion of averaging: Let x be batch of data with unique labels 1 , In this setting 0c, By
for all * t, so that each row actually recovers
Oc, Oy
OC. dyi
ac.
0L. dyt
FI
Oy:
xz Ø
For batch of n data points with unique labels, could thus recover all data points exactly for this multinomial logistic regression. We visualize this in Appendix Fig. [or ImageNet data Rus- sakovsky al. (2015) (image classification, 1000 classes), where could technically recover up to 1000 unique data points in the optimal case. However, this setup is impractical and suffers from several significant problems:
Averaging: Multiple image reconstruction as described above is only possible in the linear setting, and with a logistic regression loss, a loss that depends on sparse logits is used. Even in this restrictive setting, reconstruction fails soon as labels update (which is the default case and outside the control of the server), especially the accumulation size user update larger than the underlying label space the data. In this case, the server reconstructs the average of repeated classes, In Appendix Fig. [I we see that in the worst-case scenario where all data points fall into the same class, each piece of user data contributes to the gradient equally, resulting in mashup reconstruction that leaks little private information.
repeated in
Integration: As stated above, the naive reconstruction is only guaranteed to work only the linear (single-layer) model is standalone model, and not within larger network. If the naive linear model was placed before another network, like a ResNet- 18, then gradi- ent entries for the linear layer contain elements averaged over all labels, as the combined network ostensibly depends on each output of the linear layer.
Scalability: on an industrial scale dataset like ImageNet, the naive logistic regression model would require > 150M parameters to retrieve an image from each label, which is course far from any practical application.
3.3 IMPRINTING USER INFORMATION INTO MODEL UPDATES
Nonetheless, the perfect reconstruction afforded by the linear model remains an attractive feature. To this end, we introduce the imprint module class of modifications which overcome the previously de- scribed issues, while maintaining the superior reconstruction abilities of an analytic reconstruction described above. Further, the imprint module be constructed from combination of com- monly used architectural features with maliciously modified parameters that can create structured gradient entries for large volumes of data.
4

---

Published as a conference paper at ICLR 2022
The imprint module can be constructed with single linear layer (with bias), together with a ReLU activation. Formally, let {-,}²= = X E R"xm be a batch of size n of user data, then a malicious server can define imprint module whose forward pass (on single datapoint, 2:) looks like
((x) f(W,x - b.),
where f is a standard ReLU nonlinearity. The crux of the imprint module lies in the construction of W. E R*xm and b RR We denote the 7th row (or channel) of W. and the ith entry of b. W and 6 respectively. We then construct ( so that
h(x),
where h is any linear function of the data where the server can estimate the distribution of values (h(x))>> of this function on the user data distribution. For example, if the user data are images, could be average brightness, in which case W is simply the row vector with entries identically equal to m. In order to define the entries of the bias vector, assume that the server knows some information about the cumulative density function (CDF), assumed to be continuous for the quantity measured by h. Note this attack does not require the server to know the full distribution of user data, but rather can estimate the distribution of some scalar quantity associated with the user data (a much easier task). In Appendix Fig. 5 we see this can be quite easy for server, and can even be done with a small amount of surrogate data. We also stress that the choice of h here is not important to our method. For the purpose of explanation, we will assume that the quantity measured by is distributed normally, with = 0, a = Then, the biases of the imprint module are determined by
b - + ( k
Cc.
where 6-1 is the inverse of the standard Gaussian CDF In plain language, we first measure some quantity, like brightness, with the matrix W.. We duplicate this measurement along the k channels (rows) of W,, In the meantime, we create k "bins" for the data corresponding to intervals of equal mass according the CDF of h. Then, the measurement for given datapoint will land somewhere in the distribution of h.
For example, consider the case when the brightness of some image z, lands between two values: & h(xx)) I Ct+1- and no other image in the same batch has brightness in this range. In this situation, we say that 24 alone activates bin l. Then, if the image xe passed through the imprint module, we have
( (C 0b!
Oc
(VwiC- Beu--Vwwii W @
06!+1
Z2
Zi.
Zt
(4)
where images {xr, } are images from the batch with brightness > qt- That is, the difference in successive rows 1 + 1 of the gradient entry for W. correspond to all elements with brightness a S h(x) S ct+1 (in this case, assumed to be just 24 see Appendix B for remark). This because all of xe U (x.,) } activate the non-lincarity for layer I, since all these images have brightness > et, however, only images {x+} } have brightness 2 C141, so only these images activate the non-linearity for layer + 1. An interesting biproduct of this setup is that it would be difficult even for hand inspection of the parameters to reveal the inclusion of this module as the server could easily permute the bins, and add random rows to W. which do not correspond to actual bins and only contribute to model performance. The gradient does not directly contain user data, so that the leak is also difficult to find by analyzing the gradient data and checking for matches with user data therein.
How successful will this attack be? Recall the parameter defined the construction of the imprint module. This corresponds the number of bins the malicious server can create to reconstruct user data. If a batch of data is passed through the imprint module, depending on the batch size n used to calculate the update sent to the server, and number of bins, k. the server can expect several bins to activate for only one datapoint. And the corresponding entries of the gradient vector can be appropriately combined, and inverted easily. The following result quantifies the user vulnerability in terms of the number of imprint bins, k and the amount of data, n., averaged in given update.
Proposition 1. If server knows the CDF (assumed to be continuous) of some quantity associated with user data that can be measured with linear function h R.m - R. then for batch of size and a number of imprint bins k >> n > 2, by using an appropriate combination of linear layer and ReL.U activation, the server can expect exactly recover
5

---

Published as a conference paper at ICLR 2022
j-1 ( --11---
Σ
r(n,k) k)
k+n
samples of user data (where the data is in RT) perfectly. Note: r(n, k) is correction term (see proof for full expansion).
Proof. See Appendix A.I
This can be thought of as a lower bound on privacy breaches since, often, identifiable information can be extracted from mixture of images. Note that increasing the expected number of per- fectly reconstructed images requires increasing the number of imprint bins, which in turn requires increasing the number of channels of the matrix W. and thus the number of parameters. Thus, to visualize the result above in terms of the server-side hyperparameter, k, we plot the expected pro- portion of data recovered as a function of number bins Fig. Ta). Note that this inversion analytic, and significantly more efficient and realistic than optimization based methods which often require tens of thousands of update steps.
Propertion Recantrueed
03
Prateablleyg
0.00
0.
0.37
100
150
200
250
330
350
430
420
8.804
3.01 08012 0.014 80005
Number of Bllss
Masi captured in Oue Shot Bin
Figure 1: Left (a): Expected proportion of a batch of 64 images perfectly recovered as a function of number of bins added via imprint block in front of ResNet-18 18 ImageNet. With only 156 bins, an attacker can expect to recover over 50% of batch of user images perfectly. Right (b): Probability of successful "one-shot" attack on batch of 4096 images as a function of mass captured in the one-shot bin. An attacker can optimize their bin size given an expected batch size.
The imprint module as described can be inserted in any position in any neural network which re- ceives non-zero gradient signal. To recover the input feature dimension of the module, second linear layer can be appended, or- if no additional parameters are interest the sum of the outputs of the imprint module be added the next layer in the network. The binning behavior of the imprint module is independent of the structure of succeeding layers as long as any gradient signal is propagated. This flexibility allows for wide trade-offs between inconspicuousness and effectiveness of the imprint module. The module can be placed in later stages of model whereas an early linear layer might be suspicious observers (now that they have seen this trick), but depending on the
Figure 2: Left: Ground truth batch of 64 user images. Right: Analytic reconstruction for an imprint model with 128 bins in front of a ResNet-18. Gray reconstructions denote bins in which no data point falls.)

---

Published as a conference paper at ICLR 2022
data modality, early linear layers can be feature of architecture anyway, in which case there is even no model modification necessary. only parameter changes. The parameter alterations necessary to trigger this vulnerability "dormant", functioning and training as a normal linear layer initialized with random activations, as long the server desires. At any point, party with access to the server can send parameter updates containing W 6* and trigger the attack.
furthermore be hidden from inspection until use. The layer can lay
4 EXPERIMENTS
In the following section, we provide empirical examples of imprint modules. In all experiments we evaluate ImageNet examples to relate to previous work (Yin et al.] 202I). but stress that the approach is entirely data agnostic. Given an aggregated gradient update, we always reconstruct as discussed in Section g When, in the case of imprecision due to noise, more candidate data points are extracted than the expected batch size, only the candidates with highest gradient mean per row in W are selected and the rest discarded. For analysis, we then use ground truth information to order all data points in their original order (as much as possible) and measure PSNR scores as well as Image Identifiabilityy Precision (IIP) scores as described in Yin et al. (202T) For IIP we search for nearest-neige neigrbbss in pixel space to evaluate a model-indeeenn boodel-indeeenden distance a more strict metric compared to IIP as used in Yin et al. (2021) All computations run in single floating point precision.
4.1 FULL BATCH RECOVERY
We begin with a straightforward and realistic case selling point for our method user gradient updates aggregated over a batch of 64 ImageNet images. We modify a ResNet-18 to include an imprint module with 128 bins in front. This relatively vanilla setup presents is major stumbling block for optimization based techniques. For example, for a much smaller batch size of 8, the prior art in optimization based batched reconstruction achieves only 12.93 average PSNR YYin et al.) 2021]. We do of course operate in different threat models (although Yin et al. (2021) also uses non-obvious parameter modifications). However, our imprint method is successfully able to recover almost perfect reconstructions of a majority of user data (see Fig. 2 and achieves an average PSNR of 75.75. We further stress that a batch size of 64 by no means a limitation of the method. bins, and hence additional rows in W are added proportionally to the expected batch size, then recovery of significant proportions (see Fig. D of batches of arbitrary size - albeit with the incurred cost in additional parameters for each row is possible. Note that concurrent work, Boenisch et al. (202T) also proposes a method to recover user data from large-batch updates by malicious modifications operating in the same setting/threat model our work However, in a direct comparison, we find that our imprint module presents a much more significant threat to privacy compared to Boenisch et al. (021)) see Appendix Fig. 21
4.2 PRIVACY BREACHES IN INDUSTRIAL-SIZEE BATCHES ONE- ATTACKS
A breach in privacy occur if even single piece of user data is compromised. Unfortunately, there is a threatening modification of the imprint module for exactly such an attack. If a server has access to enough users, then it becomes feasible for the server to start fishing for private data among all updates, and attempt to recover a single data point from each incoming batch of data. Attacks of this nature require only as many additional parameters as twice the size of single piece of targeted user data, as only two bins are needed. For this, k bins are constructed initially, and then all bins are "fused" create 2 final bins: a one-shot bin containing mass n n/k, and the other containing the remaining mass (n 1)/k. For perspective, for a ResNet- 18 on ImageNet, this would require only an additional 1% of parameters this tiny compared to potentially massive increases in the number of parameters when no bins were fused, which could raise suspicion under inspection. Further, based on Proposition [ it is always possible to select an optimal bin size, so that a data point is leaked on average once every four batches of incoming data, matter how large the batch size. We demonstrate this statistical property by recovering single image from an aggregated batch of 2 16, 384 ImageNet images (see Fig. B). Even though the gradient updates are averaged over a vast number of data points, there can be no perfect privacy, and one data point is leaked in its entirety by only minor model modification.
7

---

Published as a conference paper at ICLR 2022
Figure 3: Left (a): A true user image from class "minibus". Right (b): The reconstructed image from class "minibus" captured via our one-shot attack from averages aggregated over 16,384 data- points. The PSNR is 161.36, i.e. verbatim copy machine precision. This user could potentially be identified via their recovered license plate, which was blanked out (by us!) to preserve privacy.
4.3 VARIANTS
Flexible placement The imprint module does not depend on its placement within a given model. No matter its position in network, the incoming input features will be leaked to the server. Fur- thermore, the server can also change the parameters of preceding layers to represent (ncar)-identity mappings, allowing for the recovery of raw input data from inconspicuous positions deep in net- work. For a convolutional network, we show off this variant in Fig. 14 for a ResNet-18. Here, the model parameters are manipulated contain identity maps up to the location of the imprint module, while the downsampling operations remain. Even these later layers leak enough information that their inputs can be upsampled to breach privacy of the inputs. We remark that we show a simplified version of this attack here where the first three channels in each layer act as an identity, and all other channels are zero, but the model can also be modified provide off-set pixels in all other channels, effectively increasing the spatial resolution in any layer proportionally with the number of channels.
Multiple local updates: In several federated learning protocols, such fedAVG, users take several local update steps on data before sending model updates to the server Konečný et al. (013). The imprint module is threatening when large amount of user data is used for a single update step. In this case, the only variable that matters is amount of total data used in the update. That is to say, 10 users sending updates on 100 datapoints each equivalent (recovery-wise) to a single user sending update calculated on 1000 datapoints. However, when multiple steps taken, inverting gradients from pairwise differences (as in Eq. 4 becomes more difficult, as entries in W shift with local updates. However, imprint variant that produces sparse gradients per data point is a threat to such federated averaging. Defining a forward pass this new variant as M'(x) as M' M'(x) g(W, æ b.) where the non-linearity g is thresholding function:
t 0
s(()
l 05151
1 1st
Note this non-linearity can be simply constructed with combination of two ReLUs, or with an implementation of Hardtanh. We now define Wi as: (Wi, h(z) function of the data described before, and the biases are defined 6,
where h is the linear
6
C₁ oi
where 6, =
This setup creates sparse bins where inversion is directly possible (without taking pairwise differ- ences) in gradient entries, at the cost an additional activation layer. Analyzing local update step with W as the tth row of W. at update step j, reveals that
W wiih-1 OC
dai.j 2
where zy denotes the data from the previous batch that activated bin i, and ad denotes the 7th acti- vation at step and is the local learning rate. Note that if either linear layer, convolutional layer follows this imprint module, then Ja
dc does not depend the scale of Wis. Therefore,
8

---

Published as a conference paper at ICLR 2022
100
130
I1P (gécuf-lumed)
-0
star
Miiddt Mocal
+ Barch size 14
HIP
Late
Batch sice 256
of
100
1000
100
Nomber bins
Number of bins
Figure 4: Left: Identification Success vs bin size. Right: Identification Success (via IIP score) vs. bin size and position in ResNet-18 model. We find that the attack is stable over range of batch sizes and positions in a model.
simple way iccrease the effectiveness of the new imprint module, M' in the fedAVG case is to scale the linear function associated to the rows of W. i.e. M(x)) co h(x).. This "flattens" the distribution of values, and increases the relative size of b compared to the gradient update ac Bara which prevents the bins from shifting too significantly during local updates. As bin shift goes to O, recover the situation where the only variable that matters is the total data used in an update.
With this modification, reconstruction quality remains similar to the fedSGD setting. For example, splitting the batch of 64 ImageNet images up into 8 local updates with learning rate le yields an IIP score of 70.31%, due to minor image duplications where images hit multiple shifted bins, which we visualize in Appendix Fig. 19
Yet another advantage to our imprint module over existing optimization based gradient inversion techniques is the flexibility in data domain. Other techniques have demon- strated some success in the image domain by leveraging strong regularizers including total variation (TV), image registration. matching batch norm statistics, and DeepInversion priors (Geiping al.) 2020 Yin et al. 2021). Such strong regularizers do not always exist in other domains of interest, such as text tabular data. The discussed imprint module, however, focus our experiments
Other data modalities:
dlta-agnostic,, and while we the image domain, nowhere do we any assumptions unique to vision. In fact, linear layers often appear in language models, and tabular data models cases which the attacker only needs to modify parameters of existing model breach user privacy (Vaswani et al. 2017 Somepalli et al. 2021) without architecture modifications.
5 POTENTIAL DEFENSE AND MITIGATION STRATEGIES
If aggregation is the only source of security a FL system, then the proposed attack breaks it, uncovering samples of private data from arbitrarily large batches, especially via the One-shot mech- anism. In light of this attack, the effectiveness of secure aggregation is reduced to only secure shuffing (Kairouz et al. 2021): : When private data is uncovered via the imprint module, based on data that has been securely aggregated, then the data is breached, but is not directly connected to any specific user (aside from possible revealing information in the data itself). A mitigation strategy for users that does not require coordination (or consent) of a central server is to employ local differ- ential privacy (Dwork & Roth 2013). Adding sufficient gradient noise can be defense against this attack as the division in Eq. (2) leads to potentially unbounded errors in the scale of the data. Yet, in practice, privacy often still broken even if the correct scale cannot be determined, so that the amount of noise that has to be added is large. In Appendix Fig. 20, even with o = 0.01, private data is visibly leaked. Additional discussion on defenses can be found in Appendix A.7
CONCLUSIONS
Federated learning offers a promising avenue for training models in a distributed fashion. However, the use of federated learning, even with large scale averaging, does not guarantee user privacy. Using common and inconspicuous machine learning modules, a malicious server can breach user privacy by sending minimally modified models and parameters in federated setup. We hope that construct- ing these examples clarifies current limitations and informs discussions on upcoming applications, especially concerning API design.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
In this work, we uncover an attack on federated learning that has the potential to compromise user privacy. While this method has the potential to be used for malicious purposes, the fundamental purpose of this research is to inform the community about the state of privacy in federated learning, and to help users and technical experts better understand the limitations of federated learning for the purpose of preserving user privacy.
REPRODUCIBILITY STATEMENT
We provide additional technical details and the proof Proposition [ in the appendix. Further, we provide open-source implementations of all attacks investigated this work. The repository at https://gi htw.hhoow.oo...ooooommccee http://hfwwwobbbbbbbo ng t.ee_ fed shows minimalistic implementation of the principles this attack and the repository https: /githbb........ com/ Jona sGe ping/ breaching embeds the attack in larger framework of privacy attacks against federated learning. The experiments in this work require no GPU resources and can cheaply evaluated on almost any machine with sufficient RAM.
ACKNOWLEDGEMENTS
This work supported by DARPA GARD, the Office and Naval Research, and the National Science Foundation Division of Mathematical Sciences. Addition support was provided by the Sloan Foundation, JP Morgan Chase, and Capital One Bank.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:21 2021.
Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sar- var Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical Secure Aggregation for Privacy Preserving Machine Learning. Technical Report 281, 2017. URL cuttp://eprit iiarritttacc....... iaar.
org/2017/28)
Patrick Web
Cason. and Mobile Devices,
Announcing 4 New Libraries for Federated Learning on 2020.
https://blog: http:///bloo...mmimmm.. opwnminedee..eoeeooeeeepe.eoemme org/ announcing new libraries for fl on-on web -ad-ann and-mobile/ mobile/
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning, vrXiv:2003.04297 [cs]. March 2020. URL http:tttt////it org/ abs/ 2003. 04297
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, (1:4):211-407) 2013. ISSN 1551-305X, 1551- 3068. doi: 10. 10.1561/0400000044..
Lixin Fan, Kam Woh Ng, Ce Ju, Tianyu Zhang, Chang Liu, Chee Seng Chan, and Qiang Yang. Rethinking Privacy Preserving Deep Learning: How to Evaluate and Thwart Privacy Attacks. In Federated Learning: Privacy and Incentive, Lecture Notes in Computer Science, pp. 32-50. Springer International Publishing, Cham, 2020. ISBN 978-3-030-63076-8. doi: 10.1007/978-3 3-030-63006-----
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Invert- ing Gradients How easy it to break privacy in federated learning? In Advances in Neural Information Processing Systems, volume 33. December 2020. cttlaece//ppeeee hpc//ppprr.e./ psles//paer/222//haahh Awstsaabbbtt..a.aaeeeeeaaaaaaa.e html
URL httpps://proceeedngs...eee (
Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth or backpropaganda? An empirical investigation deep learning theory. Eighth International Conference Learning Representations (ICLR 2020, Oral Presentation), April 2020. URL
huttps://icl: Ttttps//AcLE..cc/vittaaa cc/virtuall.l................ 2020 20120/poste: SwxyIgHEvr.htm.
Oded Goldreich. Foundations of Cryptography: Volume 2, Basic Applications. Cambridge Univer- sity Press, Cambridge, 1st edition edition, September 2009. ISBN 978-0-521-11991-7,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs], December 2015. URL http://arxiv.on abs/
Jinggang Huang and David Mumford. Statisties of natural images and models. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (Cat. No PR00149), volume 1, pp. 541-547 Vol. 1, June 1999. doi: 19.1199/CVPR.1999.888899
Jinwoo Jeon, Jacchang Kim, Kangwook Lee, Sewoong Oh, and Jungseul Ok. Gradient Inversion with Generative Image Prior. In International Workshop on Federated Learning for User Privacy and Data Confidentiality, pp. 12, 2021.
Arthur Jochems, Timo M. Deist, Johan van Soest, Michael Eble, Paul Bulens, Philippe Coucke, Wim Dries, Philippe Lambin, and Andre Dekker. Distributed learning: Developing a predictive model based data from multiple hospitals without data leaving the hospital A real life proof of concept. Radiotherapy and Oncology, 121(3):459444 467, December 2016. ISSN 0167-8140, 1879-0887. doi: 10.101 r2.0016/.radoone2211.10..

---

Published as a conference paper at ICLR 2022
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D' D'Oliveira, Hubert Eichner, Salim El Rouayheb. David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, 7aid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateck Mittal, Mchryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi. Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong. Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and Open Problems Federated Learning. arXiv:1912.04977 [cs, stat], March 2021. URL
Joshi,
http::/ .rg. abs /tdlb//99911 19912.111..0 04977
Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated Optimization: Optimization Beyond the Datacenter. arXiv:1511.0355 [cs, math], November 2015. URL
http:://rxii: org/ habbs//511.03577
E.Y. Lam and J.W. Goodman. A mathematical analysis of the DCT coefficient distributions for im- ages. IEEE Transactions on Image Processing, 160):1661-1666, October 2000. ISSN 10577149, doi: 10.1 10911088899777.
Brendan McMahan and Daniel Ramage. Federated Learning: Collaborative Machine Learning with- out Centralized Training Data, April 2017. URL http: http:/al..oogllbblggggoo cwm.cmmm//////// com/2017 2017/ html
Df/federatee--eammaag-o-
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--ffiiei Learning of Deep Networks from Decentralized Data. arXiv:1602.05629 [cs], February 2017. URL httt::t/ta/xii/t//t
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unin- tended Feature Leakage in Collaborative Learning. arXiv: 04049 [es], November 2018.
URL httphhttp///xi....o.m org. abs 1805. 04049
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture net fottt/oorumm11dd
Models. November 2016.
Xudong Pan, Mi Zhang. Yifan Yan, Jiaming Zhu, and Min Yang. Theory-Oriented Deep Leakage from Gradients via Linear Equation Solver. arXiv:2010.1333 [cs, stat], October 2020. URL http: org/ abs/ 13356
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentia- tion in PyTorch. htw.:nneeeoommmm2222eeeeoot
NIPS 2017 Autodiff Workshop, Long Beach, CA, 2017. URL https:
//openzeview.nete
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang. and Shiho Moriai. Privacy- Preserving Deep Learning via Additively Homomorphic Encryption. Technical Report 715, 2017a. /ttp::t:://epriin iacr. www.rii..rrg........... wttg//11//2//1/
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang. and Shiho Moriai. Privacy- Preserving Deep Learning: Revisited and Enhanced. In Applications and Techniques in Informa- tion Security, Communications in Computer and Information Science, pp. 100-110, Singapore, 2017b. Springer. ISBN 978-981-10-54211 doi: (00007977-981-1111 181--005421--19)
Jia Qian and Lars Kai Hansen. What can we learn from gradients? September 2020. URL https: //openreview. net. wwr.m/mrmmmmmmmmmmmmmmmmmm wdw.mddddddgddddddddmme
Jia Qian, Hiba Nassar, and Lars Kai Hansen. Minimal conditions analysis of gradient-basce re- construction in Federated Learning. arXiv:2010.15718 [cs, eess], March 2021. URL http: / www.iii.rg/aa////ee...
15718
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Leamers, pp. 24, 2019.
12

---

Published as a conference paper at ICLR 2022
Daniel L. Ruderman. The statistics of natural images. 5(4):517-548, January 1994. ISSN 0954- 898X. doi: 10. 10.0088//94--00 898X/5/4/006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision,
115(3):111-522 December 2015. ISSN 1573-1405. doi: 10.1007/s1 22126661----16---1
Reza Shokri and Vitaly Shmatikov. Perracc-reesrving Deep Learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security - CCS '15, pp. 1310-1321, Denver, Colorado, USA, 2015. ACM Press. ISBN 978- 10311---00--822-55 doi: 10. 10.1145/2810133 2813687.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint www.ixi206.0014440 342, 2021.
Jingwei Sun, Ang Li, Binghui Wang. Huanrui Yang. Hai Li, and Yiran Chen. Soteria: Provable
Defense Against Privacy Leakage in Federated Learning From Representation Perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. htttpps://opeaacessssss thecwh www.ww.cww//ooeeeeeeeeeetoeeeeee 9311-9319, 2021. URL https:
html/ Soteria Provaaaereaeeeeeee Beeevaaaeeeeeeeseeeeeeeeee Defense Cerainst__Pivveeee Leakage__eaaa__a__ in_
Frrm__roo________ 2021 paper.pppp............ html.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. 170.v:1706 [cs], De- cember 2017. http:ttttaxyi....///// wttpxiiiwrrgaa////ooeo 1706. 1111/0366660222222
Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: Model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematicul, Physical and Engineering Sciences, 376(2133):201 2008003, November 2018. doi: 10.1098/ssta. 2018.0083.
Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zim- mer, Tim Grube, Kristian Kersting, and Max Mühlhäuser. User Label Leakage from Gradients in Federated Learning. arXiv:2105.0995 [cs], June 2021. URL http: /ttppp///axivv..o wwtpiib.rrg/a... 21 09369
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera Ar- cas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mchryar Mohri. Hang Qi. Sashank Reddi, Peter Richtarik. Karan Singhal. Virginia Smith, Mahdi Soltanolkotabi. Weikang Song. Ananda Theertha Suresh, Sebas- tian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wen- nan Zhu. A Field Guide to Federated Optimization. arXiv:2107.06917 [cs], July 2021. URL fctp::/aaxxle org/ abtp:/ab//20 /210. 06917
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Be- yond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning. [cs], December 2018. Utt
abs/ 1 1812. 00535
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Paylo Molchanov. See Through Gradients: Image Batch Recovery via GradInversion. In Proceedings of the
IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021. URL. https: http::s///ppeaaccesee
com/ cctpc///ooteee///////////ttt Through_ Gradients Image Reccccereeevvrrycce svieerryyvviiiaeeeeeeeeeeeeeeeee GradInversion CVPR_
See.
2021 1acl1_apper.o html
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. arXiv:1605.0714 [cs], May 2016. http:: www./org/////e/eee///e.o 0001///000000001066666
13

---

Published as a conference paper at ICLR 2022
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, Pp. 586-
595, 2018. URL ott html Zhang_The
wwwphhhwwwcccccoooooooooecc cetomm/ontent_yyye 2ttp:/teeee_eprr200110111-1 2018/html
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. iDLG: Improved Deep Leakage from Gradients.
arXiv:2001.02610 [cs, stat], January 2020. URL http:p///aaxx....o//////2001..1.
Junyi Zhu and Matthew Blaschko. arXiv:2010.07733 [cs], March 2021. URL http:://axxiv..org/bb/////
R-GAP: Recursive Gradient Attack on Privacy.
007033333
L Ligeng Zhu, Zhijian Liu, and Song Han. Deep Leakage from Gradients. In Advances in Neural In- formation Processing Systems 32, pp. 14774-147888 Curran Associates, Inc., 2019. URLhttp:
papers. nips. wttpc/ppppe/9111eeeeeeeeeee leakage from gradients.pdr
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF OF PROPOSITION
Proposition. the server knows the CDF (assumed be continuous) of some quantity associated with user data that can be measured with linear function h R - R. then for a batch size of n, and a number of imprint bins k > > 2, by using an appropriate combination of linear layer and ReLU activation, the server can expect to recover
---------2 1-1 (
Σ
+ r(n, k.)
,=1
amount user data (where the data is in IR") perfectly. Note: r(n, is correction term (see proof for expression).
Proof. By construction of the imprint module, given random sample (batch) XI...., Xn (iid) the server perfectly recovers data whenever an imprint bin has exactly 1 element of the batch. Because we know the CDFs, we can create partitions of equal mass corresponding to imprint bins {b,} = [aj, 6,13 where P(X, b,), = 1/k Vi,j.
We can then phrase the problem of expected number perfectly recovered samples as a modified "stars and bars" problem. For given batch of data, to calculate the amount of data recovered, first calculate:
1 ())
Σ
N.
Where (+) is the number of ways to select the i bins that have exactly element, and N₂ is the number of orientations of the remaining data into the remaining bins so that no bin has exactly element. Note that we can do this because the bins all have equal mass, and thus we can factor out the (uniform) probability of any configuration from the sum.
Simply put, first take the configuration where there is only 1 bin with exactly 1 element, and weight it by 1, then we take the number of configurations with 2 bins with exactly I element, and weight it by 2, and so on.
In order to calculate N, we notice that our construction, once the bins with exactly 1 element are chosen, every other bin has either 2 2 elements. We focus the bins that have elements. By a simple "reverse" pigeon hole argument, we can now have at most _بي of the remaining bins containing any elements, otherwise, one bin would be guaranteed to contain exactly 1 element.
So we further select any 1 S j < [ number of the remaining
bins all to contain at least elements. Formally, this is equivalent calculating the number of orientations of integers {x1
so that
01
n2
constrained with 2, 2 2 vl Now, we make a change of variables instead calculate the number of orientations of integers
{ptk
so that
+ .... + Pj = - 2j
constrained with pl 2 0 VI. Now we just have a "stars and bars" problem with A'' = bars and n' i- stars. This reduces to:
n ("-j=1-')
orientations for the remaining bins with exactly j elements. Once these bins with 1 element, and bins with > 2 elements are chosen, all the other bins required have 0 elements.
15

---

Published as a conference paper at ICLR 2022
Linear Function
Assumed Distribution
MSE
PSNR
IIP-Pixel
Mean Mean
Normal
0.0183
75.75
65.62%
Laplacian
0.0174
79.63
71.88%
Cosine
Laplacian
0.0167
99.82
79.69%
Random
Normal 0.0203
91.29
75.00%
Table Ablation study linear functions and distributions.
So adding these parts together, we have the expected amount data the server can expect to reconstruct perfectly becomes:
r(a,k)
Σ
j j-1 (
k
n ))
72
Y2
k+n-1
k
We call the last "residual" terms r(n, k). The first of these terms corresponds the term in the expectation where all elements of the batch end up in separate bins, and the second term is the expected number of elements that land in the tail of the CDF not covered in any bin.
A.2 OTHER CHOICES LINEAR FUNCTIONS AND DISTRIBUTIONS
In previous experiments we have restricted our investigations to the linear function h R" - R that measures average brightness, i.e. hxx) = 1 Σi=1 I2 which we approximate to be normally distributed. Given that ImageNet (and this also applies to most image datasets) is pre-processed by normalization by color each channel, and that the number of pixels is large and they are not perfectly correlated, this is a reasonable approximation based on the central limit theorem that could similarly apply to other data modalities as well. For analysis, we visualize the closeness of this approximation based
m
evaluation over the full ImageNet validation set in Fig. 6
We verify that the actual ground truth distribution can be approximated by a normal distribution, but also see that the approximation imperfect. The attack works well in Fig. even with this discrepancy, however it could be further improved if the attacker has more accurate about the CDF. Image brightness better described by Laplacian distribution Ruderman [994]; Huang & Mumford (199).. Replacing the normal distribution by Laplacian distribution with scale 1/v2 does improve the accuracy slightly. This distribution can be further stabilized by considering higher frequencies compared to the mean, e.g. via DCT coefficients Lam & Goodman (2000); [Haan & Mumford (1999). We accordingly also visualize this distribution for e.g. the 32nd DCT coefficient Fig. and use this cosine wave for the imprint module (with scaling factor 4 m f. This leads the strongest attack against image data, but of course utilizes attacker knowledge that the users train on natural images.
On the flip side, the estimation can also be improved by replacing the linear function h with Gaus- sian random vector of independent draws from (0, J/m) The resulting distribution (4th figure in Fig. 6 approximates a normal distribution much better. While not as optimal as the Laplacian distribution for higher frequencies, this variant is applicable for other data modalities the data has bounded variance. Visualizations of the reconstruction with other linear functions can be found in
Fig. 7
Finally, even with small amount of data, the server could estimate the density of the quantity of interest. Visually, we plot the the estimated density as for several amounts ImageNet data used to estimate the brightness distribution. We find that even with 0.1% of the data used, the server could obtain a close approximation to the distribution of interest (see Fig.
A.3 COMPARISON TO HONEST SERVERS AND OPTIMIZATION BASED ATTACKS
We argue that the proposed attack operating in our threat model is significantly more threatening than existing optimization based attacks in the honest- but-curious server model. To illustrate this point and show by example that the change is threat model which amounts only a minor architectural
16

---

Published as a conference paper at ICLR 2022
Estimated density function for image brightness
Proportion of Dataset
Entire Dataset
Dataset
0.1 11% of Dataset
Density
0.5
0.4
02
01
0.0
brightness (ofter normalization)
Figure 5: Density of image brightness estimated from access different amounts of data from the ImageNet dataset.
OF EUN
DIN
Outiiaaion CT dea
303
43
Figure 6: Distributions on the ImageNet validation set for several linear query functions. From left to right: Mean compared to normal distribution, mean compared to Laplacian distribution, 32nd DCT coeffcient compared to Laplacian distribution, random normal vector compared to normal distribution. In each plot the approximate distribution used by the attacker is visualized in blue/black and the ground-truth (GT) distribution in green/red.
change in the neural network leads to a massive difference in reconstruction, we run the attack of Geiping et al. 2020 in the scenario of Fig. The results can be found in Fig. 8 The attack leads to an IIP score of 6. 6.25% when measuring in pixel space, 6.25% when measuring in LPIPS (Zhang et al. 2018) and, and 20.31% when measuring the cosine distances in feature space of this model (the metric of Yin et al. (2021)).
As additional ablation, also investigate whether the optimization- based attacks can find the optimal solution in the malicious server threat model that we consider. However, the right side of Fig. 8 shows that least conventional ppptiiization-basee methods have trouble finding the vulner- ability introduced by the imprint module. The vulnerability that the attack solves analytically might be hard exploit by first-order optimization or require specifically tuned optimization schemes to succeed. This also shows that defenses that attempt to detect privacy breaches by evaluating a range of optimization-ban based attacks would not have triggered an alarm for this attack.
A.4 OTHER DATA MODALITIES TEXT
As discussed in the main body. the attack is entirely data-agnostic and could be launched against any kind of input data, e.g. not only image data but also tabular features or text. In principle the input data could be comprised of random signals- these can still be binned and separated by the proposed attack. We verify this property by launching the same attack against text data.
We investigate the transformer architecture discussed in Wang et al. (2021) specifically for language tasks in federated learning scenarios and insert the imprint module as malicious block right after the word embeddings. This is strictly maliciously modified architecture normal feedforward blocks in a transformer would not span across the entire length of the sequence. We initialize the linear function a Gaussian random vector and make no modifications to the attack hyperparameters. We recover the input tokens ids from a direct lookup of their closest match in the word embedding layer (Zhu et al. 2019). To evaluate the success of the attack we generate sample batches of sentences
17

---

Published as a conference paper at ICLR 2022
Figure 7: Analytic reconstruction for an imprint model with 128 bins in front of ResNet-18. Left: The linear function the 32nd DCT coefficient and bins are based on a Laplacian distribution. Right: Linear function is Gaussian random vector and bins based normal distribution. Gray reconstructions denote bins in which no data point falls.)
Figure 8: Optimization-basae based attack of Geiping et al. (2020) for a ResNet- 18 and a batch size of 64, the setting of Fig. 2 Left: An honest server model. Right: Gradient inversion attack applied to module that contains the imprint module.
18

---

Published as a conference paper at ICLR 2022
from the wikitext dataset (Merity al., 2016) with batch size of 128 and sequence length of 32, tokenized via the GPT-2 tokenizer (Radford et al. 2019). Instantiating the attack with 512 bins immediately reveals 1 10 out of 128 sentences perfectly, leading to overall reconstruction accuracy of 86.33% which also a BLEU score of 88% and ROUGE-L of 87%. We refer to our open source implementation for further details.
We show examples of recovered data below, but note that printing the recovered text is not that insightful. The attack perfectly recovers subset of sentences of user text and cannot recover some other sentences entirely, in full analogy to the results for images in e.g. Fig. 7
Recovered wikitext data:
The Tower Building of the Little Rock Arsenal, also known as U.S. Arsenal Building, is building located in MacArthur Park in downtown Little Rock, Arkansas Built in 1840, it was part of Little Rock's s first military installation. Since its decommissioninn
The Tower Building has housed two museums. It was home to the Arkansas Museum of Natural History and Antiquities from 1942 to 1997 and the MacArthur Museum of Arkansas Military History since 2001. It has also been the headquarters of the Little Rock Esthetic Club since 1894. The building receives its name from its distinct octagonal tower. Besides being the last
A.5 TECHNICAL DETAILS
All experiments were implemented in PyTorch (Paszke et al. (0117) and were several laptop and machine CPUs, as the reconstruction itself requires only a few tensor operations. Especially, compared optimization-based reconstruction techniques, this makes the approach significantly faster and significantly more portable. For visualization purposes and to measure accurate PSNR and IIP scores all images (which are recovered in the order given by the chosen function h) matched with possible correspondences in the ground truth batch. The matching is found based on LPIPS feature similarities scores (Zhang et al., 2018) which are matched using linear sum assignment solver. No labels are recovered using the proposed approach which is entirely label- agnostic, but labels could be assigned a- a-posteriori using model predictions of the reconstructed data if required. For experiments where the imprint module is placed deeper into network, the network (which is here a ResNet) is linearized by resetting batch normalization parameters and buffers to the identity map, setting all residual paths to zero and initializing the first convolution and the shortcut convolutions to identity maps. The nonlinearities can be bypassed by bias shifting as in Goldblum et al. (2020).
PSNR scores are computed as average PSNR where we first compute PSNR scores per image and then average. This procedure is standard in computer vision, but does bias the score toward success- ful reconstructions, as the minimal PSNR score is bounded at 0, but its potential upside unbounded. For the image identifiability precision (IIP) score of Yin et al. (202T) we implement the score as proposed therein, but measure nearest neighbors not in the model feature space (which we consider biased, given that the model parameters are already used for reconstruction), but directly in image pixel space, where we check whether the given reconstruction is indeed closer to its true counterpart euclidean distance than any other image from this class in the validation set.
A.6 ADDITIONAL IMAGES
This section contains additional image examples, such using CIFAR-10 in Fig. 12 and Fig. 13 where an attack with the imprint module on this dataset shows that almost all of the user data is perfectly recovered. Furthermore. example panels of imprint modules inserted in later ResNet layers are visualized as well as the results of the sparse variant that is used to attack federated averaging scheme.
A.7 DEFENSE DISCUSSION
An algorithmic defense against the proposed attack would be to validate the incoming model pa- rameters on the user side. There, the attack with multiple bins requires repeated computations of
19

---

Published as a conference paper at ICLR 2022
Expected Data Points Reconstructed
450
Batch Size 256
400
Batch Size 512
350
Batch Size 64
300
250
200
1.00
1000
50
0.01
0.1
Proportion of added Parameters
Figure 9: Expected number of data points recovered for several batch sizes and increased bins and corresponding parameter increase. Model: ResNet50 with ImageNet, targeting the input to the 3rd residual block.
Figure 10: Left: Raw data for the 64 ImageNet images with separate classes. Right: Raw data for the 64 images from the white shark class.
the same quantity. At first, this pattern could be detected by rank analysis of all linear layers (which would return a rank of 1 for the linear layer of the imprint module described above). Yet, the at- tacker easily randomize few entries of the linear layer increase its rank without significantly weakening the attack, or introduce additional rows that compute normal deep features, so that we do not believe a defender can win by model analysis under the given threat model.
A.8 COMPARISON TO CONCURRENT WORK
We include comparison to the concurrent attack of Boenisch al. (2021) operating in the same threat model in Fig. 21 The threat model is characterized purely as parameter modification therein that only applies models with input-sized linear layers followed by ReL.U activations. In the same vein the attack discussed this work does not require malicious architecture modifications if these vulnerable layers are already present.
B REMARK ON RECOVERY
Note that for the recovery in Eq. ( we place either an averaging operation, or linear layer following 8C 8C
W with identical row elements. This is because in Eq. (2). the attacker needs b6, sufficient condition for this that the operation proceeding the imprint module, such averaging.
ōb A
20

---

Published as a conference paper at ICLR 2022
Figure 11: Left: Analytic reconstruction for linear model of 64 ImageNet images with separate classes (PSNR: 36.45 versus true user data). Right: Same recovery algorithm but for 64 images from the same class (white shark), (PSNR: 13.84 versus true user data.)
Figure 12:
Left (a): A batch of 64 CIFAR10 images. Right (b): The same batch of images recon- structed naively using Eq.
can be expressed as a matrix operation with identical row elements. This falls squarely within our assumed threat model and is empirically how implement attack.
C CODE RELEASE
We provide open-source implementations of all attacks investigated in this work. The repository thub. com/ hwe.ool//rbbbiii of the principles of this attack and the repository at httpp:: /github....... com/ JonasGe breaching embeds the attack in larger framework of privacy attacks against federated learning.
the fed shows a minimalistic implementation
21

---

Published as a conference paper at ICLR 2022
Figure 13: Left (a): A batch of 64 CIFAR1O images. Right (b): The same batch of images recon- structed using the imprint module with 300 bins. Gray images can result from collisions within given bin.
Figure 14:
Different placements of the imprint module in a ResNet- 18. From left to right: Before the first block (56x56), before the third block (28x28), before the fourth block (14x14) and before the average pooling (7.7). Compare to a placement before the first convolution (224x:224) and raw input data in Fig. Enlarged versions of each panel can be seen in Figs. L5
18
22

---

Published as a conference paper at ICLR 2022
Figure 15: Different placements of the imprint module in ResNet- 18. Before the first block (56 x 56). Only the first three channels at this position are utilized in this demonstration.
Figure 16: Different placements of the imprint module in a ResNet-18. Before the third block
(28 28). Only the first three channels at this position are utilized in this demonstration.
23

---

Published as a conference paper at ICLR 2022
Figure 17: Different placements of the imprint module in a ResNet-18. Before the fourth block
(14 x 14)
Figure 18: Different placements of the imprint module in ResNet-18, Before the last average-
pooling layer 7). Only the first three channels at this position are utilized in this demonstration.
24

---

Published as a conference paper at ICLR 2022
Figure 19: Results for federated averaging for 8 steps with 8 images each, i.e. 64 unique data points for a single user, and 128 bins. PSNR: 32.65. IIP: 70.31%. Drift of bins during local updates leads few duplicated entries.
(picel-baard)
+ Barch size bins, Mode) homs, Midel
Irap noll
Gradient
Figure 20: Left: IIP score vs Laplacian gradient noise. Right: Exemplary recovery for a 0.01. Recovery is stable for a large range of Laplacian gradient noise injections.
100
80
UP (pixel-baed)
Batch size 128, Ours
20
Batch size 128, CAH
0-
100
1000
Number of bins
Figure 21: Comparison to the "Curious Abandon Honesty" (CAH) attack proposed in Boenisch
all 2021) We find our attack outperforms their attack at every scale.
25


Reviews:
[{"review_id": "sEUfQNp6UU0", "reviewer": "Reviewer_AxhH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Provides clear and easily understood high level message: Federated Learning is very vulnerable if the server cannot be trusted. \n- The paper does not rely on simplifying and unrealistic assumptions. It would be (reasonably) simple to implement the method in practice. \n\nWeaknesses:\n- Paper's builds heavily on key insights from previous works, such that from the previous works one could deduce the results in this paper without too much difficulty. And this insight came from previous works on privacy in deep learning, meaning this paper is not making a contribution by newly bringing this insight into the community. ", "summary_of_the_paper": "This paper explores the privacy of federated learning when the server is malicious, specifically that it is allowed to control the architecture and and set certain weights in adversarial ways. By building on a weakness inherent to the RELU activation function, they show that w.h.p the malicious server can obtain exact reproductions of at least some inputs. They then empirically explore how well this works in practice, showing that the method introduced does pose a significant risk to user data privacy. ", "summary_of_the_review": "The paper provides another piece of evidence that federated learning on its own is not sufficient to protect user privacy, and explains how even with only a minor malicious change the server can exactly recover user data. And the message is very clear and well exposed in the paper. However method is only a slight variation on ideas from previous work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ctJEUlX5XNQ", "reviewer": "Reviewer_t8bw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Well-written and organized paper. The threat models and attacks are clearly described and well-motivated. \n2. The proposed imprint module is very novel and exploits the vulnerabilities of gradient computation in popular network design.\n3. The attack experiments are well-executed. The reconstructed results are really close to the original inputs, even for gradients aggregated over a large batch size or multiple local iterations or with local DP, which is much better than reconstructing the averaged inputs as most previous works do.\n\nWeaknesses and questions\n1. Stealthiness of the attacks, as acknowledged by the authors, might be a problem and the attacks could be easily detected by inspecting the model architecture or parameters. Could the parameters in the imprint modules themselves reveal the attacks, given their distribution might be very different from other layers? Is it possible to disguise the parameters as normal layer parameters as well?\n2. Since the imprint module changes the intermediate activations, how does this affect the overall utility of the models? Unfortunately, I did not see any results on how the architecture might change the model performance. If the performance is surprisingly different for a data on device, then the device owner might know there is something wrong with the particular model received from the server.\n3. I did not understand the setup for the federated learning experiments in Section 4.3. How many users are sampled in each iteration? How are their data splitted? Does the attack work even when there are multiple local iterations and a large number of users sampled in the central iteration? I.e. aggregation of model updates from a large number of users (i.e. magnitude of thousands) trained with multiple local SGD (which could be the setup when large tech companies deploy FL)\n4. There could be another DP related mitigation where the model updates are first securely aggregated and then added with central DP noise before sending to the server. Could the server still infer anything from the noised aggregate?  \n", "summary_of_the_paper": "This paper introduces novel attacks for reconstructing training data given the device contribution in the federated learning setting. The attacks exploit the gradient behavior of ReLU and insert carefully tailored feed forward layers into a benign network so as to reconstruct the input from the gradients. The demonstrated examples show the effectiveness of the attacks that even training with large batch size, attacks with certain knowledge about the distribution of the data could recover almost the exact training data. \n", "summary_of_the_review": "This paper proposed novel and effective data reconstruction attacks in federated learning. Overall, I recommend accepting this paper for its solid technical contribution and convincing results. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HgxxT5zHOfi", "reviewer": "Reviewer_nVzn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The result (Proposition 1) in the imprint module section has a strong assumption that the server knows the CDF of a quantity associated with user data.  This seems contradictory to the statement in the paper that claims the results here are more data agnostic.  \n\nAlthough the imprint module can be inserted in any position in a neural network (subject to a constraint), I am left wondering whether it would make it better for the attack to put the imprint module earlier or later in the neural network.  I would be interested to see results when testing different positions in the neural network.\n\nThe experiments show the power of the attack, while demonstrating that not every image is reconstructed.  Regarding Figure 4, I understand that the license plate is blocked for privacy reasons, but I wonder if the reconstructed image still has a readable license plate.  \n\nIn the potential defense and mitigation strategies section, I would be interested to know the overall local privacy guarantee, in particular what dimension are the updates?  I would expect sigma = 0.01 to offer better privacy guarantees, but maybe the updates are high dimensional.  I would have liked this section to be more fleshed out.  \n\n### UPDATE ###\nI have read over the author feedback and think the suggested changes can improve the work.  I will keep my score unchanged.\n", "summary_of_the_paper": "This paper presents a new attack on federated learning, demonstrating that model updates shared in a federated setting can still leak user data.  Although federated learning has been used as a privacy technique, this paper shows that other mitigation techniques should also be used.  Attacks before were based on updated from a single data point, and hence aggregating data in an update was used as one approach against this attack.  The adversarial setting here is with a malicious server who may modify the model’s architecture in a seemingly innocuous way.  Allowing models to be changed in ways that are currently allowed with FL APIs, this paper shows that user data can still be reconstructed even with large aggregations.  Another benefit of this approach is that it is more data agnostic, not relying on strong data priors.  \n", "summary_of_the_review": "Overall a nice attack that shows the keeping data distributed while only sharing model updates does not provide strong privacy guarantees.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2s9t02wCKXS", "reviewer": "Reviewer_Pq3S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea looks interesting, but I have some concerns as below.\n\n1. The assumption used in Proposition 1 is not realistic\n    1. I’m not sure how come the server knows the CDF of “some quantity associated with user data”. By the way, would you clarify “some quantity associated with user data”? It is vague.\n2. Does the reconstruction available for all cases?\n    1. It seems like several figures has gray boxes, meaning that the image is not recovered. Do we have some understanding why/when such failure cases occur? If not, it is better to delve into what happens to that failure scenarios and what is the possible fix for that failure.\n3. No comparison with related works?\n    1. I have no idea what are other schemes handling malicious server scenario, but if we can find some, it is necessary to compare this scheme with conventional schemes. \n", "summary_of_the_paper": "This paper provides a privacy attack scheme for federated learning, retrieving the private data from the model aggregation techniques.\n", "summary_of_the_review": "This is an interesting paper, but it is better to include more concrete theoretic/empirical analysis on why/when the suggested scheme works. Moreover, comparison with existing schemes is missing. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide an interesting improvement on privacy attacks in federated learning, demonstrating the ability to extract individual points even over large batches. While there were some concerns about the technical difficulty of the approach, reviewers were broadly in support of the work. As I tend to agree, this is an interesting strengthening beyond what it appears we were able to do before. This is yet another piece of evidence against the canard in FL that only sharing gradient updates provides privacy guarantees.
Decision: Accept (Poster)