Data Leakage in Federated Averaging
Authors: Dimitar Iliev Dimitrov
Mislav Balunovic
Nikola Konstantinov
Martin Vechev
Venue: Accepted by TMLR ()

Abstract: Recent attacks have shown that user data can be recovered from FedSGD updates, thus breaking privacy. However, these attacks are of limited practical relevance as federated learning typically uses the FedAvg algorithm. Compared to FedSGD, recovering data from FedAvg updates is much harder as: (i) the updates are computed at unobserved intermediate network weights, (ii) a large number of batches are used, and (iii) labels and network weights vary simultaneously across client steps. In this work, we propose a new optimization-based attack which successfully attacks FedAvg by addressing the above challenges. First, we solve the optimization problem using automatic differentiation that forces a simulation of the client's update that generates the unobserved parameters for the recovered labels and inputs to match the received client update. Second, we address the large number of batches by relating images from different epochs with a permutation invariant prior. Third, we recover the labels by estimating the parameters of existing FedSGD attacks at every FedAvg step. On the popular FEMNIST dataset, we demonstrate that on average we successfully recover >45% of the client's images from realistic FedAvg updates computed on 10 local epochs of 10 batches each with 5 images, compared to only <10% using the baseline. Our findings show many real-world federated learning implementations based on FedAvg are vulnerable.

Published in Transactions on Machine Learning Research (10/2022)
Data Leakage in Federated Averaging
Dimitar I. Dimitrov
to: dimittrr dww..
Deportment of Computer Science ETH Zurich
Mislav Balunović
mislav. miiidav..atunoviieiiff.......
Department of Computer Science
ETH Zurich
Nikola Konstantinov
ETH AT Center
Department of Computer Science
ETH Zurich
Martin Vechev
Deportment of Computer Science
ETH Zurich
Reviewed OpenReview: http:///ooenneeeeeee///////////a
Abstract
Recent attacks have shown that user data be recovered from FedSGD updates, thus breaking privacy, However, these attacks are of limited practical relevance federated learning typically uses the FedAvg algorithm. Compared to FedSGD, recovering data from FedAvg updates much harder as: (i) the updates are computed at unobserved intermediate network weights, (ii) large number batches are used, and (iii) labels and network weights vary simultaneously across client steps. In this work, we propose a new oplimization- attack which successfully attacks FedAvg by addressing the above challenges, First, we solve the optimization problem using automatic diffcrentiation that forces simulation of the client's 's update that generates the unobserved parameters for the recovered labels and inputs to match the received client update. Second, we address the large number of batches by relating images from different epochs with a permutation invariant prior. Third, we recover the labels by estimating the parameters cxisting FedSGD attacks at every FedAvg step. On the popular FEMNIST dataset, we demonstrate that on average we successfully recover >45% of the client's 's images from realistic FedAvg updates computed on 10 local epochs of 10 batches each with images, compared to only <10% using the baseline. Our findings show many real-world federated learning implementations based FedAvg are vulnerable.
1 Introduction
Federated learning (MMMahan et al., 2017) is a general framework for training machine learning models fully distributed manner where clients communicate gradient updates to the server, as opposed to their private data. This allows the server to benefit from many sources of data, while preserving client privacy, and to potentially use these massive amounts of data to advance the state of the art in various domains including computer vision, text completion, medicine and others [Kairouz et al. 2021
Unfortunately, recent works (Zhu et al. 2019 Zhao et al. 2020 Geng et al. 2021 Geiping et al. 2020 Yin al.) 2021) have shown how to compromise the privacy aspect of foderated learning. In particular, they demonstrated that an honest-but-curious server can reconstruct clients' private data from the clients' gradient updates only phenomenon termed gradient leakage. While these aftacks recover data, even for

---

Published in Transactions on Machine Learning Research (10/2022)
Cinv
(,,.3)
=((x22001)
Label counts
x 1
4
A 1
Xi,
Xí.2
X1,1
X2
CKCeein Liov]
Ay 2
8X
AS 2
Siim
0
0f
2
05
Figure 1: An overview of our attack on FedAvg. Our attack proceeds in two phases: first it estimates the label counts X (left) and then uses these estimates to simulate the FedAvg updates on dummy inputs X (right). We then use automatic differentiation (groen) to match the final weight 02.2 to the observed weight 02 by minimizing the combination of the reconstruction loss Lsim and the epoch order-invariant prior Linv-
complex datasets and large batch sizes, they are still limited in their practical applicability. More specifically, they typically assume that training is performed using FedSGD where clients compute a gradient update on a single local batch of data, and then send it to the server. In contrast, in real-world applications of federated learning, clients often train the model locally for multiple iterations before sending the updated model back to the server, via an algorithm known as federated averaging (FedAvg) (McMahan et al. 2017), thus reducing communication and incrcasing convergence speed (Kairouz et al., (021). Under FodAvg, the server only observes the aggregates of the client local updates and therefore, one may expect that this makes the reconstruction of client's private data much more difficult.
This work: Data Leakage in Federated Averaging In this work, we propose a data leakage attack that allows for reconstructing client's data from the FedAvg updates, which constitutes realistic threat to privacy-preserving federated learning. To this end, we first identify three key challenges that make at attacking FedAvg more difficult than FedSGD: (i) during the computation client updates, the model weights change, resulting in updates computed on weights hidden from the server, (ii) FedAvg updates the model locally using many batches of data, (iii) labels and parameters are simultaneously changing during the update steps.
These challenges are illustrated in Figure D where client with a total of 6 samples trains the model for 2 epochs, using a batch size of 3 resulting in 2 steps per epoch. Here, the model weights 0 change at every step and there are many different batches, with the samples being processed in different order during the first (blue) and sccond (rod) epoch.
Addressing these challenges, we introduce a new reconstruction allack that targets Fed Our method first runs a FedAvv-speccfic procedure that approximately recovers the label counts X within the client's data (left in Figure [ by interpolating the paramcters, thus addressing the last challenge. This information is then used to aid the second phase of the attack, which targets the individual inputs. To recover the inputs, our algorithm uses automatic differentiation to optimize reconstruction loss consisting of two parts. The first part is a FedAvg reconstruction loss Laim, designed by simulating the client's 's training procedure, that accounts for the changing parameter values, addressing the first challenge. The second part solves the remaining challenge using an epoch order-invariant prior, Linv; which exploits the fact that the set of inputs is the same in every epoch, thus mitigating the issuc of having a large number of batches during the local updates. To leverage this information, this term uses an aggregating order-invariant function g to compress the sequence of samples in an epoch, and to then compare the results of this operation across different epochs.
We cmpirically validate the effectiveness of our method at recovering user data when the FodAvg algorithm is used on realistic settings with several local epochs and several batches per epoch on the FEMNIST (Caldas et al. 2018) and CIFAR100 (Krizhevsky et al. 2009) datasets. The results indicate that our method can recover significant portion of input images, as well as accurate estimates of class frequencies, outperforming previously developed attacks. Further, we perform several ablation studies to understand the impact that individual components of our attack have on overall attack performance.
2

---

Published in Transactions on Machine Learning Research (10/2022)
Main Contributions Our main contributions are:
Identifying key challenges in applying gradient leakage algorithms to FedAvg updates (Section
4.1
Solving these challenges through novel algorithm based on automatic differentiation through a simulation of the FedAvg client update (Section 4.3) and an additional term incorporating prior
information about the optimization process (Section 4.4 as well as new FodAvg-specing label reconstruction attack (Section 4.5
An implementation of this algorithm, that successfully attacks realistic FedAvg updates with multiple epochs and batches per epoch. Code is available at httpss//gghuub.coo//eess///////
lea leakage
A thorough evaluation of our attack on the FEMNIST and CIFAR100 datasets that validates its effectiveness at recovering private client data from Aed updates. In some of the settings we consider, our attack is able to accurately recover more than 50% of the clients' images, while prior methods are only able to recover 10%.
2 Related work
In this section we discuss closely related work in the area of data leakage for federated learning.
Federated learning Federated learning (FL) was originally proposed by McMahan et al. (2017) as a framework for training machine learning models with private data from edge devices (or clients), without. transferring the original data to central server and instead communicating only gradient updates. Because of the potential of FL to enable training machine learning models at scale while respecting the sensitive clients' data, a substantial amount of recent research focuses on studying and enhancing the privacy guarantees provided by FL (Abadi et al. 2016 Bonawitz et al., 2017). However, recent work has cast doubt on the extent to which FL is inherently private, by demonstrating that in many cases the information about the model gradients alone is enough to reconstruct the clients' data.
Gradient leakage Melis al. 2001 Zhu al. (2019) demonstrated that clients' data can be reconstructed
from federated gradient updates. Next, Zhao et al. (2020) improved the previous attack by showing
that reconstructing inputs and labels separately simplifies the optimization process and leads to better
reconstruction. Orthogonally, Zhu & Blaschko (2020) formulated gradient lcakage attacks as system of equations and provided an analylical solution under the assumption that the client has computed the update on a single data point. More recently, much work has been done on gradient leakage attacks on specific data domains through the use of input priors (Geiping et al., 2020 Dimitrov et al. 2022 Yin ct al., 2021 Li et al., 2022. Deng et al., 2021). What these attacks demonstrate is that it is possible to reconstruct individual inputs from gradients aggregated over a. batch of imputs, provided that strong prior in the form of regularizer is added to the optimization problem. Our proposed prior in Section 4.4 provides similar benefits. Additionally, following the observations of Zhao et al. (2020), Geng et al. (2021) designed attacks that approximate the frequency counts of labels in a single client's 's batch of data by looking at the gradients of the last linear layer of the model. We introduce a FedAvg-speccfic extension of this method in Section 4.5)
While the aforementioned works have achieved substantial progress in making gradient leakage attacks practical, the majority of these works focus on the reconstruction of single data point or single batch, given the respective gradient. Even works that consider FedAvg updates, such as Geiping et al. (2020) and Geng el al. (2021) focus a setling where eit small amount of epochs or small nunber of steps per epoch are executed on the clients' side. In contrast, we develop an end-to-end attack able to recover clients' data points from FodAvg updates computed on several epochs each with multiple intermediate steps - a problem of greater practical relevance given that FL clients usually perform many local iterations before sending their updates to the server to speed up convergence (McMahan et al. 2017
Defenses against Gradient leakage Several works have studied the task of defending against gradient leakage. There are three main types of defenses heuristic (e.g (unn et al. (221)) | and [Scheliaa et al. (2022)
3

---

Published in Transactions on Machine Learning Research (10/2022)
which are often broken by stronger attacks Balunovic et al. 2021 : defenses based adding random noise, that provide differential privacy guarantees (Abadi et al. 2016) but often result in reduced model accuracy; and defenses based on secure aggregation (Bonawitz et al. 2017) that ensure that only aggregated updates from multiple clients are disclosed to the server thus making the reconstruction harder. In Appendix C.1 wc experiment with networks defended using Gaussian and Laplacian noise, as well as random gradient pruning first suggested by Zhu et al. (2019). Further, while attacking securely-aggrogateed updates not the main focus of this work, following Wen et al. (2022). in Appendix C.5 we provide experiments on recovering data from aggregated FedAvg updates. As attacking aggregated updates from large number of clients remains unresolved problem in the gradient leakage community, future work is needed to obtain better results in this setting, however, we see even aggregated updates do not provide absolute client privacy.
Other threat models In this work, we focus on the honest-but-curiou server model for analyzing client's data privacy. That is, we assume that the server follows the standard training algorithm, but can attempt to reconstruct clients data through the received updates. Stronger threat models have also been studied. Works such Fowl et al.] (2022, 2021): Boenisch al. (2021) demonstrate that the server can achieve much better reconstruction of private data instances by adapting the model weights and/or architecture.
3
Background
In this section, we review the classic components of foderated learning, provide an overview of cxisting gradient leakage methods and their limitations, and introduce the relevant notation.
3.1 Federated learning
Throughout the paper, we assume that a server aims to train a neural network So parameterized by a parameter vector 0, with the help of clients. We also assume that the neural network has L layers sizes R1, ..,,..,, my respoctively, with the dimension of input variables being d and the number of classes equal K (with nt K for notational convenience).
The vast majority of FL methods used in practice are based on the FedAvg algorithm (McMahan et al. 2017 The optimization process consists of T communication rounds. In each round, the server sends the current version of the model Of to a chosen subset of the clients. Each selected client e then computes an update to the model, based on their own data, and sends an updated version 0f of the model back to the server.
Algorithm D outlines the FedAvg's client update algorithm for a selected client e at some communication round. We denote X e R and y [K] the client's private inputs and labels respectively, where No signifies the number of private data points of that client and [K] denotes the set (1,2,..., K). At each local client's epoch E {1, 2, .,..,...)) }), the client randomly partitions its private data into sets of 8 [ N batches {X e [B')} and (Y b [B'],, each of size m (Line 6). For each batch and epoch e, local SGD update [Ruder 2016) is computed for the cross entropy loss (denoted CE in Algorithm D on the respective batch of data. The SGD update is then applicd to the network weights, producing the new intermediate weights 0%,, (Line Finally, the client sends the last updated weights 0 = 0g back to the server.
As special case of FedArg, when only one epoch (E 1) and one batch (B- 1) are used locally in all rounds, one obtains the so-called FodSGD algorithm. While casier to attack, FedSGD is rarely used in practice especially when the number of clients is large, as it leads to significant increase in communication.
3.2 Gradient leakage attacks
In this section, we review existing attacks based on the honest-buutcurrios server model. The majority of these reconstruction methods, known as gradient leakage attacks, target the FedSGD training procedure ZZm ct al. 2019 Zhao et al.] 2020), where the clients communicate a single gradient update to the server. As we attack FedAvg updates in this work, which share weights updates instead of individual gradients, we use the more general term data leakage allacks to refer to leakage attacks from both types of updates.
4

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm The FedAvg's ClientUpdate Algorithm (Adapted from McMahan et al. (2017))
1: function CLIENTUDDATE(XX, y S, 0% 71, m, 8)
B
m 1 dū -
4: for epoch E [E] do
5:
0 02- 02-1.8-
6:
(X (Ye,)) *AARTIIIONDATTAXY, YS, m)
7: for batch b E [ do
8:
0²bb - 8 6-b 17- V, CE(f (. (.b))
9: end for
10: end for
11: return g
12: end function
Reconstructions based on single gradient To describe how existing attacks work, consider the case where the batch size m = 1, so that 8'- = VxCE(f(X;, Y,) is a gradient with respect to a single data point (Xy, Y of the client The most common approach for reconstructing (Xf, Y,S) from the gradienl is to search for an input-output pair, whose gradient most closely approximates the one that was sent back to the server. Given a distance measure D in gradient space, this translates to the following optimization problem:
arg arg min D (VoCE(((s, 0*), VoCEU((X;,0) (a.y)
(1)
The distance measure D is commonly chosen be the L2 (Zhu et al. 2019), T, (Deng al. 2021 cosine distance (Geiping et al. 2020). In the case of FedSGD, it is known that additionally using prior knowledge of the input distribution can improve the attacks (Geiping et al.) 2020: Yin et al. 2021 Dimitrov et al. 2022 as it can guide the design of regularization terms that help in navigating the large optimization search space.
Label reconstruction methods Previous work (Zhao et al., 2020; Huang et al. 2021 has shown that in the cases when the labels of the associated data points are known, the optimization problem in Equation | is easier to solve for x by standard gradients-based optimization methods. For this reason, a common approach in the gradient leakage literature is to separate the process of recovering the clients' labels from that of the
input reconstruction (Geiping et al. 2020: Geng et al. 2021 Yin el al., 2021
Here we describe the procedure proposed by Geng et al. 20221 to reconstruct a client's label counts from FedSGD updates, which we later on adapt to the case of FedAvg. The paper assumes that the last layer of the net work is linear, i.e. s(X;, 0) WFC (Xf 0) + brc, where WFC RR= 1xna a trained weight, matrix, bFC E 1e is an optional bias term and fL-1 (X;,0) 0) is the activation of the network for input Xद at layer L-1. 1. In addition to the linearity assumption, Geng et al. (2021) assume that the K network outputs are converted into probabilities py by applying the softmax function and are then fed to the cross entropy loss for training, as in Algorithm 1 We note that these assumptions hold for most neural network architectures.
Under these assumptions, Geng et al. (2021) shows that one can prociscly compute the label counts Ag from FedSGD updates if the following quantities, computed on 8*, are known: (i) Ag the sum of the client's gradients with respect to the CE loss of the weights in WFC that correspond
the 4th layer output, (fi) pf, the softmax probabilite for class i computed on (F), and (iii) O the sum of the neuron activations in layer L - 1 computed on (XFYY;) While A can be computed by the server from the client update, Ph, and Of are unknown they depend on the client's data. Instead, Geng et al. (021)) suggest to use approximations Pi. and Ö*, computed by feeding the network at parameters 6 with random dummy data, as opposed to the unknown client's data point. This results in the following approximation to the label counts:
N A.Wic
E = N N
PE
(2)
A crucial limitation of these gradient leakage attacks that they are tailored to FedSGD. As we show in the next section, both the input and the label reconstruction are significantly harder in the context of FedAvg, yielding previously developed attacks inapplicable.
5

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm Overview of our attack
function ATTACK(J, 8 8% 77, m, E)
2:
v0 < - (0= -0%
(ig) & RECLABELS(( 8 ,U*)
4:
Y f RANDORDER((A)) ((EE))
5:
X + RANDINIT(M,, E)
6:
while not converged do
8r + SIMUPDATE(X*) X y f, 8" m, E)
7:
8:
V0 t vite (0+ - -0)
l t- Laim (V0", (V0 V0%) + Ainv Liuv (X²)
10: X X 'hoor" ox
df
11: end while
12: return MArcHEPocn(XX9) Fe
13:
end function
4 Effective Reconstruction Attack on FedAvg
In this section, we first discuss the challenges FedAvg updates pose for data leakage attacks in Section 4.1 We then present an outline of our reconstruction attack in Section 4.2 and discuss how it addresses these challenges. Finally, for the rest of the section discuss each of the elements of the attack in more details.
4.1 Challenges to Data Leakage in Federated Averaging
We first outline the key challenges for performing gradient leakage when training with FedAvg, making the leakage more difficult than in the case of FedSGD where these problems do not exist.
The most discernible challenge is that in FedAvg clients perform multiple local updates, each computed intermediate model parameters 0% that are unavailable to the server. If the nunber of local updates is large, the change in parameter values is likely to be large, making the data leakage problem substantially more difficult. We address this issue in Section 4.3
The second important challenge that FedAvg updates the model locally using many different batches of data. More concretely, the client updates the model for several local epochs, in each epoch performing several local steps, and in each step using different batch of data. As the server only sees the final parameters, it does not know which batches were selected at each step. This is more difficult case than FedSGD which performs its update using only a single batch of data. We address this issue in Section 4.4
Finally, the third challenge is that labels and parameters are simultancously changing during local updates. In practice, this makes it difficult to reconstruct label counts (necessary to reconstruct the inputs) via exiting FedSGD mothods. In Section 45 we introduce our method for label reconstruction which more robust. w.r.t to changing batch sizes and number of epochs.
4.2 Federated Averaging Attack
We now present our attack, which aims to address the aforementioned challenges by reconstructing the clicnts" data from the original weights 8", the client weights 8 and client parameters 7, m and E. We provide an overview of our method in Algorithm 2 First, we invoke a label reconstruction procedure to generate the and then use y an input an optimization problem that minimizes a reconstruction loss é (Line 9 with respect to a randomly initialized (via RANDINIT) input set X (Line
reconstructed labels Y (Line
To address the challenge with label reconstruction, Algorithm relics on our algorithm RECLABELS. This am improved, more robust. version of the method of Geng et al. (2021) discussed in Section 3.2 and tailored to FedAvg. The algorithm no longer directly estimates network statistics Po and Ö* from 0² (as introduced in Section 3.2). but instead estimates the statistics for each 0%, by interpolating between the statistics at 0 and the statisties at 0=

---

Published in Transactions on Machine Learning Research (10/2022)
For the input reconstruction, Algorithm 2 uses novel FedAwg-specific reconstruction loss E (Line 9 which consists of two components. The first component the gradient similarity loss Sain (VØ', V0"), which uses a server-side simulation of the client's updates via the SIMUPDATE method to obtain the approximated averaged update V (Line 8) and measures its similarity to the true averaged update V0- = - (0 8), where U = 3 B denotes the number of local update steps of the weights. This component aims to address the issue of having multiple hidden local steps by simulating the client's 's updates end-to-end. The second part is our epoch order-invariant prior (X') which takes advantage of the fact that, even though the number of local steps Ut is large in realistic FedAvg settings, each imput is part of exactly one batch per epoch. Finally, we reconcile our predictions by invoking MATCHEPOCH (discussed later in the section).
For simplicity, we present the optimization step in Algorithm 2 SGD update with fixed learning rate Thee. In practice, we instead rely on Adam (Kingina & Ba 2014), as it was shown to perform better for solving gradient leakage problems (Zhao et al. 2020). The gradients
at required by SGD and Adam, are axx
calculated via an automatic differentiation tool in our case JAX (Bradbury et al. 2018
Over the next few sections we formally describe the components of Algorithm 2 In particular, we first discuss the reconstruction of inputs X from given label counts Xg, as the inputs are typically of primary interest. to the attacker. To this end, we describe our server simulation method SIMUPDATE in Section 4.3 and our epoch order-invariatt prior (X*) in Section 4.4 Finally, in Section 4.5 we explain how to compute the estimated label counts Xg. thus creating a complete end-to-end attack.
4.3 Input Reconstruction through Simulation
As described in Section 4.1 FedAvg client computes its gradient steps the intermediate model parameters 0% unknown to the server. These parameters can substantially differ from both 0 and 0 when the number of local steps U is high. To address this issue, our attack relies on the method SIMUPDATE to simulate the client's updates from Algorithm T and thus generate approximations of the intermediate 02,6's.
An important consideration for our simulation is how to split the optimization variables X initialized at Line and repeatedly updated at Line in in Algorithm 2 as well as the label counts XE (or their reconstructions X in the case of an end-to-end attack), into batches, the client's original split is not available to the attacker.
We fix the split randomly into B batches y = (Yg E [B*]] (via RANDORDER) before the optimization in Algorithm 2 begins and keep the same split for all & epochs throughout the reconstruction. This provides a stable input order throughout the reconstruction and making the optimization simpler. We do this for two reasons: (i) the original client split of the labels into batches is hard to reconstruct, and (ii) we experimentally found that most batch splits produce similar weights at the end of cach epoch. For the optimization variables Xe, we choose to split it into U = E B separate variables X = (X2, I E [8],b E [B*]), one for each batch and epoch e.
With these batch splits, SIMUPDATE executes a modified version of Algorithm I The two modification are: (i) The original partition of the data (Line G is replaced by our partitioning as described above, and (ii) we replace all occurrences of X and Y.,, in Line with X and Y,. The final reconstruction error is then calculated as the distance fsim between the truc average weight update v and the simulated update V
( as Geiping et al. (2020)
In this paper, we rely on the cosine distance for Lsim; i.c. faim ||00 |V showed it reconstructs data well a wide range of challenging scttings.
In addition to the approach described above, also experimented with other variants. In particular, we investigated version that uses 8 input optimization variables X= [Xg b e [B*]] that are shared across all & epochs, as in Geiping et al. (2020). This change is applicd at initialization time (Line 5 in Algorithm 2 Opposed to our algorithm (described above), the optimization problem now becomes more complicated, since the optimization variables enter into the computation of multiple local steps. Further, we considered a simulation where 8= 1, which is relatively precise when the number of batches that the client uses per epoch 8 is small, but significantly diverges for larger B as typically used by FodAvg. We compare to these variants in Section 5 and find that they perform worse, confirming the effectiveness of our method.
7

---

Published in Transactions on Machine Learning Research (10/2022)
4.4 Epoch Order-Invariane Prior
The use of multiple local batches of data for several epochs in the FedAvg client update makes the optimization problem solved by Algorithm 2 harder compared to FedSGD. In particular, several reconstructions of each input have to be generated and combined, as the inputs are repeatedly fed to the network. Idcally, we want to enforce the property that all of the reconstruction variables corresponding to the same input at different. epochs hold similar values. However, the correspondence between the variables across epochs is hard to establish as the client randomly splits its inputs into different sets of batches at each epoch. We make the observation that the value of any order-invariant function g that takes all of the client data points as inputs has the same value at different. epochs, assuming that each input is processed exactly once per epoch. To this end, during the simulation our prior encourages this property for every pair of epochs a chosen distance function Day between the values of g on the optimization variables X at epoch ej and eg. Let X E,, denote the 7th element in batch b and epoch in X This results in the loss term:
by penalizing
8
Dinv (o(18e, 1 b E [89, [m])))(((X ez b i (m]])).
Linv
E
02
experiment with several different order-invariant functions g and show that the best choice is dataset-dependent. Further, we show that the & distance is a good choice for D In Line Of of Algorithm 2 the prior is balanced with the rest of the reconstruction loss using the hyperparameter Anv-
In Section 5.4
Once we have reconstructed the individual inputs at different epochs, we need to combine them into our final input reconstructions {X; i E [N We do this in MATCHEPOCи (Line 12 in Algorithm 2) by optimally matching the images different epochs based on a similarity measure and averaging them. The full details of MATCHEPOCH are presented in Appendix A In Appendix C.6 we experimentally show that our matching and averaging procedure improves the final reconstruction, compared to using the epoch-specific reconstructions, which aligns with the observations from previous work Yin et al. 2021
4.5 Label Count Reconstruction
As we point out in Section 2 state-of-the-art FedSGD attack methods aim to recover the client's ' label counts Xg before moving on to reconstruct the inputs X This is commonly done by computing statistics on the weights and the gradients of f (Yin et al. 2021 Geng et al., 2021). As demonstrated by our experiments in Section 5.2 existing FodSGD mothods struggle to consistently and robustly reconstruct the label counts Ag. We postulate that this is due the simultaneous changes across local steps of both the labels inside the processed client's batch and the network parameters. To address the challenge of recovering label counts robustly (to different batch and epoch sizes), we make the obscrvation that the model statistics used by these methods often change smoothly throughout the client training process.
As the label counts reconstruction mechanism demonstrated in Geng et al. 2021 is the current FedSGD state-offthe-art, we chose to focus on adapting it to the FedAvg setting using the observation above. Recall from Section 3.2 that the method depends on the estimates P and O of the average softmax probabilities p and of average sum of neuron activation at layer T. - 1. For FedSGD, both estimates are computed at O and with respect to dummy client dataset, since the true data of the client is unknown to the server. However, in the case of FedAvg, multiple local steps are taken by the client, with each batch using different labels and changing the intermediate model parameters (both labels and parameters remain unobserved by the server). To account for these changes, we compute approximate counts X2, for each local step i [Ue] and aggregate these counts to obtain the approximate label counts X for the full datasct of the client.
We proceed as follows. First,, we compute approximations 79 and Ō+ at 0a (using client dummy dataset) and approximations Pe and O at d (using the same client dummy dataset), as described in Section 3.2 Then, we linearly interpolate these estimates between P3 and Ōa and Pi and OF for the U steps of local training, that is, we set ph = J. + -0-1 Pi and Ō; = th 2 + U. Ue-i0e for each i e [U9] and k e [K]. Next, we use these approximations to obtain approximate label counts Xe, for every training step E [U%. using
8

---

Published in Transactions on Machine Learning Research (10/2022)
Equation To compute the final approximate counts, we set X -E=,", Note that we may need to adjust Xg in order to enforce the invariant that Σx= Xf N We remark that this interpolation of the local statistics is general and can be applied to other label reconstruction methods.
5 Experimental Evaluation
In this section, we present an experimental evaluation of our proposed attack and various baselines.
Experimental setup We conduct our experiments on two image classification datasets. One is FEMNIST, part of the commonly used federated learning framework LEAF (Caldas ct al. 2018). FEMNIST consists of 28 x 28 grayscale images of hridwritten digits and letters partitioned inlo 62 classes. We evaluate with 100 random clients from the training set and select N = 50 data points from each. The other dataset is CIFAR100 (Krizhevsky et al. 2200), which consists of 32 x 32 images partitioned into 100 classes. Here, we simply sample 100 batches of size N 50 from the training dataset to form the individual clients' data.
We ran all of our experiments on a single NVIDIA RTX 2080 Ti GPU. The runtimes of different methods and datasets are shown in Appendix D alongside with a discussion on the computational complexity of SIMUPDATE. We apply reconstruction methods on undefended CNN networks at initialization time. We provide additional experiments on defended and trained networks in Appendix C In all experiments, we assume the attacker has knowledge of E, m, and N We relax this assumption in the experiments in Appendix C.4. Throughout the section, FEMNIST attacks that use our epoch prior set the order-invariant function g to the mean of the images in an epoch, while CIFAR100 attacks set it to the pixelwise maximum of the randomly convolved images. We investigate different choices for g in Soction 5.4
We provide the exact.
network architectures and hyperparameters used in our experiments in Appendix B
The rest of the section is organized as follows. First, in Section 5.1 focus image reconstruction with known label counts. In this setting, we compare our attack to using non-simulation--eseed reconstruction methods and justify our algorithm design choices by showing that our method compares favorably to a selection of baselines. Next, in Section 5.2 we experiment with label counts recovery and show that compared (2021) our method is more robust while achieving similar or better label reconstruction results. In Section 5.3 we evaluate our end-to-end attack that reconstructs both the inputs and label counts. Finally,
to Geng et al.
in Section 5.4 we justify our choice for g for the different datasets.
5.1 Input Reconstruction Experiments
In this section, we compare the image reconstruction quality, assuming the label counts per epoch are known (but not to which batches they belong), with methods from prior work (Geiping et al. 2020 Geng et al. 2021 as well as variants of our approach: (i) Ours (prior), our full input-recconstruction method, including the order-invariant prior Liav, as described in Section 4.4 as well as the simulation- reconstruction error Caim that uses separate optimization variables for the inputs at different epochs, (ii) Ours (no prior), same Ours (prior), but not using the epoch prior Linv (iii) Shared (Geiping et al. 2020 , the approach proposed by Geiping et al. (2020) which assumes same order of batches in different epochs in SIMUPDATE, as described in Section 4.3 (allows sharing of optimization variables without using the prior), (iv) FedSGD-Epoch, this
variant simulates the FedAvg update with a single batch per epoch (B" = 1) and thus no explicit regularization is needed, (v) FedSGD (Geng et al. 2021), the approach proposed by Geng et al. (202T), disregards the simulation, and instcad reconstructs the inputs from the average update V8 like in FedSGD.
To deal with the unknown label batch split at the client, for all attack methods pick the same random batch split in Line 4 in Algorithm 2 We report the results for different values of the number of epochs E and batch sizes m in Table I resulting in different number of local client steps U For all experiments, we consider a reconstruction on FEMNIST (CIFAR100) successful if the corresponding PSNR value is > 20 (> 19) and we report two measures: (i) the percentage of images that were successfully reconstructed by the attack, and (ii) the average peak signal-to-noise ratio (PSNR) (Hore Ziou| 2010). which is a standard measure of image reconstruction quality, between the client and the reconstructed images. We note that PSNR is on logarithmie scale, so even small differences in PSNR correspond to large losses in image quality.
9

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: The effectiveness of different methods for reconstructing images with known labels. We measure the percentage of successfully reconstructed images and the average PSNR of the reconstructions.
Ours (prior)
Ours (no prior)
Shared
FedSGD Epoch
FedSGD
Dataset
Ne
U
Rec (%)
PSNR
Rec (%)
PSNR
Rec (%)
PSNR
Rec (%)
PSNR
Rec (%)
PSNR
5
50
10
12.3
19.7
42.3
19.7
42.3
19.7
37.7
19.5
37.7
19.5
5
50
50
63.1
21.2
56.2
20.7
59.0
20.6
56.1
20.4
13.7
17.4
FEMNIST
10
50
63.8
21.3
58.5
20.9
62.0
20.7
60.2
20.6
11.8
17.2
100
65.5
50.6
20.3
59.9
20.3
56.0
19.6
8.3
16.7
1
500
19.3
20.1
21.5
16.7
14.9
16.1
1.6
1.5
5.2
16.6
5
10
9.1
16.2
9.1
16.2
16.2
7.7
16.1
7.7
16.1
5
50
55.2
19.2
53.0
19.1
10.8
16.4
10.9
16.3
1.2
14.2
CIFARICO
10
10
50
50
62.7
20.3
60.1
20.0
23.8
17.2
26.2
17.4
0.6
13.8
10
5
50
100
58.8
19.8
56.4
19.6
15.7
16.4
8.5
15.0
0.4
13.2
10
1
50
500
12.2
15.4
11.0
15.2
0.4
12.5
0.0
8.9
1.3
12.8
For the methods which produce multiple reconstructions of the same image (the two Ours methods and FedSGD-Epoch), we use matching and averaging approach, described Section 4.4 to generate a single final reconstruction. We use linear assignment problem similar to the one in Appendix A match these final reconstructions to the client's original images before computing the reported PSNR values. All results are averaged across all images and users. In Appendix C.3 we further experiment with the setting where both the label counts and their batch assignments are known.
Evaluating reconstructions From Table we make the following observations. First, the full version of our method provides the best reconstructions in essentially all cases. Second, we find that our epoch prior Line improves the results in almost all cases except for the case of a single client epoch (E = 1). We point out that in this case the three methods Ours (prior), Ours (no prior) and Shared, as well as the two methods FedSGD-Epoch and FedSGD, are equivalent in implementation. We obscrve that the use of the prior results in bigger reconstruction gap on the FEMNIST dataset compared to CIFARI00. Our hypothesis is that the reason is the added complexity of the CIFAR100 dataset. In Section 5.4 we further demonstrate that more complex order- inveriii functions 9 can help close this gap. Additionally, we observe that using separate optimization variables in SIMUPDATE sometimes performs worse than having shared variables when no epoch prior is used. However, this is not the case on the harder CIFAR100 dataset or when the prior is used, justifying our choice of using the separate variables. Third, while the FedSGD method performs worse when the number of epochs & is large, all other methods, perhaps counter-intuitivlly, benefit from the additional epochs. We think this is due to the client images being used multiple times during the generation of the client update resulting in easier reconstructions. Finally, our experiments show that FedSGD-Epoch performs well when the number of batches per epoch 8 is small, but its performance degrades as more updates are made per epoch, to the point where it becomes much worse than FedSGD on batch size m = 1.
5.2 Label Count Reconstruction Experiments
We now experiment with the quality of our label count reconstruction method and compare prior work (Geng et al.) 2021). We consider the following methods: (i) Ours, our label count reconstruction algorithm RECLABELS described in Section 4.5 (ii) Geng et al. 8", the label count reconstruction algorithm, as described in Geng et al. (2021), (iii) Geng et al. 0% same Geng et al. 0=, but with the parameters Oc and Pi estimated on the weights 0 returned by the client.
In Table 2 report the numbers of incorrectly reconstructed labels and their standard deviation for different values of the number of epochs E, batch sizes m and different methods averaged across users. Unlike image and text reconstructions where prior knowledge about the structure of inputs can be used to judge their quality, judging the quality of label count reconstructions hard for an attacker as they do not have access to the client counts. To this end, a key property we require from label count reconstruction algorithms that they are robust w.r.t. FodAvg parameters such as datasets, number of epochs E and batch sizes m, as choosing them on the fly is not possible without additional knowledge about the client. label distribution. Results in Table 2 show that the algorithm in Geng et al. (2021). as originally proposed, performs well when the number of local FedAvg steps U is small. Conversely, when approximating the parameters O and P. at the client weights 09, the algorithm works well predominantly when U is large, except on CIFAR100 and U 500.
10

---

Published in Transactions on Machine Learning Research (10/2022)
Table The effectiveness of our method and the baselines for the task of label counts reconstruction.
Dataset
E
Ne
U
Ours
Gong et al. 0"
Geng et al. 0
1
50
10
3.4
2.01
3.2 = 1.84
3.6 + 2.07
5
50
50
3.4
2.10
7.8 6.77
3.2 1.92
FEMNIST
10
50
50
3.2
1.94
8.2 = 7.06
3.2 1.96
10
50
100
5.2
2.98
9.2 = 4.35
5.2 +.25:3555
10
50
500
14.0
6.28
15.4 = 4.73
13.3 + 6.35
10
1.41
3.9 L,30
4.6 1.55
7.6 L83
2.5 1.09
CIFARI00
10
50
1.22
8.1 L.84
2.0 1.000..0
100
1.57
12.2 1.96
5.2 2 1.45
10
50
500
8.1 1.77
10.2 + 1.95
15.7 + 3.48
Table 3: The effectiveness of our method and the basclines for the image and label counts reconstruction.
Ours (prior)
Ours (no prior)
Shared
FedSGD-Epoch
FedSGD
Dataset
E
Ne
U
Rec(%)
PSNR
Rec(%)
PSNR
Ree(%)
PSNR
Ree(%)
PSNR
Ree(%)
PSNR
1
50
10
37.2
19.3
37.2
19.3
37.2
19.3
35.7
19.2
35.7
19.2
5
50
50
51.9
20.3
47.5
20.1
40.8
19.5
47.0
19.8
12.5
17.3
FEMNIST
10
50
55.0
20.5
50.0
20.2
45.7
19.7
50.0
19.9
10.6
17.0
10
100
48.5
20.2
43.0
19.8
43.6
19.3
45.8
19.1
7.2
16.6
10
500
21.4
18.3
14.5
16.9
7.5
15.2
1.1
3.6
16.3
1
50
10
5.0
15.7
5.0
15.7
5.0
15.7
4.8
15.7
4.8
15.7
5
50
50
46.7
18.5
45.0
18.3
7.0
15.8
7.7
15.8
0.8
14.1
CIEARIO0
10
10
50
50
54.3
19.4
52.0
19.1
15.1
16.5
18.8
16.7
0.5
13.7
10
50
100
46.0
18.5
43.2
18.2
8.7
15.7
8.1
14.9
0.6
13.1
10
50
500
6.7
14.7
5.7
14.6
0.4
11.9
0.1
9.0
1.4
12.7
This makes it difficult to decide which variant of Geng et al. results in Table 2 show that our label count reconstruction algorithm performs well across different settings. Further, achieves the best reconotruction the most challenging setting of CIFAR100 with small batch sizes (m 1, and its predictions often vary less than the alternatives. Therefore, we propose our method as the default choice for the FedAvg label count reconstruction algorithm due to its robustness.
2021
to apply in practice. In contrast, the
5.3 End-to-End Attack
In this section, we show the results of our end-to-end attack, that combines our methods for label count and input reconstruction. We evaluate under the same settings and compare it to the same methods as Section 5.1 For all methods under consideration, we use our label count reconstruction method since, as we showed in Section 5.2 performs well under most settings. We present the results in Table 3 Compared to the results in Table 1 (where assumed label counts known), the performance of our end-to-end now reconstrueting the labels as well, showing the effectiveness of
attack remains strong our label reconstruction. In particular, most results remain within 10% of the original reconstruction rate (resulting in > 45% success rate of our attack in most settings) except for the case of U = 500, where due to the higher error rate of the label reconstruction we observe more significant drop performance. Note that even in this setting our attack performs the best and is able to reconstruct 20% of the FEMNIST images successfully. Furthermore, we observe that the Shared method is less robust. to errors in the label counts compared to the other methods, as with known label counts it performs better than the Ours (no prior) method FEMNIST, but the trend reverses when the label counts are imperfect. We think the reason is the sensitivity to label count errors of the shared variables optimization procedure which further justifies our use of separate optimization variables.
though
Reconstruction Visualizations In addition to Table 3 we also provide visualizations of the reconstruc- tions from the different attacks for È 10, m 5, and N 50 in Figure For both FEMNIST and CIFAR, we show the reconstructions of images from the first user's batch. We provide the full batch reconstructions in Appendix F.I We observe that reconstructions obtained using our method are the least noisy. As we show in Appendix C.6. this due the matching and averaging effect.. Further, we see that the prior has positive effect on the reconstruction it can sharpen some of the edges in both the FEMNIST and CIFAR reconstructions. Finally, we also observe that the reconstructions of the FedSGD method very poor.
11

---

Published in Transactions on Machine Learning Research (10/2022)
Table 4: The effectiveness of our method with different priors on 10 5 50 FEMNIST and CIFAR100.
FEMNIST
CIFAR100
Dwsv
Rec(%)
PSNR
Rec(%)
PSNR
mean
61.6
21.2
55.1
19.5
conV + mean
&
61.5
21.2
56.0
19.6
E
36.8
19.2
56.9
19.6
conv + max
f
56.0
20.7
58.0
19.8
mean
E2
65.5
21.5
56.0
19.5
+
82
64.4
21.4
56.3
19.6
maX
82
39.6
19.4
56.0
19.6
+ maX
82
63.6
21.2
58.8
19.8
?
7
0
Original
Ours (prior)
Ours (no prior)
Shared
FedSGD-Ep Epoch
FedSGD
Original
Ours (prior) Ours (no prior)
Shared
FedSGD-Ea MaSSDD-Epoch
FedSGD
Figure 2: Reconstructions of 4 images from the 10 x 5 50 FEMNIST and CIFAR end-to-end experiments.
5.4
Epoch Order-Invariant Priors
In this section, we investigate the effectiveness of various eeder-invariant functions and distance measures Davv We experiment with the f and le distances for Dinv, they are the most natural choices for a distance functions. For g, inspired by PointNet (Qi et al. 2017] that also relies the choice of an order-invariant function, experiment with the mean and max functions. Further, following Li et al. (2022) that show image features produced by randomly initialized image networks can serve as good image reconstruction priors, we also consider versions of that apply the mean and max functions on the result of applying 1 layer random convolution with 96 output channels, kernel size 3 and no stride on the images (denoted with conv + mean and conv + max Table 4 As our performance increased by adding more output convolution channels, we chose the largest number of channels that fits in GPU memory. For both FEMNIST and CIFAR, we ran our experiments on ε 10, m 5, and No 50 with known label counts. We chose Xinv by exponential search the Dinv lo and g mean (Dinv & and g cony + max) combination FEMNIST (CIFAR100). Due to the computational costs, adapted these values of Ainv to the other combinations in a way that the value of Ainv Liv matches on average at the first iteration of the optimization process. Comparison is performed on the same 100 clients as before, as the attacker can select the prior which obtains the highest quality the rectnonruction. The results are shown Table 4 We observe that Dunv = la performs consistently better in all experiments and thus we chose to use it. Furthermore, on FEMNIST g does not benefit from the additional complexity of the convolution, likely due being simpler dataset and that the max is a very poor choice for this dataset. To this end, we choose g to be simply the mean. In contrast, that the random convolution is always helpful to the roconstruction. Also, we see that the combination of the convolution and the max is particularly effective. To this end, we choose to use this combination on CIFAR. We believe more complex feature extracting functions such as ones that use stacks of convolutions or more output channels will have even greater benefit on this dataset, We leave that as future work.
CIFAR observe
6 Conclusion
In this work, we presented new data leakage attack for the commonly used federated averaging learning algorithm. The core ingredients of our attack were novel simulation-bast reconstruction loss combined with epoch order-invariant prior, and our FedAvg-specific extension of existing label reconstruction algorithms. We experimentally showed that our method can elfectively atlack FedAvg updales compuled on combinations of large number of epochs and batches per epoch, thus for the first time demonstrating that realistic FedAvg updates are vulnerable to data leakage attacks. We believe that our results indicate need for more thorough investigation of data leakage in FedAvg and refer to our Broader Impact Statement Appendix that further discusses the practical implications of our work.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Acknowledgments
This publication was made possible by an ETH Al Center postdoctoral fellowship granted to Nikola Konstantinov.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Nya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Mislav Balunović, Dimitar I Dimitrov, Robin Staab, and Martin Vechev. Bayesian framework for gradient leakage. arxic preprint 0rXiv:2111.04 2021.
Franziska Boenisch, Adam Dziedzic, Roci Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Paper- not. When the curious abandon honesty: Pederated learning is not private. Xiv preprint Kkw:2112.02916 2021.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, machine
Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Nocula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings, arXiv preprint arXiv:1812.01097, 2018
Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. Tag: Gradient attack on transformer- based language models. arXiv preprint Xiv:21109. 06819, 2021.
Dimitar Dimitrov, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. arXie preprint arXiv:2202.088: 2022.
Vania Vieira Estrela, Hermes Aguiar Magalhães, and Osamu Saotome. Total variation applications in computer vision. In Handbook of Research on Emerying Perspectives in Intelligent Pallern Recognition, Analysis, and Image Processing, pp. 41-64 IGI Global, 2016.
Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Robbing the fed: Directly obtaining private data in federated learning with modified models. arXiv preprint arXim:2110. 13057, 2021.
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy in federated learning for language models. arXiv preprint ar Xiv:22011 12675, 2022.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-hn how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33: 16937 16947, 2020.
Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chumming Rong. Towards general deep leakage in federated learning. arXiv preprint arXi::2110.0900 2021.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In International conference pottern recognition (ICPR), 2010.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2022)
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trendso in Machine Learning, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint SrXiv:1412.9980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Lin. Auditing privacy defenses in federated learning via generative gradient leakage. arxiv preprint arXiv:2203. 15696, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282, PMLRR 2017.
Luca Melis, Congzheng Song, Emiliano De Cristolaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pp. 691-706.
IEEE, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint ar arx: xxXx:1609.04777, 2016.
Daniel Scheliga, Patrick Mäder, and Marco Seeland. Precode-a generic model extension to prevent deep gradient leakage. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1849 1858, 2022.
Jingwei Sun, Ang Li, Binghui Wang, Huanrui Yang, Hai Li, and Yiran Chen. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pottern Recognition (CVPR), pp. 9311-93155 June 2021.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, îhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Hemriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedrogosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 272, 2020. doi: 1111038//41522-0199
Yuxin Wen, Jonas Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated learning via gradient magnification. arXiv preprint ar arXim:2202.005.0 2022.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through
gradients: Image batch recovery via gradinversion. Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16337 16346, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 2020.
Junyi Zhu and Matthew Blaschko. R-gap: Recursive gradient attack on privacy. arXiv preprint arXiv:2010.07733 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Newral Information Processing Systems (NeurIPS), 2019.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm Overview of our matching and averaging algorithm
1: function MATCHEPOCH(XXq
X f REORDEREPOCH((Y)
3: for < 1,.... No do
4: b; f _
5: I
b; m.m
6:
X +- De=1 X c,
end for
return {X; [
end function
10: function REORDER.EPCH( (X))
11:
for E ,,...,, do
12:
for (i.j) E [N x [N9 do
13:
b; m
14:
by < 1
15:
< m.mm
16:
J + j - -b; m
17:
M, = sim(Xi, I. X X ).
18:
end for order, + LINSUMASSIGX(MM
19:
(Xg, b [B*]] t REORDER((XE, (1EE, 1 [B*]] order.)
20:
21: end for
22: return X
23:
end function
A Matching and Averaging Algorithm
In this section, we present in details our malching and averaging method that takes our per-epoch reconstruc- tions X and combines them into our final input reconstructions {X; j [Nq]. The method is shown Algorithm 3. The matching part of the method is separated in the function RPOcDEREPoCH which computes an optimal reordering of the reconstructions at every epoch e so that its reconstructions match the ones in the first epoch best. To this end, we compute similarity measure between every image Xi. Xf.,. in every batch by in the first epoch and every image Xe e.by. in every batch bj in epoch for every epoch except the first one, and store them in the matrix M (Line 12 18). use the PSNR similarity measure. Once M is computed,
For the image experiments in this paper, we
assignment problem solver LSIGN provided by SciPy (Virtanen et al. 2020 to find the optimal reordering order, of the images in Finally, the matched inputs are then averaged across epochs in the MATCHEPOCH
the linear
epoch e (Linc 20.
function (Line 7
B Further Experimental Details
B.1 Network Details
In Table we show the neural work architectures we use for our FEMNIST and CIFAR100 experiments. They both consists of convolutional layers with average pooling, followed by 2 lincar layers. As CIFAR is a harder dataset than FEMNIST, double the sizes of the layers in the CIFAR architecture.
B.2 Attack Hyperparameters
In all experiments, we use total variation (TV) EEtrela al. 2016) an additional regularizer, similarly to Geiping al. (2020), well as, the clipping regularizer Relip presented in Equation 17 in Geng al.
15

---

Published in Transactions on Machine Learning Research (10/2022)
Table 5: The network architectures for the networks attacked in this paper.
do chianelsm3, out chnntolee 12 aerall 5600m,, stridowl, I, pudillig:)
Convad thomess-i, out chouness BL kernel sleoolll siridew,, packiigg-
A Havel 2l
JugPonliijaazad Jan striloo2)
Comv chomis 32, ond channel-bl,, larnel wouml, prctting
Cnnvellin clar sereewle-me 66, re. chainde 128, lornel sire puihling
RolC(]
(ebUU)
AogPoooi sioo-2, aizss0m2)
AvgPonl Rac-2, strill-
Linear in Sistues--iB4, out Bostures- (00))
Lisoaa (a. leatcres 18868, our features-20)
RelUi
ReLU))
Tiwerfiin 00, fostmmem60)
Tinesçin festuree 200, fasture-100)
(a) FEMNIST network architecture.
(b) CTFAR100 network architecture.
2021). We balance them with the rest of our recomustruction loss using the hyperparameters λTv and Aeip: respectively. Additionally, we use different learning rates Tree and lcarning rate decay factors "ree for solving the optimization problem in Algorithm / For both datasets we use 200 optimization steps and client. learning rate y 0.004 which was suggested good learning rate for FEMNIST in LFAF et al. 2018). We selected the rest of the hyperparameters by optimizing the performance of the FedSGD reconstruction on FedSGD updates with 50 images. We did this on FEMNIST and CIFAR100 separately. The resulting hyperparameters are shown below and used for all experiments.
FEMNIST We use the following hyperparameters for our FEMNIST experiments λTV 0.001, Aelip 2, Ainv 1000, free 0.4, and exponential learning rate decay Yeee = 0.995 applied every 10 steps.
CIFAR100 We use the following hyperparameters for our CIFAR100 experiments λrv = 0.0002, Aetip = 10, Annv = 6.075, Thee = 0.1, and exponential learning rate decay Tree = 0.997 applied every 20 steps.
C Further Experiments
In this section we present a number of additional experiments that further validate the effectiveness of our attack. In all cases, we focus for simplicity on the FEMNIST dataset and we use = 10 epochs and batch size m on N = 50 images. Unless otherwise stated, we assume known label counts per epoch as in Section
C.1 Attacking Defended Networks
First, we experiment with attacking FL protocols that adopt known defenses against data leakage. In particular, consider defending by adding small amounts of Gaussian and Laplacian noise to the commuicatedd gradients. We note that if clipping is additionally applied this is equivalent to the differential privacy defenses in Abadi et al. (2016). Additionally, we experiment with defense that randomly prunes the entries of the gradient. with a certain probability, as suggested by Zhao et al. (2020). For each of the defense methods, we present the results of the attacks in Section 5.1 at two different defense strength levels in Table 6 We chose the defense rengthss such that the resulting networks lose around 1% and 4% accuracy, respectively. We also report the resulting network test-sct accuracies after training for 250 communication rounds, cach using random clients.
As expected, the results show that stronger defenses sacrifice more network accuracy in exchange for more data privacy. Even so, our attack is able to reconstruct > 15% of the images on all strongly defended networks while also being the best performing method across the board. Further, we see that the noise-based defenses provide a better trade-off between accuracy and defense performance. This expected, due to their connection with differential privacy. particular, we see that the Gaussian noise defense results only 17.2% reconstructed images, while achieving 66. 14% accuracy, which results in the best trade-off among the methods considered.
16

---

Published in Transactions on Machine Learning Research (10/2022)
Table 6: The effectiveness the variations of our method and the baselines with known labels and different defenses on FEMNIST with 10 epochs of batch size and 50 images.
Oura (prior)
Ouzs (no prior)
Shared
RedSGDD-poock
FedSGD
Defense
Accuracy(35 (()
Ree(%)
PSNR
Rex(%)
PSNR
Ree(%
PSNR
Roc(%)
PSNR
Rec(%)
PSNR
No Defense
70.09
66.5
21.5
50.6
20.3
39.9
20.3
56.0
19.6
8.3
16.7
Gamsian Naioe a 0.01
68.91
48.0
20.2
36.8
19.1
25.0
18.3
24.9
17.9
7.5
16.6
Gaussian Noise,
15.8
Laplacian Noise, 0.01
68.88
38.9
19.4
29.3
18.4
17.1
15.6
16.7
6.6
16.4
Laplacian Noise, 5 0.02
65.91
19.0
17.0
14.4
15.7
3.7
13.4
5.0
13.5
4.8
15.7
Random Proning, 0.2
68.81
51.5
20.4
40.2
19.5
20.5
19.2
10.8
18.9
9.2
17.0
Random Pruniug, p 0.5
66.62
27.6
18.7
22.3
18.1
18.3
17.5
18.6
17.5
10.2
17.2
Table 7:
The effectiveness of the variations of our method and the baselines with known labels at different training rounds on FEMNIST with 10 epochs of batch size and 50 images.
Ours (prior)
Ours (no prior)
Shared
BLaSGD-Epook
FedSGD
Round
Rec(%)
PSNR
Rec(%)
PSNR
Rec(%) (%)
PSNR
Rec(%)
PSNR
Rec(%)
PSNR
1
65.5
21.5
50.6
20.3
59.9
20.3
56.0
19.6
8.3
16.7
6
58.8
20.8
27.4
16.2
39.2
17.9
22.8
15.6
7.3
15.4
11
38.4
18.0
7.4
13.1
21.3
15.4
10.8
13.2
5.9
15.4
C.2 Trained Networks Experiments
Next, we present our experiments which the attack is conducted further into the training process, rather than during the first communication round. In Table 7 we report the results for all attack methods from Section 5.1 when applied at FedAvg communication rounds 6 and 11 and compare them to the results from the main body (Round=1). We use 10 clients per communication round. While the effectiveness of the attacks is reduced further into the training, method is still able to recover as much as 38.4% of the images at the later stages of the optimization process. In addition, the benefits of our design choices, in particular using the order-invariance prior and epoch-specific optimization variables, become more apparent in this context, as the performance of the rest of the attack methods Table 7 is significantly reduced compared to our full attack.
C.3 Label Order Experiments
Additionally, conduct two ablations studies to evaluate the effectiveness of attacks depending whether the split into batches at each local epoch is random consistent across epochs and whether the server has knowledge of the label counts per batch or not. We present the results of this ablation study in Table S We observe that while the addition of the batch label count information results in consistently better reconstructions, for most methods the gap is not big. This suggests that our choice of randomly assigning the labels into batches works well in practice. The exception is the Shared method that really benefits from the additional information and with it, it is capable of matching the performance of our method in terms of number of reconstructed images even though it still achieves lower PSNR. We believe this is due to the sensitivity of the optimization problem when shared variables are used and motivates our choice of keeping separate optimization variables per epoch when the label counts per batch is unknown. Another somewhat surprising observation in Table S that using random batches actually makes the reconstruction problem slightly easier. We hypothesize that the randomization helps as every image gets batched with different images every epoch, making casier to disambiguate from the other images in the batch least one of the epochs. This suggests that using consistent batches might be cheap way increase the privacy of Avd updates.
17

---

Published in Transactions on Machine Learning Research (10/2022)
Table 8: The effectiveness of the variations of our method and the baselines with known labels depending on the information available the server on FEMNIST with 10 epochs of batch size 5 and 50 images.
Known Counts
Ours (prior)
Ours (no prior)
Shared
FedSGD Epoch
FedSGD
per Batch
Random Batches
Rec(%)
PSNR
Ree(%)
PSNR
Ree(%)
PSNR
Rec(%)
PSNR
Reo(%)
PSNR
68.2
21.7
52.8
21.1
56.0
19.6
16.7
63.4
21.3
49.4
20.1
63.4
20.6
52.7
19.4
7.9
16.7
65.5
21.5
50.6
20.3
59.9
20.3
56.0
19.6
8.3
16.7
X
X
60.7
21.1
46.6
19.9
55.4
20.0
52.7
19.4
7.9
16.7
Table 9:
The effectiveness of the variations of our method and the baselines with known labels and known values of Nc and U but unknown values of & and m. The True columns represent the value used by the clients to compute their FedAvg updale, while the Chosen columus represent the values used by the algorithms during reconstruction.
N=
Ours (priot]
Ours (to petor)
Shared
FedSGD Epoch
PedSGD
Chocee
Chocen
Chocea
Chosen
Bee(S)
PSNN
Rec(S)
PSNR
Bee(%)
PSNR
Ree(S)
PSNR
Bes(%))
PSNR
5
5
50
50
63.1
21.2
36.2
20.7
39.0
20.6
56.1
20.4
13.7
17.4
10
10
50
50
50
55.8
20.7
55.2
20.6
53.8
20.
53.8
20.1
12.7
17.3
25
20.2
51.7
20.3
19:0
39.7
50
50
50
50
50
50
52.0
20.2
51.4
20:2
34.3
18.5
34.6
18.5
10.8
17.0
C.4 Data Reconstruction without Knowledge of and m
In the paper, so far, we assumed that the server has knowledge of E, m, and No when mounting the reconstruction attack. This allows the server further calculate B and U" as well, giving it full knowledge over the clients' dataset parameters. Tn this section, we look at weaker attack model where the server only knows U and N This attack model is motivated by the fact that the FedAvg server only needs U and N to decide how to weight the different client updates when aggregating them, suggesting that the clients can choose not to share & and m. One possible way to mitigate this, is to run our reconstruction with multiple values of & and m and pick the best reconstruction. To this end, we test the robustness of our method to using a wrong value of & in our reconstruction. In particular, we generate FedAvg client updates with = 5, m 5 and No 50 on the FEMNIST dataset and then reconstruct data from them by plugging different values of & and m in algorithm. As U E B is avaliable to the attacker, it is known that E is divisor of U = 50. Thus, choose È to be either 10, 25 or 50, resulting in values for 8 of 5, 2 and 1, respectively. In this experiment, we assume that all batches in the client are of equal size, thus allowing us to compute m using the formula m B The results of reconstructing the data with these (wrong) choices of the parameters & and m for the mothods in Table and known labels are presented in Table In Table g we also present the reconstruction with the correct values of & and m for comparison. We make several observations. First, the image reconstruction becomes worse the further the chosen value of E is from the correct value, 5, for all methods. Despite this, our method still outperforms the baselines and it is capable of reconstructing more than 50% of the images in all cases. Second, our order-invariant prior is less effective when the true value of & is unknown. We think there are two reasons for that. One is that we rely on the value chosen for Ann originally on the correct values of È and m in all experiments in Table 9 which can be suboptimal when & is chosen wrongly. The other is that our prior relies on the knowledge of the correct number of epochs E and therefore less effective when the wrong value of & is used. Finally, unlike our method, we see that the performance of the shared variable method degrades drastically when the choice of & is wrong. This in line with our other experiments where we also see this method is less robust. with respect to changes to the FedAvg client updates.
N
C.5 Reconstructing Data from Aggregated Updates
In this section, we experiment with reconstructing images from updates aggregated between several clients. The experiment was carriod out on the FEMNIST dataset with individual client updates consisting of 10 epochs with 10 batches of 5 images each. The aggregated updates were computed by taking the mean of the client updates participating in the aggregation. We show the results in Table 10 where the first column
18

---

Published in Transactions on Machine Learning Research (10/2022)
Table 10:
The effectiveness of the variations of our method and the baselines for reconstructing data from aggregated updates with known labels on FEMNIST with 10 epochs of batch size 5 and 50 images.
Ours (prior)
Ours (no prior)
Shared
FodSGD- Epoch
FedSCD
# clients
Rec(%)
PSNR
Rec(%)
PSNR
Rec(%)
PSNR
Rec(%)
PSNR
Rec(%) (%)
PSNR
1
65.5
21.5
50.6
20.3
59.9
20.3
56.0
19.6
8.3
16.7
42.1
19.8
32.3
18.7
27.6
18.3
25.7
17.7
6.8
16.1
29.7
18.8
24.8
17.8
14.8
17.0
14.3
16.3
5.7
15.5
2
3
Average
Figure 3: An end-to-end reconstruction of images at individual epochs from the 10 5 x 50 CIFAR100.
represents the number of participating clients. Thus, the first row in Table [0] showing the results of using only client, corresponds to the experiment presented in Table | Similarly to Table we report the average reconstruction results across 100 different aggregated updates and we assume that the total label counts for each client is known, but the counts per batch are not. To account for the aggregated updates, we changed SIMUPDATE to simulate all client updates separately with the procedure described in Section [.3 and then average them to produce the final simulated aggregated update used in Algorithm 2 From the results, we see that more client updates are aggregated we obtain lower reconstruction rates for all of the attacks. However, our attack method reconstructs the most images in all settings compared to the alternatives, while achieving higher PSNR. Further, our method is capable of reconstructing ~ 30% of the images when updales are aggregated across 4 different clients, suggesting that aggregated updates are vulnerable when aggregation with a small number of clients used.
C.6 Importance of averaging across epochs
In this section we visualize the effect of the matching and averaging, described in Section 4.4 used to generate the final reconstructions for our method. In Figure 3 we show the reconstructions at different epochs for our end-to-end attack computed on the same setup as Section (5.4oon/_.4]]]] with = 10, m 5, and N 50. Further, we provide full batch reconstructions with known and unknown label counts in Appendix .2 We make two obsorvations. First, the reconstructions later epochs are of worse quality. We think this duc the imperfect label reconstruction making the simulation worse at later epochs, since in Appendix 2 we show this effect is not seen when the label counts known. Second, while the reconstructions at individual epochs are noisy, the average reconstructs the original images significantly better than any individual epoch reconstruction. This confirms that matching and averaging substantially improves the reconstruction results.
D Computational Complexity
D.1 Runtime and Memory Complexity of SIMUPDATE
The forward pass through SIMUPDATE has exactly the same time complexity as a regular FedAvg update. Thus, has O foow-+bock) time complexity, where T is the number of local weight update steps used compute the original FedAvg update and Toowwbbcck is the time takes compute forward and backward pass through the network single batch of data. Further, SIMUPDATE'S forward pass has O(Ne) memory
19

---

Published in Transactions on Machine Learning Research (10/2022)
Table 11: Runtimes of our method and the baselines on NVIDIA RTX 2080 Ti GPU for the FEMNIST experiments originally presented in Table measured in hours.
Dataset
Ne
U
Ours (prior)
Ours (no prior)
Shared
FedSGD-Epoch
FedSGD
1
50
10
0.50
0.50
0.50
0.25
0.25
5
5
50
50
2.25
2.25
2.00
0.75
0.25
FEMNIST
10
10
50
2.50
2.50
2.50
1.25
0.25
10
50
100
4.25
4.25
4.50
1.50
0.25
10
50
500
7.50
7.50
7.50
1.25
0.50
1
5
50
10
3.50
3.50
3.50
3.00
3.00
5
50
18.00
17.00
17.00
14.50
3.00
CIFAR100
10
10
50
50
35.50
30.50
35.00
28.50
3.00
10
5
50
100
36.00
35.50
33.50
29.00
3.00
10
50
500
7.00
5.50
5.50
29.00
3.50
complexity in the case of shared epoch variables as requires to all client images in GPU memory and OEE No) in the case of individual epoch variables.
D.2 Runtime Comparison
In Table 11 provide comparison between the total runtime of the different variants of our method and the baselines for the FEMNIST experiments originally presented in Table I
Predictably, methods based simulation, while obtaining significantly better reconstructions (See Table D are several times slower to execute than FedSGD. On FEMNIST, the FedSGD-Epocc method presents an interesting tradeoff wetween reconstruction results and speed as its runtime is much closer to the FedSGD method while still being able to reconstruct a big portion of the images in many settings. It is important to nole that in practice the attacker can store the client's updates locally and only later mount the attack outside of the FedAvg learning process. Thus, the time taken to reconstruct the client data. is less important than obtaining the data accurately, provided that the attack itself can be executed in reasonable time.
E Broader Impact Statement
This paper proposes an algorithm for data leakage in federated learning, that is, a method for reconstructing clients' data from their gradient updates only. Given that our attack applicable to the general FedAvg setting, it could be used by real-world FL service providers for reconstructing private data.
We believe that reporting the existence of effective data leakage methods like ours is a crucial step towards making future FL protocols less vulnerable to such attacks. Indeed, understanding the vulnerabilities of currently used algorithms can inspire further work on defending private data from malicious servers and on understanding what FL algorithms provide optimal protection in such scenarios. In addition, our evaluation in Appendix C.1 suggest that existing techniques that add noise to gradients and but come with some reduction of model accuracy, might. be necessary in order protect the data of the participating clients.
F Additional Visualizations
In this section, present the complete batch visualizations of the images reconstructed by our method and baselines, a limited version of which were presented in the main text.
F.1 Reconstruction Visualizations
First we visualize the reconstructions of all the data the first user on the FEMNIST and CIFAR100 experiment with 10 epochs, batch size 5 and 50 images per client. Figures Figure 3 and Figure show the results when the labels are available to the server, while Figure 3 and Figure present the reconstructions for when the labels unknown (similarly to the visualizations in the main body).
20

---

Published in Transactions on Machine Learning Research (10/2022)
We that our attack is able reconstruct many of the images with significant accuracy while being less noisy than the other methods. We also note that, naturally, prior knowledge of the labels counts helps for the image reconstruction and in particular the recovered images are less blurry.
F.2 Epoch Reconstruction Visualizations
We also provide the complete batch visualizations for the experiment in Figure 3 from the main body for both FEMNIST and CIFAR100. Results for when the epoch label counts are known are shown in Figure g and Figure 10 while those for unknown label counts are presented in Figure 9 and Figure 11
We observe that, overall, our reconstruction performs better during the earlier local epochs, similarly to the observations the main text. However, this effect is less expressed for the case when the labels are known, possibly because the resulting simulation more accurate.
Original
Ours (prior)
Ours (no prior)
Shared
FedSGD-Epoch
FedSGD
Figure 4: A reconstruction of the images from the 10 x 50 FEMNIST experiment with known labels on the variations of our method and the baselines.
Original
Ours (prior)
Ours (no prior)
Shared
FedSGD-Fpoch
FedSGD
Figure 5: reconstruction of the images from the 10 x x 50 FEMNIST experiment with label reconstruction on the variations of our method and the baselines.
21

---

Published in Transactions on Machine Learning Research (10/2022)
Original
Ours (prior)
Ours (no prior)
Shared
FedSGD-Epoch
FedSGD
Figure 6: A reconstruction of the images from the 10 x 50 CIFAR100 experiment with known labels the variations of our method and the baselines.
Original
Ours (prior)
Ours (no prior)
Shared
FedSGD-Epodd Epoch
FedSGD
Figure 7: A reconstruction of the images from the 10 x 5 x 50 CIFAR100 experiment with label reconstruction on the variations of our method and the baselines.
22

---

Published in Transactions on Machine Learning Research (10/2022)
2
Average
4
5
Figure 8: A reconstruction of the images at individual epochs from the 10 x 5 50 FEMNIST experiment with known labels by our full method.
2
3
4
Average
5
Figure 9: A reconstruction of the images at individual epochs from the 10 x 50 FEMNIST experiment. with label reconstruction by our full method.
23

---

Published in Transactions on Machine Learning Research (10/2022)
2
3
Average
5
Figure 10: A reconstruction of the images individual epochs from the 10 x 50 CIFAR100 experiment with known labels by our full method.
1
2
3
Average
8
Figure 11: A reconstruction of the images individual epochs from the 10 x 50 CIFAR100 experiment with label reconstruction by our full method.
24


Reviews:
[{"review_id": "5qM0FCCo4t", "reviewer": "Reviewer_JRGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**  \n- The paper is well motivated - existing attacks usually do operate in limited settings.\n- Presentation and writing is generally clean and easy to follow. \n- The paper proposes several novel approaches that are well motivated and successful (compared with the baseline)\n\n**Weaknesses**\n- The exposition of the epoch order-invariant prior could use some improvement. It's not completely clear to me what purpose this serves/the motivation for this additional loss term. \n- Experimentation is on \"simple\" datasets. I realize that resources could be limited, but it really would have been nice to include some results on a more complex dataset such as ImageNet. Previous data leakage attacks have done this, and I'm quite curious about the reconstruction quality in the FedAvg scenario for higher resolution inputs. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors describe how existing attacks against FL pipelines do not operate in the more realistic setting of FedAvg wherein multiple local update steps are taken before parameter updates are shared with the central server. Many existing attacks instead assume a simplified setting where a single local update is calculated and shared. The authors then describe several adaptations/additions to data leakage attacks that improve their operation in the FedAvg setting. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I propose a few main adjustments\n\n- Please add more detail, explanation, and motivation for the epoch order-invariant prior. I would expand section 4.4, as well as 5.4. (critical)\n- Please explain what \"Rec %\" means in your tables. How are you measuring the number of recovered images? (recommended)\n- Include experiments on ImageNet (recommended)", "broader_impact_concerns": "I do not have any concerns about the ethical implications of the work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rihrZ6FWi2", "reviewer": "Reviewer_NCe3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is written is adequately well and is easy to follow. \n2. The design of experiments validates the importance of each building block of the end-to-end attack. \n3. The attack performs significantly better than the existing baselines in various FedAvg attack settings. \n\nWeakness:\n1. Attacker's knowledge on the exact number of local epochs and batches might be unrealistic. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers reconstructing private training data of clients by inspecting their submitted model updates (i.e., gradients). Unlike existing works that assume local clients perform FedSGD, this paper considers the more practical and challenging setting of FedAvg, where the local clients performs multiple steps of model updates (number of batches times number of epochs). This paper proposes improved version of label reconstruction and input reconstruction algorithms, coupled with epoch order invariant prior and epoch matching strategy. Empirically, the proposed method achieves state-of-the-art attack performance, compared to existing baselines designed for FedSGD, in various FedAvg settings.  ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I still do not understand why the attacker can have exact knowledge on the exact number of local epochs and batches. I think, for FedAvg, what the server needs is their product, which will then be used to do weighted averaging over the submitted model updates. Therefore, I think the paper should perform more experiments when the attacker (server) only knows $U^c$, but not the exact local updates and batches. We mainly want to see how sensitive the proposed solution is to the choice of different local updates and batches during simulation. Although I like the overall idea presented in the paper, I believe this adjustment is critical for securing my recommendation for acceptance. \n\nPresentation wise, it will be helpful to highlight the main difference between the proposed one and existing baselines  (e.g., Geiping et al., (2020), Geng et al., (2021) in terms of input and label count reconstruction by comparing them in a side by side fashion. This way, it can better highlight the major contributions in this paper: linear interpolation of local statistics and grouping optimization variable helps a lot. This will simply strengthen the work and is not critical for acceptance. ", "broader_impact_concerns": "The authors addressed the ethical concerns well. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5BC0FUjnz9", "reviewer": "Reviewer_EebW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The paper aims to evaluate an important problem. The FedAvg algorithm is generally more popular than FedSGD. The local steps for communication efficiency is one of the main techniques in FL. Proposing a more practical attack that can work with multiple local steps can help us audit and develop privacy principles. \n\nWeakness\n- The main weakness is the clarity of writing. I am not confident that I understand the algorithm to reproduce the results. \n- I would hope to see more discussions on the attack and defense practice. Specifically, [Huang et al. 2021 Evaluating Gradient Inversion Attacks and Defenses in Federated Learning] suggests that attacks often make unrealistic assumptions and very simple defense methods can have significant impact. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed an algorithm to reconstruct images from model delta (aggregated gradient updates) as an attack for federated learning. The main contribution is that the attack considers model deltas from multiple local steps instead of a single gradient. The proposed method is evaluated on EMNIST and CIFAR-100 datasets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I feel the general writing/organization can be significantly improved. For example,\n\n - In section 4.3, how to get \\tilt{X}? What is the complexity of “SimUpdate”, i.e., reconstruction time and resources? What is the exact reconstruction loss?\n\n- In section 4.5 and section 5, what is U^{c}, and why would it affect the performance in table 1 and 3.\n\n- In section 5, how are “the percentage of images that were successfully reconstructed by the attack” computed?\n\n\nIf I understand correctly, the reconstruction assumes the model updates from each client can be observed. What if only aggregated results from multiple clients (e.g., Federated Learning and Privacy https://dl.acm.org/doi/pdf/10.1145/3500240, https://arxiv.org/abs/2202.00580) can be observed? \n\n\n\n", "broader_impact_concerns": "I would encourage the authors discuss limitations and possible defenses of the proposed method. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers raised important points of concern and clarification on the details, but were overall happy with the direction of the paper. It has been a couple of years since the first work on data leakage from FedSGD. The time is ripe for exploring leakage in more practical scenarios with FedAvg. As this is one of the first papers to specifically address this practical discrepancy, the motivation of the paper is highly relevant. The authors come up with a novel epoch order invariant prior and epoch matching strategy and to perform input and label reconstruction and perform a robust evaluation with FedSGD as the baseline. Reviewers did not take issue with the correctness of the evaluations and mostly focused their concerns on presentation, clarification, or further experiments.

Experiments were performed on small datasets, FEMNIST and CIFAR100, making it harder to judge reconstruction quality. One reviewer requested larger scale (ImageNet) experiments. While this would have made the paper's claims more clear, I don't think it is a show-stopper. ImageNet experiments with 2nd order gradients would indeed require significantly more resources (authors mentioned they're using RTX2080 cards which are for typically for those on a smaller budget), and I think it's reasonable to allow having only small dataset experiments as long as the evaluation is good enough and the ideas are novel.

Another reviewer raised an importance concern that the attacker's knowledge of the epoch count and batch size is too impractical, and suggested that knowledge of only the total number of steps (U^c) is a more realistic situation. The authors were able to respond with experiments in this more realistic scenario and show that the attack remains pretty robust (certainly better than FedSGD) when the attackers guess wrongly the epoch count and batch size. Further, the reviewer that brought this up, NCe3, also agreed that they were satisfied with the authors' response.

Finally, yet another reviewer had the concern about computational cost of this attack. The authors provided new data showing that the computational cost measured via runtime is about 10x higher than the baseline. This info is certainly good to know for downstream analysis of this threat vector, and I see it as something that makes the paper more thorough. I don't see the fact that it takes more compute as disqualifying and it seems the reviewer that brought it up also doesn't think so given their strong recommendation of the paper.

Lastly, reviewers were in agreement that the ethical concerns were handled properly in this paper.

Overall I think this is a good paper and recommend Accept with minor revision.

Minor revisions required:
- remove claims that “the aggregation happens on the clients”
- improve clarity of Algorithm 2 and surrounding text
- incorporate all clarification statements and new data from this rebuttal process into the manuscript (appendix section if space-limited for the data)
Decision: Accept (poster)