Hiding in Plain Sight: Disguising Data Stealing Attacks in Federated Learning
Authors: Kostadin Garov
Dimitar Iliev Dimitrov
Nikola Jovanović
Martin Vechev
Venue: ICLR 2024 poster (2024)
Keywords: Privacy
Federated Learning
Gradient Leakage
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We study detectability of malicious server attacks in federated learning, show that prior attacks are detectable, and propose SEER, a novel attack framework that reconstructs data from large batch sizes and is by design harder to detect.

Abstract: Malicious server (MS) attacks have enabled the scaling of data stealing in federated learning to large batch sizes and secure aggregation, settings previously considered private. However, many concerns regarding the client-side detectability of MS attacks were raised, questioning their practicality. In this work, for the first time, we thoroughly study client-side detectability. We first demonstrate that all prior MS attacks are detectable by principled checks, and formulate a necessary set of requirements that a practical MS attack must satisfy. Next, we propose SEER, a novel attack framework that satisfies these requirements. The key insight of SEER is the use of a secret decoder, jointly trained with the shared model. We show that SEER can steal user data from gradients of realistic networks, even for large batch sizes of up to 512 and under secure aggregation. Our work is a promising step towards assessing the true vulnerability of federated learning in real-world settings.

Published as a conference paper at ICLR 2024
HIDING IN PLAIN SIGHT: DISGUISING DATA STEALING ATTACKS IN FEDERATED LEARNING
Kostadin Garov Dimitar I. Dimitrov
Nikola Jovanović
Martin Vechey?
INSAIT, Sofia University "St. Kliment Ohridski"
EET Zurich
(kostadin. garov, dimitar. iliev. dimi www.iitooeeiiiaiittt.. jovanovic, martin. vertin.veccev)ginff wtt.heeewenn...ehechhhh.
ABSTRACT
Malicious server (MS) attacks have enabled the scaling of data stealing in federated learning to large batch sizes and secure aggregation, settings previously considered private. However, many concerns regarding the client-side detectability of MS attacks were raised, questioning their practicality. In this work, for the first time, we thoroughly study client-side detectability. We first demonstrate that all prior MS attacks are detectable by principled checks, and formulate necessary set of requirements that practical MS attack must satisfy. Next, we propose SEER, a novel attack framework that satisfies these requirements. The key insight of SEER is the use of a secret decoder, jointly trained with the shared model. We show that SEER can steal user data from gradicnts of realistic networks, even for large batch sizes of up to 512 and under secure aggregation. Our work is a promising step towards assessing the true vulnerability of federated learning in real-world settings.
INTRODUCTION
Federated learning (FL, McMahan et al. [2017]) was proposed as a way to train machine learning models while preserving client data privacy. Recently, FL has seen dramatic increase in deployment [McMahan & Ramage; Paulik et al., 2021; FedAI]. In FL,, a server trains a shared model by applying aggregated gradient updates, received from numerous clients.
Gradient leakage attacks A long line of work [Zhu et al., 2019; Geiping et al., 2020; Zhu & Blaschko, 2021; Geng et al., 2021; Yin et al., 2021] has shown that even passive servers reconstruct client data from gradients, breaking the key privacy promise of FL. However, these attacks are only applicable to naive FL deployments [Huang al., 2021]-in real-life settings with no unrealistic assumptions, they are limited to small batch sizes with no secure aggregation [Bonawitz et al., 2016]. In response, recent work has argued that the butrr-uu threat model underestimates the risks of FL, as real-world servers can be malicious or compromised. This has led to malicious server (MS) attacks, which have demonstrated promising results by lifting honest attacks large batch sizes.
Most prior MS attacks rely on one of two key underlying principles. One attack class [Boenisch et al., 2021; Fowl et al., 2022b; Zhao et al., 2023: Zhang al., 2023] uses malicious model modifications to encourage sparsity in dense layer gradients, enabling the application of analytical honest attacks-we refer to these as boosted analytical attacks. Other attacks utilize example disaggregation [Pasquini et al., 2022; Wen et al., 2022], reducing the effective batch size in the gradient space by restricting gradient flow, which permits the use of optimization-benn based honest attacks.
Client-side detectability Nearly all prior work in the field [Geiping et al., 2020; Boenisch et al., 2021; Fowl et al., 2022b; Pasquini et al., 2022: Wen et al., 2022: Fowl et al., 2022a; Chu et al., 2023; Zhao et al., 2023] raised the issue of client-side delectability of MS attacks, i.e., an FL client may be able to detect malicious server activity, and decide to opt out of the current or future rounds. Despite such concerns, no attempts were made to study, quantify, or reduce the detectability of MS attacks.
This work: detecting and disguising malicious server attacks We thoroughly study the question of client-side detectability of MS attacks. We demonstrate that while boosted analytical and example disaggregation attacks pose a real threat as zero-day exploits, now that their key principles are known, all current (and future) attacks from these two classes are client-side detectable in a principled manner, bringing into question their practicality. Notably. we demonstrate the detectability of (the more promising) example disaggregation attacks by introducing D-SNR, novel vuinerability metric.

---

Published as a conference paper at ICLR 2024
We observe that such limitations of prior MS attacks arise from their fundamental reliance the honest attacks they lift. Namely, boosted analytical attacks always require handcrafted modifications which weight space detectable, and example disaggregation attacks rely the success of disaggregation, which is equally evident to any party observing the gradients, i.e., it is gradient space detectable. This illustrates the need for fundamentally different attack approaches.
As a step in that direction, we propose a novel attack framework SEER, which recovers data from batch sizes up to 512, yet is by design harder to detect than prior attacks. Our key insights are that (i) gradient space detection can be evaded using a secrel decoder, disaggregating the data in a space unknown to clients, and (ii) jointly optimizing the decoder and the shared model with SGD on auxiliary data avoids handcrafted modifications and allows for effective reconstruction. Importantly. SEER does not lift any prior honest attack and does not require restrictive assumptions such as architecture tweaking, side-channel information, or knowledge of batch normalization data or labels.
Key contributions Our main contributions are:
We demonstrate that both boosted analytical and example disaggregation MS attacks are detectable using principled checks-for the latter, we introduce D-SNR, a novel gradient space metric of data vulnerability that can protect clients from unintended leakage. We formulate a necessary set of requirements for realistic MS attacks and make the case that detection should become key concern when designing future attacks (Sec. 3).
We propose SEER, novel attack framework which satisfies all requirements based on malicious training of the shared model with a secret server-side decoder. SEER is harder to detect by design as it does not rely on honest attacks, avoiding previous pitfalls (Sec. 4). We provide an implementation of SEER at https://github.comm /ttpa//tw.....oiiiiiiiiitttt///e
We present extensive experimental evaluation of SEER several datasets and realistic network architectures, demonstrating that it is able to recover private client data from batches large as 512, even under the presence of secure aggregation (Sec. 5).
2 RELATED WORK
In this section, we discuss prior work on gradient leakage attacks in federated learning.
Honest server attacks Optimization-baser attacks [Zhu et al., 2019; Zhao et al., 2020; Geiping et al., 2020; Geng et al., 2021; Wu al., 2021; Yin al., 2021] optimize dummy batch to match the user gradient. Analytical attacks [Phong et al., 2018; Kariyappa et al., 2022] recover inputs of linear layers in closed form, but are limited to batch size B and do not support convolutional networks. Recursive attacks [Zhu & Blaschko, 2021] extend analytical attacks to convolutional networks but are limited to B= < 5. Several works thoroughly study all three attack classes [Yue et al., 2022; Balunovic et al., 2022b; Jin et al., 2021; Huang et al., 2021]. Crucially, Huang et al. [2021] show that in realistic settings, where clients do not provide batchnorm statistics and labels, all honest attacks are limited to B < 32 for low-res data, and fail even for B 1 on high-res data. This implies that large B and secure aggregation Bonawitz et al. [2016] are effective protections against honest attacks.
Malicious server (MS) attacks We focus on broadly applicable boosted analytical [Boenisch al., 2021; Fowl al., 2022b; Zhao et al., 2023; Zhang al., 2023] and example disaggregation attacks [Wen et al., 2022; Pasquini et al., 2022], discussed in Sec. 3. Here, we reflect on other MS attacks that study more specific or orthogonal settings. Several studies [Pasquini et al., 2022; Zhao et al.. 2023] require the ability to send a different update to each user, which was shown easy to mitigate with reverse aggregation [Pasquini et al., 2022]. Lam et al. [2021] focuses on the rare setting with participation side-channel data. While we target image reconstruction, some works consider other modalities, such as text [Balunovic et al., 2022a; Gupta et al., 2022; Fowl et al., 2022a; Chu et al., 2023] or tabular data [Wu et al., 2022; Vero et al., 2022]. Further, while we focus on the threat of data reconstruction, Pasquini et al. [2022] studies weaker privacy notions such as membership [Ye et al., 2022] or property inference [Melis et al., 2019]. Finally, sybil- based attacks are a notably stronger threat model orthogonal to our work [Fung et al., 2020; Boenisch et al., 2023]. We further detail our exact threat model in App. B.
2

---

Published as a conference paper at ICLR 2024
3 DETECTABILITY OF EXISTING MALICIOUS SERVER ATTACKS
Most malicious server (MS) attacks rely on one of two strategies based on which we group them into two classes boeesed analytical and example disaggregation. We now discuss client-side detectability of these classes and show that both are detectable with principled checks. We identify the root cause of detectability and formulate necessary requirements that future attacks must satisfy to be practical.
Boosted analytical attacks The works of Boenisch et al. [2021], Fowl et al. [2022b], Zhao et al. [2023], and Zhang et al. [2023] use model modifications induce different variants of sparsity in dense layer gradients, enabling the application of honest analytical attacks to batch sizes beyond one. Applying such attacks to the realistic case of convolutional networks requires highly unusual architectural modifications, i.e., placing a large dense layer in front, which makes the attack obvious. The only alternative way to apply these attacks is to set all convolutions to identity, such that the inputs transmitted unchanged to the dense layer. As this is pathological case that never occurs naturally and requires handcrafted changes to almost all parameters (e.g.. 98% of weights in ResNet 18), this approach is easily detectable by inspecting model weights (e.g., by searching for convolutional filters with single nonzero entry, see App. A). More importantly, high levels of transmission are, in fact, impossible in realistic networks due pooling and strides [Fowl et al., 2022b], and further attempts to conceal the changes (e.g., by adding weight noise) would additionally worsen the results.
Example disaggregation attacks While the detectability of boosted analytical attacks was rec- ognized in prior work [Geiping et al., 2020; Boenisch et al., 2021; Wen et al., 2022], example disaggregation attacks [Wen et al., 2022: Pasquini et al., 2022] are considered more promising. These attacks use model modifications to restrict the gradient flow for all but one example, causing the ag- gregated gradient of a batch to be equal to the gradient a single example. This undoes the protection of aggregation and allows the attacker to apply honest optimization-basee example. While most instantiations of example disaggregation attacks rely on unusual handcrafted parameter changes, which detectable in the weight space (as for boosted analytical attacks), it may be possible to design variants that better disguise the gradient flow restriction. For this reason, we focus on a more fundamental limitation of all (current and future) example disaggregation attacks and demonstrate it makes them easily detectable in the gradient space. Moreover, such detection is possible without running costly optimization based attacks by using simple principled metric.
attacks reconstruct that
We now propose one such metric, the disaggregation signal-to to-ni- ratio (D-SNR). Assuming the use of the standard cross-entropy loss L(x, y), a shared model with parameters 0, and batch of data D = (1),. (I Un)], we define D-SNR as follows:
aC(2..2) ow
ma b}
D-SNROO.
max WEO
(1)
Ei=
dC(x aw
OC(z.v)) aw
max,e(1,....bb
where on denotes the set of weights of all linear layers (dense and convolutional; 98% of ResNet18). 88). Intuitively, D-SNR searches for layers where the batch gradient (the average of example gradients) is dominated by the gradient of a single example, suggesting disaggregation. We conservatively use max to avoid false negatives and account for attempts at partial disaggregation, i.e., there is any layer that disaggregates single example, D-SNR will be large, and the client may decide that their batch is vulnerable and skip the current training round. While we focus on the case of disaggregating a single example, our approach can be easily generalized to any number of examples.
We use D-SNR to experimentally study the detectability of example disaggregation attacks in realistic settings (see App. A for experimental details). As D-SNR is always oc for attacks proposed by Wen et al. [2022], we modify them in an attempt to smoothly control the strength of the gradient flow restriction. Our key observation, presented in Fig. (red x), is that in all cases where the attack is successful, D-SNR is unusually large, making the attack easily detectable. Reducing the strength of the gradient flow restriction further causes a sharp drop in D-SNR, entering the range of most non-malicious networks (blue x), i.e., the attack is undetectable. However, in all such cases, the attack fails, as the aggregation protects the examples. In rare cases (e.g., when overfitting), even natural networks can produce high D-R and be flagged. This behavior is desirable, as such networks indeed disaggregate a single example, and (unintentionally) expose sensitive user data. Thus, metrics such D-SNR should be interpreted detecting vulnerability, and not necessarily maliciousness.
3

---

Published as a conference paper at ICLR 2024
Doaraple Disuggregation
(Priie Work)
SEER Method)
x
Non-Malliioos Networko
x X00%
XB0X XOHDOWOOG
XXC XX
X
Disaggregation Sigml-to--oise Ratio (D-SNR; log scale)
Figure 1: D-SNR (Sec. 3) of real (model, data batch) pairs. High values indicate vulnerability to data leakage, which can manifest even in malicious models (X). Example disaggregation attacks (x) are easily detectable as they can successfully reconstruct data only when DSNR is unusually high (note the log scale), and fail otherwise an example even when D-SNR is low
Our method, SEER (x, Sec. 4), successfully reconstructs and is thus hard to detect in the original gradient space.
Requirements for future attacks Our results show that all prior MS attacks are client-side de- tectable using generic checks. We argue that this is caused by fundamental problems in the design principles of the two attack classes and can not be remedied by further refinements. Any attempt to lift an honest analytical attack will inherit the limitation of being inapplicable to convolutions and will require architectural changes or handcrafted modifications detectable in the weight space. Lifting optimization baasbd attacks always requires example disaggregation, which is gradient space detectable. More broadly, as information needed to execute these attacks is in the user gradients, the server has no informational advantage and no principled way to conceal the malicious intent.
This suggests that new attack principles are required to better exploit the potential of the MS threat model. To help guide the search, we now state the necessary requirements for future MS attacks guided by our results above and observations from prior work [Wen et al., 2022; Huang et al., 2021], We argue that realistic MS data stealing attacks for image classification should: (i) target realistic deep convolutional networks with large batch sizes and/or secure aggregation; (ii) only utilize the attack vector of weight modifications, with no protocol changes (e.g., standard architectures, asymmetric client treatment) and no sybil capabilities; (iii) not assume unrealistic side information, such as batch normalization statistics or label information [Huang et al., 2021]; J; and (iv) explicitly consider the aspects of weight and gradient space detection (e.g., avoid obvious handcrafted modifications).
4 SEER: DATA STEALING VIA SECRET EMBEDDING AND RECONSTRUCTION
In this section, we propose SEER, a novel attack framework that steals data from large batches while satisfying the requirements in Sec. 3. SEER avoids both pitfalls of prior MS attacks that caused them to be detectable. Namely, SEER does not lift any honest attack and evades gradient space detection by disaggregating the data in hidden space defined by server-side secret decoder. As result, SEER-trained networks (green x) have D-SNR values indistinguishable from those of natural networks (Fig. 1). Further, SEER does not use handcrafted modifications, and instead trains the shared model and the secret decoder jointly with SGD, evading weight space detection.
Overview Once trained, SEER is mounted as follows (Fig. 2). As in standard FL. the client propagates their batch (X, y) of B examples (æ;; yi) through the shared model f with parameters 0, sent by the server, and returns the gradient g (for simplicity we assume FedSGD) of the public loss i w.r.t. Oy. When B > 1, aggregates gradients g; of individual examples, i.c., g = (1/B) Ei=1 9:- When secure aggregation is used (discussed shortly), the sum also includes gradients of other clients.
The server's ' goal is to break this aggregation. To this end, the server feeds to secret decoder consisting of a disaggregator d, followed by reconstructor r. Crucially, d is trained to project g onto a hidden space in which the gradient projections of all images not satisfying some property P are removed. While the exact choice of P is not essential, the goal is that for most batches only one batch example satisfies P. In Fig. 2, P = "iimages with brightness at most 7" with chosen only 35 satisfies it, allowing d to extract the projected gradient d(gs), and r to steal the client image æs-
To train SEER, the server chooses P and interprets 0,, Oa, and O, an encoder-decoder framework, trained end-to-end using auxiliary data to simulate a real client. The goal of training is for d to nullify the contributions of images not satisfying P (Cnui in Fig. 2). and for to reconstruct the image satisfying P from the output of (Crc). The shared model f is also trained to encode client data in the gradient space in way that supports the goals of disaggregation and reconstruction.
4

---

Published as a conference paper at ICLR 2024
Client:
Lul
P
Gradient Encoding
gt. 922
93
gs
Leo
21
0
1
Od
X
X
22:
DEC (.).)
80,
Disageregator
d(gs)
Reconsteuctor
X
Server:
Gradient Decoding
Overview of SEER. A client propagates batch X (of which one image satisfies the property P only known to the server) through the shared network f with malicious weights f and returns the aggregated gradient g. hoping that the aggregation protects individual images. The server steals the image satisfying P by applying secret disaggregator d to remove the impact of other images a hidden space, followed by secret reconstructor SEER is trained by jointly optimizing 0f. Oa. and 0, to minimize a weighted sum of Caul and Lree.
Figure 2:
4.1 KEY COMPONENTS OF SEER
We next describe the individual components SEER more detail.
Selecting the property P Let mul C [B] = (1...., B} denote the set of examples in the client batch that do not satisfy the secret property P. and Liee S [B] the of those that do. Following Fowl et al. [2022b], we use properties of the type m(x) < T, where m represents some image measurement, e.g., brightness, and chosen maximize P([ = 1). Similarly to Fowl al. [2022b], in our experiments we use m based on brightness and color, but we emphasize that the attacker can use many different types of m (as we experimentally demonstrate App. F.6), i.e., this choice is non-restrictive and is simply way to single out images from big batches.
Fowl et al. [2022b] propose the global setting, where the server sets single value of 7 for all client batches. In particular, they choose such that the probability of satisfying P is 1/B on the distribution of m over the whole space of input examples. Wen et al. [2022] later showed that this choice is optimal in the global setting, resulting in P@ improve upon this by allowing to be dependent on the client batch data (X, y). i.e., we propose a local setting. We make the novel observation that when batch normalization (BN) is present (like in most convolutional networks), we can choose P with respect to the local (in-batch) distribution of m, e.g., the minimal brightness in every batch. We find that SEER can be trained such local P with auxiliary data, empirically achieving P((rce 1) > 0.9 for B as large as 512, which is significant improvement over the probability of 1/e achieved the global setting. Our method is enabled by the fact that each BN layer normalizes the distribution of its input, intertwining the computational graphs of images in the batch which are otherwise independent. Unlike prior work, our local properties P. allow the attacker to, most of the time, steal the client data after only a single communication round.
1) - 1/e from above B - co. We
We note that secure aggregation [Bonawitz et al., 2016] is more challenging, and not equivalent to a large batch from one client when BN is present, an aspect overlooked in prior work. To overcome this, we design more elaborate P that combines local and global properties, resulting in P([ece| = 1) 1/e, in prior work. We provide detailed explanation in App. C, and in our experiments in Sec. evaluate both large batch and secure aggregation variants SEER.
Training 0, for suitable gradient encodings Training the shared model weights O, along with the secret decoder is essential for the success of SEER. Intuitively, we can interpret the client-side gradient computation as a latent space encoding of the client data. The failures of honest attacks, discussed in Sec. 2, suggest that the gradient encoding often lacks the required information to reconstruct user data. Our key observation is that the MS threat model uniquely allows to overcome this issue by controlling the gradient encoding by tuning 0y. particular, we maliciously optimize 0, with SGD to allow the recovery of single example by the other modules of SEER, regardless of the information lost at the encoding step. While Zhang et al. [2023] also considered optimization- based modifications with auxiliary data, their approach still inherits the fundamental limitations of all boosted analytical attacks, requiring additional handcrafted modifications which, as noted in Sec. 3, are easily weight space detectable- an issue that SEER circumvents by design.
5

---

Published as a conference paper at ICLR 2024
Training 0₂ for secret disaggregation The secret disaggregator d addresses the key limitation of example disaggregation attacks (discussed in Sec. 3), i.e., disaggregating examples in the gradient space. In contrast, d embeds the gradients from g into a lower-dimensional space R using secret linear map Od concealing the disaggregation in the original gradient space. The benefits of using such linear map are twofold. First, the linear map commutes with gradient aggregation due additivity. Second, the lower-dimensional space allows us to more easily drive the projected gradients of !ul to which happens when they are in or close to the null space of Og. Combining the two properties ((i) and (it) in Eq. 2) ideally allows us to retain only the chosen sample from the aggregated gradient g:
B
B
d(g) = d(
9:
Σ
d(g.) = Σ d(g:) +
Σ d(g.) i
d(g.)-
(2)
TT
A
To achieve this in practice, f and d should be set such that d(g:)) x 0 for all i Amu. while tolerating d(g.)) o for the single i E Lrce To this end, for chosen discussed above, we define the following objective:
Caul Σ [l 12:
which SEER aims minimize during training. We ensure that this does not also nullify d(g,) for the example of interest in Lree, So is able recover that example from d(g:), ), as described next.
Algorithm 1 The training procedure of SEER
Training O, for image reconstruction The final component of SEER we discuss is the secret reconstructor r: Rd - R"r, which receives d(g). i.e., the (noisy) iso- lated embedding of the target image gra- dient, as seen in Eq. 2. The reconstruc- tor aims to map d(g) back to the original image Tree. effectively stealing that exam- ple from the original batch, compromis- ing client privacy. To this end, define the following & reconstruction objective, which is odds with Laut:
1:
function TRAINSEER(F,
B, X, ))
2: Choose P, initialize d and r
3: while not converged do
4:
X, y { yi (X, 27 i [B]}
5:
Imul, Iree f- P(X, ,))
6:
Xual: Vaul X Mmal) y [mml]
7:
Xiee Yree +
[ree [ree)
8:
Imul: Frec BPP, l, Xuul, Xico Youul Mroe)
9:
L f- d(gmul)
D. Eq. 3
10:
Lree |((((rre))
X
Eq. 4
11:
C + Lroe + a Lmu)
> Eq. 5
oc Vm (f,,d, r)
Om Om Pm 30m end while
(4) Leec = (I - Dree
14:
return f.d.r
The final loss function of SEER weighs the two losses using a hyperparameter a > 0:
15:
function BP(f, L, X Xmul: Xrees Youl: Broe)
16:
[n Trc]] f- {[ Xreel)), [mol yroel)
L = Lree + a Lmul-
(5)
0g. BL..
17:
return 30,- 30,
All three key components of SEER jointly trained to minimize L.
4.2 END-TO-END ATTACK DESCRIPTION & DISCUSSION
Algorithm 1 describes the training of SEER. We train on client-sized batches (see App. F.3 for related study) sampled from our auxiliary data (Line 4). Based on P, we select the index sets Inu and I (Line 5), representing the examples we aim disaggregate. Then, we simulate the client updates gree and Inai computed on the full batch X (Line 8), and use them to compute our optimization objective (Line 11). We minimize the objective by jointly training f, d, and r using SGD (Line 12).
Mounting SEER once the malicious weights Oy have been trained using Algorithm is simple, as we illustrate Al- gorithm 2. The server, during an FL round, sends the client the malicious model f (Line 2), and receives the gradient update g. Then, it applies its secret disaggregator d and reconstructor r (Line 3) to obtain x Paolens the reconstructed private example from the client batch.
Algorithm 2 Mounting SEER
1:
function MOUNTSEER(S, d, r)
2:
g +-EECLIENTUPDATEE
3:
Estolen f- r(((g))
4:
return Estolen

---

Published as a conference paper at ICLR 2024
Table I: Large batch reconstruction for different batch sizes B. The metrics are introduced at the top of Sec. 5. Results with more settings (CIFARIO0, Bright and CIFARIO, Red) are given in App. F.I.
CIFAR1O, Bright
CIFAR100, Red
B
Rec (%)
PSNR-Top
PSNR-AIl t
Rec (%)
PSNR-Top 1
PSNR-AII 1
64
89.4
32.1 +: 2.0
5.3
97.1
31.7 + 1.1
29.0 + 3.4
128
94.2
31.9 1.7
28.2 4.3
97.4
31.8 1.1
29.3 + 3.2
256
93.5
32.8
2.0 2.0
28.5 + 5.0
97.7
31.3
28.6 4.2
512
87.8
26.6 1.8
23.2 + 3.5
98.6
33.1
#111
30.5 + 3.1
SEER satisfies all requirements We now reflect on the requirements listed in Sec. 3 and discuss how SEER satisfies them. First, SEER does not utilize any attack vector apart from maliciously modifying the weights of f. does not assume unrealistic knowledge of BN statistics batch labels, and makes no assumptions regarding label distributions, in contrast with some prior work [Yin et al., 2021; Wen et al., 2022]. We remark that the necessity of such side information is the artifact of optimization-baser attacks, and another reason why approaches that do not attempt to lift honest attacks (such as SEER) may be more promising. SEER was greatly influenced by the assumption that clients will inspect the models, aiming to detect malicious updates. Namely, SEER avoids weight space detectable handcrafted modifications and introduces secret disaggregation as means to also avoid gradient space detection. As we show in Sec. 5, SEER successfully steals client data on realistic convolutional networks with large batch sizes and secure aggregation, demonstrating its practicality.
EXPERIMENTAL EVALUATION
In this section, we present our experimental results, demonstrating that SEER effective at recon- structing client images from realistic networks, in both large batch and secure aggregation settings. These results are especially valuable given the important advantages of SEER over prior work in terms of satisfying the requirements for practical attacks (Sec. 3), as we have discussed in Sec. 4.
Experimental setup
We use ResNet18 [He et al., 2016] in all experiments. We use the CIFARIO dataset, well as CIFARI00 [Krizhevsky al., 2009] and ImageNet [Deng et al., 2009], to demon- strate the ability of SEER to scale with the number of lahels and input size, respectively. We generally use the training set as auxiliary data, and mount the attack on randomly sampled batches of size B from the test set for CIFAR 10/100 and validation set for ImageNet. We further experiment with auxil- iary datasets of different sizes in App. F.8, and clients with different heterogeneity levels in App. F.9 where we show that SEER is highly effective even when only small amount of auxiliary data is available and when clients data is highly heterogeneous. We run all experiments on a single NVIDIA A100 GPU with 40GB (CIFAR 1 10/100) and 80GB (ImageNet) of VRAM. Each CIFAR experiment took < 7 GPU days to train and- < 1h to mount on 1000 batches. The ImageNet model trained for 14 GPU days. with 0.5h to mount the attack on 100 batches. In our CIFAR experiments, we set r to linear layer and subsume d in it. For ImageNet, we use a linearized U-Net decoder [Ronneberger et al., 2015] (see App. G). We defer additional implementation details to App. H and App. I.
In all experiments, we use the properties of maximal brightness (Bright) and redness (Red), training separate malicious weights for each (dataset, property, batch size) triple. We report 3 reconstruction quality metrics: (i) the fraction of good reconstructions (Rec), i.e., batches where reconstructions have PSNR > 19 [Horé& & Ziou, 2010] to the ground truth; (ii) the average PSNR across all attacked batches (PSNR-All;; and (iii) the average PSNR for the top 1 ~ 37% of the batch reconstructions (PSNR-Top) that allows to compare SEER in large batch (1 client) and secure aggregation (many clients) settings. We provide experiments with more properties and metrics in App. F.6 and App. J.
Large batch reconstruction on CIFAR10/100 the full results deferred to App. F.1 and follow similar trends. We make several key observations. First, in most experiments, the use of local properties (see Sec. 4. allows us to steal an image from most batches (up 98.6%). greatly improving over 1/e% achieved by prior work. Second, we obtain good reconstructions for both Red and Bright property (average PSNR up to 30), which confirms that SEER can handle a diverse set of properties, and that property choice is not crucial for its success. Finally, SEER successfully steals images even from very large batch sizes such as 512, showing clear degradation in performance. On top of these quantitative results, we show example reconstructions in Fig. 3 (left, C' = 1), visually confirming their quality.
subset of our main results is shown in Table 1;
7

---

Published as a conference paper at ICLR 2024
Ground Truth
oriece
Ground Truth
C=1
dehec
C=1
C-8
Figure 3: Example reconstructions of SEER with 128 total examples and different number of clients C' on CIFAR1O (Left) and 64 examples on ImageNet (Right), both using the Bright property.
Large batch reconstruction on ImageNet To show scalability high-res images, train SEER on ImageNet with B = 64 and the Bright property (due to high computational costs, we leave more thorough studies of ImageNet to future work). To speed up convergence,
pretrain parts of the transposed convolution stack alongside f on downsized images, and use it as initialization. We obtain PSNR-AIl of 21.9 + 3.5 and PSNR-Top of 25.1 + 2.5, corresponding 82.1% successfully attacked batches, Our results are significant, as these PSNR values on Imagenet represent very high quality reconstructions and are higher than the state-of-the-art attack in Wen et al. [2022]. This further visually confirmed by the example reconstructions we show in Fig. 3 (right). From the recovered images we conclude that SEER can be efficiently instantiated on high-resolution images, resulting in very detailed reconstructions that allow the identification of complex objects and individual people, constituting a serious violation of privacy. We note the significance of these results, as stealing even single ImageNet image is impossible with honest attacks without restrictive assumptions [Huang et al., 2021]. We see these results as an encouraging signal for general applicability of SEER.
Secure aggregation In Table 2, we present the results of CIFARIO experi- ments with secure aggregation with C' = 4 and C 8 clients and batch sizes cho- sen to match the total number of images (#Imgs) as Table Most importantly, as before, SEER consistently obtains image reconstructions with average PSNR > 25, i.e. recovers most images almost perfectly.
Table 2: CIFAR10 RI0 reconstruction with secure aggrega- tion, varying the number of clients (C) and total images (#Imgs). See App. F.2 for results with another property.
4, Bright
Bright
#Imgs
Rec (%)
PSNR-Top Top
Rec (%)
PSNR- Top
64
41.4
27.3 + 3.1
41.3
26.6 +:
128
44.2
26.8 3.0
40.6
27.3 + 3.3
Comparing to Table 1, the success proba- bility degrades with C, confirming our in- tuition (see Sec. 4) that secure aggregation provides additional protection in the presence of BN, compared to simply using large batches. Despite this, the success probability Rec is significantly higher than 1/e% of prior work. We suspect this due the model learning restricted version of single-client reconstruction for each client, and further compare the two variants in App. F.5. We note that Rec rises with the number of images, which we believe due to the better estimation of the property threshold for larger batches. Finally, in Fig. 3 (left), we can visually compare results for different number of clients, noting no obvious degradation, which reaffirms that SEER can breach privacy even when secure aggregation is used.
256
51.9
27.3 + 2.5
41.9
25.4 3.1
512
52.9
25.7 2.4
51.7
25.9 + 2.8
Robustness to distribution shifts A ques- tion that naturally arises is if the need for aux- iliary data restricts the applicability of SEER. In this experiment we show otherwise, demon- strating robustness to distribution shifts be- tween the attacker's auxiliary dataset (Da) and the client dataset (D.). i.c., an attacker can suc- cessfully mount SEER without the knowledge of De., relying only on public data. We set C' = 1, 128, and D. = CIFARIO and explore several options for D.. illustrating dif- ferent levels of shift. Namely, CIFAR 10. 1v6 [Recht et al., 2018] and CIFAR10.2 [Lu et al., 2020] represent naturally occurring shifts of the data source, TinylmageNet [Le & Yang, 2015] (mapped to 10 classes) models different data sources for D, and D.. and ISIC2019 [Tschandl et al., 2018;
Table 3: SEER is robust to distribution shifts be-
tween the auxiliary dataset (CIFARIO) 10) and the client dataset D... We use B 128 and the Red property.
De
Rec (%)
PSNR Top 1
PSNR-AII 1
CIFARIO
93.5
31.1 + 1.2
27.8 + 4.1
CIFAR10.1v6
96.0
31.6 + 1.0
28.4 3.8
CIFARIO.2
90.2
31.611
27.5 5.0
TinyImageNet
80.2
27.6
2..7
ISIC2019
98.0
29.1
2.84
Codella et al., 2018; Combalia et al., 2019] models
severe domain shift between D,, and D..
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison between SEER and prior state-of-the-art MS attacks.
Method
Und-Ret (%)
PSNR-Und Rec
PSNR-Und
Rec (%)
PSNR-AII f
Fishing 3 400
20.2 0.5
17.1 2.3
77
21.7 = 3.2
Fishing / 100
8
20.6 1.6
16.4 2.9
63
20.2 20.2+4.11 4.1
Fishing B 55
19.4 + 0.3
15.5 + 2.2
52
19.4 4.5
Fishing 3 12.5
22.4 1 0.0
13.7 2.0
8
14.5 3.4
Zhang23
0
N/A
N/A
5
15.8 + 1.8
LOKI SEER
N/A
N/A
100
143.4
+ 10.3
90 24.6 + 2.2
23.8 + 3.3
90
23.8 = 3.3
The results are shown in Table 3. We observe no degradation for CIFAR10.1v6 and CIFARI0.2, confirming that SEER can handle naturally-occurring data shifts. For TinyImageNet and ISIC2019, despite the large discrepancy to CIFARIO in image and label distributions, we observe high quality reconstruction on 80% and 98% of images, confirming that SEER is not limited by the choice of D We further investigate the robustness to batch size mismatch in App. F.3 and corruptions in App. F.4.
Comparison to prior MS attacks We compare SEER to state-of-the-an MS attacks: (i) Fish- ing [Wen et al., 2022], an example disaggregation attack; (ii) Zhang23 [Zhang et al., 2023], a boosted analytical attack; and (iii) LOKI [Zhao et al., 2023], boosted analytical attack that relies stronger threat model that permits architectural changes and sending different models to clients. We attack 100 batches CIFARIO, 10, with C 1, B 128. We use the Red property for SEER. As in Sec. 3, we explore different variants of Fishing by varying the parameter 3 which should control the grennth-detectabilii tradeoff. We report the usual metries Rec and PSNR-AII, the percentage of undetected successful attacks (Und-Rec) based on D-SNR (Sec. 3) and T-SNR (App. A), as well as the average PSNR of all undetected reconstructions (PSNR- Und), and the successful undetected reconstructions (PSNR- Ciie-Undd-KKe Rec). We provide more details about the experimental setup in App. I.5.
The results are shown in Table 4. Setting detectability aside, SEER outperforms all methods but LOKI, while also being very fast to mount (<2 sec per batch). We emphasize that LOKI's performance is largely due to its architectural changes to the ResNet which are trivially detectable by clients and crucial to the application of the method. We also observe that Zhang23 fails to recover most images at all due to the stride>1 and pooling in realistic networks that cause severe downscaling of the image fed to the attacked linear layer discussed in Sec. 3. Crucially, only tiny fraction of successful Fishing attacks are undetected, while other prior methods completely fail to avoid detection. In contrast, for SEER all successful attacks remain undetected. Finally, we confirm our observation from Fig. that prior MS attacks need to jeopardize reconstruction quality to avoid detection, as for Fishing PSNR-Und is well below PSNR-AIl for all values of B. Our experiments reaffirm that the reliance on honest attacks of prior MS attacks makes them easily detectable, and thus unrealistic.
6 OUTLOOK
While SEER is a powerful attack that can harm user privacy, we believe our work opens the door to a more principled investigation of defenses, as it illustrates that techniques such as secure aggregation are not as effective as previously thought. To mitigate attacks like ours, prior work has discussed cryptographic techniques like SMPC or FHE, which are still largely impractical [Kairouz et al., 2019], as well as differential privacy methods, which we demonstrate in App. E to not be effective enough.
Thus, we believe that principled client-side detection is the most promising way forward. While SEER avoids pitfalls of prior attacks which make them easily detectable, and we see no clear ways to detect it currently, more mature detection techniques may be able to so. We encourage such work and advocate for efficient and robust checks based on attack categorization (such as in this work), as opposed to ad-hoc detection which attacks can easily adapt to. On the attack side, interesting future directions include other data modalities and model architectures and improving SEER's 'RER' training cost.
7 CONCLUSION
In this work, we explored the issue of client-side detectability of malicious server (MS) attacks in federated learning. We demonstrated that all prior attacks are detectable in a principled way, and proposed SEER. novel attack strategy that by design avoids such detection while effectively stealing data despite aggregation. Our work highlights the importance of studying attack detectability and represents a promising first step towards MS attacks that compromise privacy in realistic settings.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
As noted in Sec. 6, the attack introduced by this work, SEER, advances the capabilities of attackers aiming to compromise client privacy in FL. Further, as Wen et al. [2022] point out, attacks like ours based on property thresholding can lead to disparate impact, affecting outlier groups more severely as their inputs are more likely to be reconstructed. However, we believe that our principled investigation of detection and the emphasis on realistic scenarios, as well as making the details of our attack public and open souree (which impact, as they open the door to further systematic studies defenses, and help practitioners better estimate the privacy risks of their FL deployments and avoid the common error of underestimating the vulnerability.
intend to do after publication), both have a significant positive
ACKNOWLEDGMENTS
This research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
REFERENCES
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, 2016.
Mislav Balumovic, Dimitar Dimitrov, Nikola Jovanovic, and Martin T. Vechev. LAMP: extracting text from gradients with language model priors. In NeurIPS, 2022a.
Mislav Balunovic, Dimitar Iliev Dimitrov, Robin Staab, and Martin T. Vechev. Bayesian framework for gradient leakage. In ICLR, 2022b.
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv, 2021.
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. Reconstructing individual data points in federated learning hardened with differential privacy and secure aggregation. arXiv, 2023.
Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. NIPS, 2016.
Hong-Min Min Chu, Jonas Geiping, Liam H Fowl, Micah Goldblum, and Tom Goldstein. Panning for gold in federated learning: Targeted text extraction under arbitrarily large-scale aggregation. ICLR, 2023.
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168-172, 2018. doi: 10.1 10901011..000 S.T.SSS..001..3666 8.3633477
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana. Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, al. Ben20000: Dermoscopic lesions in the wild. arXiv preprint arXiv: 1eXivv/9900 190.999880.88.. 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale acagesscale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
FedAL. Utilization of fate in risk management of credit in small and micro enterprises. https//www.fedai.org...
10

---

Published as a conference paper at ICLR 2024
Liam Fowl, Jonas Geiping. Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy in federated learning for language models. ICLR, 2022a.
Liam H. Fowl, Jonas Geiping, Wojciech Czaja, Micah Goldblum, and Tom Goldstein. Robbing the fed: Directly obtaining private data in federated learning with modified models. ICLR, 2022b.
Clement Fung, Chris J. M. Yoon, and Ivan Beschastnikh. The limitations of federated learning in sybil settings. In RAID, 2020.
Jonas Geiping. Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? NeurIPS, 2020.
Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chunming Rong. Towards general deep leakage in federated learning, arXiv, 2021.
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models. In NeurlPS, 2022..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint arXiv: 0060070...7..... 2018.
Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference Pattern Recognition, pp. 2366-2366 2010. doi: 10.1 10.110//CPP.201005777
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses federated learning. NeurIPS, 2021.
Xiao Jin, Yu Chen, Chia-Y Yi Hsu, Chia-Mu Yu, and Tianyi Chen. CAFE: catastrophic data leakage in vertical federated learning, arXivv, 2021.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode. Rachel Cummings, Rafael G. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tanerède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramer, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu. Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. arXiv, 2019.
Sanjay Kariyappa, Chuan Guo. Kiwan Maeng, Wenjie Xiong, G. Edward Suh, Moinuddin K. Qureshi, and Hsien-Hsihn S. Lee. Cocktail party attack: Breaking aggregation-base privacy in federated learning using independent component analysis. arXiv, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Maximilian Lam, Gu- Yeon Wei, David Brooks, Vijay Janapa Reddi, and Michael Mitzenmacher. Gra- dient disaggregation: Breaking privacy in federated learning by reconstructing the user participant matrix. In ICMI., 2021.
F. Lauer. Vc-dimension of hyperplanes. In An interactive journey into machine learning. 2017. URL http://e///dimmhmpppeeeee wawpepppppaaae.e..e .hml.
https://mlwen htt:://mwwwbbbb loria.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Carmon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML. Workshop Uncertainty and Robustness in Deep Learning, volume 5, pp. 15, 2020.
11

---

Published as a conference paper at ICLR 2024
Brendan McMahan and Daniel Ramage. Federated learning: centralized training data. In Google Research Blog. http:://ai..ooggebbbe ct00 2017/0110000
Collaborative machine leamingg without
federated learning collaborative. htmm..
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüira Arcas. Communication-eficcie learning of deep networks from decentralized data. In AISTA7S, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. IEEE Symposium on Security and Privacy, 2019.
Dario Pasquini. Danilo Francati, and Giuseppe Ateniese. Eluding secure aggregation in federated learning via model inconsistency. In CCS, 2022.
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier C. van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde. Sudeep Agarwal, Julien Freudiger, Andrew Byde, Abhishek Bhowmick, Gaurav Kapoor, Si Beaumont, Åine Cahill, Dominic Hughes, Omid Javidbakht, Fei Dong, Rehan Rishi, and Stanley Hung. Federated evaluation and tuning for on-device personalization: System design & applications. arXiv, 2021.
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans. Inf. Forensics Secur, (5), 2018.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv: 10Xx0088000045,, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asiiteed Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241. Springer, 2015.
Stacey Truex, Ling Liu, Ka Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: federated learning with local differential privacy. In EdgeSSss@ EuroSys, 2020.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. arXiv preprint arXiv: 12152, 2018.
Mark Vero, Mislav Balunovic, Dimitar I. Dimitrov, and Martin T. Vechev. Data leakage in tabular federated learning, ICML, 2022.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. arXiv, 2019.
Yuxin Wen, Jonas Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated learning via gradient magnification. In ICMI., 2022.
Han Wu, Zilong Zhao, Lydia Y. Chen, and Aad van Moorsel. Federated learning for tabular data: Exploring potential risk to privacy. In ISSRE, 2022.
Ruihan WU, Xiangyu Chen, Chuan Guo, and Kilian Q Weinberger. Learning to invert: Simple
adaptive attacks for gradient inversion in federated learning. arXiv, 2021.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In CCS, 2022.
Hongxu Yin, Arun Mallya, Arash Vahdar, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In CVPR, 2021.
Kai Yue, Richeng Jin, Chau Wai Wong, Dror Baron, and Huaiyu Dai. Gradient obfuscation gives a false sense of security in federated learning. arXiv, 2022.
12

---

Published as a conference paper at ICLR 2024
Shuaishuai Zhang, Jie Huang, Zeping Zhang, and Chunyang Qi. Compromise privacy in large-batch batch federated learning via malicious model parameters. In ICA3PP, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv, 2020.
Joshua C. Zhao, Atul Sharma, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Salman Avestimehr, and Saurabh Bagchi. Secure aggregation in federated learning is not private: Leaking user data at large scale through model modification. arXiv, 2023.
Junyi Zhu and Matthew B. Blaschko. R-GAP: recursive gradient attack on privacy. ICLR, 2021.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurlPS, 2019.
13

---

Published as a conference paper at ICLR 2024
A DETECTABILITY EXPERIMENTS
Here we provide more details regarding our detectability experiments.
Measuring D-SNR To produce Fig. 1, we consider 4 SEER-trained malicious models (CIFARIO, Bright/Dark, batch size 128/256), well 8 checkpoints made at various points during natural training, using the same initialization as used for SEER. Then, for each value of B E {16, 32, 64). we choose 5 random batches of size B from the training set and 5 random batches of size B from the test set of CIFARIO. For each batch, we compute the D-SNR on each of the 12 networks using Eq. 1 and plot the resulting value as point in Fig. (blue for natural and green for SEER networks). For example disaggregation attacks, we use publicly available implementation of the attacks of Wen et al. [2022] and modify the multiplier parameter to control the strength of the attack. We use the default setting where batches chosen such that all images belong to the same class (car in this case). The three reconstructions of the example disaggregation attack are obtained by running the modernized variant of the attack of Geiping al. [2020] on the disaggregated batch. The modernized variant is implemented in the Breaching framework, which Wen et al. [2022] is a part of. Finally, for the reconstruction of SEER, we aimed to show an image from the same class (a car), with D-SNR slightly below the D-SNR of the leftmost example disaggregation point (0.72). To do this, we use the Dark property and the dark car image from Fig. 3, and sample the other 63 examples in the batch randomly from the test set until the D-SNR falls in the [0.62, 0.72) range. We stop as soon as we find such a batch and report the reconstruction produced by SEER.
Measuring transmission As noted in Sec. 3, be applicable convolutional networks, boosted analytical attacks require handcrafted changes to convolutional layers that simply transmit the inputs unchanged. While, as noted above, even in the ideal case, this cannot lead to good reconstructions, we illustrate the point that such change is detectable by defining a metric similar to D-SNR, which can be interpreted as transmission signal-to tonise ratio. measured on the first convolutional layer. Namely, for each filter in the first convolutional layer, we divide the absolute values of the largest entry by the sum of the absolute values of all other entries. Intuitively, we treat the entry with the largest absolute value the signal, and measure how well this is transmitted by the filter. The ratio high when the filter transmits the input unchanged, and is oo for the handcrafted changes used by the boosted analytical attacks. We compute this metric the 12 networks used in Fig. (see previous paragraph) and show the results in Fig. 4. Intuitively. the red line at 1.0 indicates the case where there equal amounts of the pixel being transmitted and all other pixels. We can observe that all networks have values below 0.3, confirming that transmission is indeed unusual and not a case that ever happens naturally, implying that if boosted analytical attacks that use this technique would be able to obtain good results, they would still be easily detectable in the weight space.
SEER (Our Method) Non-Malicious Networks
XX
XXX
0.0
0.5
1.0
1.5
Transmission Signal-to-nois Ratio (Boosted Analytical =nfiiiitt
Figure 4: The transmission signal-to-nn -onise ratio of several SEER-trained and naturally trained networks. The same metric has a value of oo for all boosted analytical attacks.
B PRECISE THREAT MODEL
SEER is MS gradient leakage attack, and thus assumes the ability of the attacker to choose the weights of the model such that client data is easier to recover. Similarly to most other MS attacks [Boenisch et al., 2021; Wen et al., 2022; Zhang et al., 2023], we focus attacking the FedSGD [McMahan et al., 2017] protocol at a single communication round, and similarly to Wen et al. [2022], we recover single user image from large batch of client images. Importantly,
14

---

Published as a conference paper at ICLR 2024
unlike previous MS attacks. SEER does not lift any prior honest attack [Boenisch et al., 2021; Fowl et al., 2022b; Zhao et al., 2023; Zhang et al., 2023], does not require restrictive assumptions such architecture tweaking [Zhao et al., 2023], side-channel information, or knowledge of batch normalization data or labels [Wen et al., 2022], and does not require the ability to send different updates different users [Pasquini et al., 2022; Zhao et al., 2023]. Similarly Fowl et al. [2022b], SEER requires auxiliary data to be available to the server. We believe this does not limit SEER's ss practical applicability, as we demonstrate that we can train SEER's is malicious weights on small amount of data (App. F.8) and that our attack shows remarkable transferability across datasets and data distributions (Sec. and App. F.4 and App. F.11).
C RECONSTRUCTING FROM
GRADIENT UPDATES
As described in Sec. 4.1, we have designed a more elaborate property P for the case of attacking securely aggregated gradient updates. Our property is based on a combination of local (in-batch) and global distribution information about the client data allowing to handle this more complex case. In this section, we describe in detail how this is done.
As described in Sec. 4. we need define the property P with respect to range of brightnesses, such that Pee4 = 1) is maximized. Thus, generating P is reduced to finding the correct threshold 7 on the client image brightnesses, with which we later train SEER.
To calculate the threshold T in the multi-clien setting, we use the insight that individual client batches are still generated the presence of BN before their aggregation. To this end, we normalize the brightnesses within individual client batches of size B for 20000 sampled client batches and use the sampled normalized brightness to generate the cumulative density function(CDFF) of their empirical distribution. We then choose the threshold on this distribution to maximize the probability that exactly one out of the aggregated clients has exactly one image with normalized brightness above the threshold. For simplicity, we demonstrate this in the case of maximal brightness, as the hunimal-brightaen case is equivalent. We estimate the probability of having exactly one image with normalized brightness above the threshold as:
(1 - (T) * (2(T)) (7(
(6)
where p is the CDF of the top brightness in sampled batch, and T is the CDF of the second-highest brightness in a sampled batch. The equation can be intuitively rephrased as follows-for for exactly one client, the highest normalized brightness within its batch is above T, and the second-highes brightness is below т. while for the rest, all brightnesses are below т. To optimize Eq. 6 for the threshold т, we use the golden section search method a numerical optimization technique that repeatedly divides search interval by the golden ratio to efficiently locate the (possibly- locall) extremum of a function of single variable.
D EXISTENCE OF THE PROPERTY P
A key assumption of our attack is the existence of an image property P satisfied by exactly one image x E IRd in the client batch. We note that if we do not impose any restrictions on P, properties of the type "cqual to x" satisfy the requirement as they can single out any from any batch. However, such properties do not generalize well across batches, limiting SEER's 's practical applicability. A more interesting question then becomes, is there always property P that can separate exactly one image from a client batch that additionally is also (i) simple enough to train SEER's 's malicious weights on well, and (ii) transferable across batches without retraining. To this end, in this section we theoretically investigate the existence of properties P of the type m(x)) < where m is linear function and T E TR is threshold, as in our experiments (see App. F.6) these types of properties show good learnability and generalizability.
Under these assumptions, we can reformulate the question if P exists for a given batch and chosen image æ in it, as the question if hyperplane m(x) represented as a point in d-dimensional space, from the rest of the images in the batch. This question is answered positively by a well-known result from theoretical ML (see e.g., Lauer [2017]) which states that the VC dimension of affine classifiers (in this case sgn (m(x) r)) in Rd is d + 1. Thus, a linear property P that is true only for the chosen images x: in the batch always exists when the batch
0 exists that separates the chosen image,
15

---

Published as a conference paper at ICLR 2024
Table 5: Effect gradient clipping when different maximum gradient norms C are applied on CIFARIO model trained with the Red property on B = 128. We report the percentage of well- reconstructed images (Rec), the average PSNR and its standard deviation across all reconstruc- tions (PSNR-AII), and on the top 37% images (PSNR-Top).
C
Rec (%)
PSNR
PSNR-AII 1
1.0
36.2
21.5 1.7
3.44
2.0
87.3
2.4
2.2
3.0
92.6
21.7
24.7
4.0
95.2
2.
25.9
232
93.5
1.2
2.1
Table 6: Effect of applying DP-SGD with maximum gradient norm of C = 3 and different noise levels on CIFARIO model trained with the Red property on B 128. We report the percentage of well-reconstructed images (Rec). the average PSNR and its standard deviation across all reconstruc- tions (PSNR-All), and on the top 37% images (PSNR-Top).
Rec (%)
PSNR-Top
PSNR-AII
0.0
92.6
27.7 1.1
24.7 + 3.3
x 10-4
92.6
27.7 1.1
24.7 + 3.3
1x 10-3
92.4
+.0
24.4+3.1
3 x 10-3
90.3
0.66
22.6+
x 10-3
84.0
2..9
1.9
x 10-3
43.5
19.7
1.84
1 10
0.0
17.2
2..3
images are in general position (which usually holds) and B < d++ 1 (which holds even for low-dim images like CIFAR1O, as + 1 3073 which is far above practical batch sizes).
E POSSIBLE DEFENSES AGAINST SEER
Here we discuss other possible defenses against SEER, and why believe they are not currently effective at preventing data leakage. Detection based on tracking the value of the loss during training is, as noted in Boenisch et al. [2021], unlikely to work. This due to the fact that FL,, per-client values can generally be noisy, even if the global loss consistently falls. Such detection is also made harder by the fact that SEER does not require application in more than one round to pose serious threat to client privacy, and that such attacks are often applied in the first round [Balunovic et al., 2022b]. Next, SEER can't be easily flagged via other kinds client-side detection, as only modifies the weights through continuous optimization, using obvious handcrafted patterns (as opposed to prior work analyzed in the paper). Finally, defenses based on differential privacy such as DP-SGD [Abadi et al., 2016] require too much noise be practical, as also demonstrate in our experiments in App. 1. As we note in Sec. 6, we think that future principled client-side defenses can be promising future direction.
E. RESULTS UNDER DIFFERENTIAL PRIVACY
We demonstrate SEER's 's performance under defenses based on differential privacy. In particular, we apply our attack on gradients obtained from the DP-SGD [Abadi et al., 2016] algorithm. DP-SGD, the clients defend their data by first clipping the norms of the per-layer gradients of each of their data points to at most C, and then adding Gaussian noise with standard deviation of C a to them. In order to better understand what effect those two components of the defense have in Table we experiment with different clipping norms C for a = (i.e., no noise added), while in Table 6 experiment with the noise strength for clipping norm of 3. We use C = 3 as Abadi et al. [2016] recommends that value for CIFAR1O, the dataset we experiment with. Both experiments were conducted on SEER model trained with B 128 and the Red property.
method,
16

---

Published as a conference paper at ICLR 2024
Table 7: Large batch reconstruction on bright and red properties from batches of different sizes B on CIFARIO. We report the percentage of well-reconstructed images (Rec), the average PSNR and its standard deviation on all reconstructions (PSNR-AII), and across the top 37% images (PSNR-Top)
CI Red
CIFARIO, Bright
B
Rec (%)
PSNR-Top 1
PSNR-AII 1
Rec (%)
PSNR-Top
PSNR-AlI f
64
87.3
30.4 1.1
26.5 5.2
89.4
32.1 = 2.0
27.2 5.3
128
93.5
31.1 1 1.2
27.8 1
94.2
31.9 1.7
28.2 2 4.3
256
94.7
31.3 1.0
28.0 +
93.5
32.8 2.0
28.5 5.0
512
94.4
30.0 + 1.2
26.6 + 3.8
87.8
26.6 + 1.8
23.2 + 3.5
Table 8: Large batch reconstruction bright and red properties from batches of different sizes B on CIFAR100. We report the percentage reconstrcctt images (Rec), the average PSNR and its standard deviation across all reconstructions (PSNR-AlI), and the top 37% images (PSNR-Top).
CIFAR100, Red
CIFARI00, Bright
B
Rec (%)
PSNR-Too Top
PSNR-AII f
Rec (%)
PSNR-Top
PSNR-AII f
64
97.1
31.7 + 1.1
29.0 3.4
95.6
32.2 = 1.5
28.2 4.3
128
97.4
31.8+
29.3 3.2
94.7
30.0 1.3
26.5 3.7
256
97.7
1.0
3.2
98.1
35.2 1.4
1.4
30.8 + 4.8
512
98.6
33.1
95.0
32.2
27.6 4.6
In experiments, the performance of SEER increased when explicitly took into account the of DP-SGD during our attack procedure. In particular, before applying the attack in all experiments we first estimate the median clipping factor for each layer individually on 1000 batches of size 128 taken from our auxiliary data. These approximate factors are then reapplied to the total gradients sent from the clients the malicious server before applying Algorithm 2.
The results in Table and Table 6 confirm the trends observed in prior work [Zhu et al., 2019; Geiping al., 2020; Balunovic et al., 2022b] that low clipping norms C and high noise levels a result in more effective defense mechanisms. Still, we observe that SEER is fairly robust to DP-SGD, even for clipping norm and high noise levels of 5 x 10-3 our method able recover private date from more than 85% of client batches.
F ADDITIONAL EXPERIMENTS
In this section, provide additional experimental results, which we did not include in the main body due to space constraints.
F.1 EXTENDED LARGE BATCH EXPERIMENTS
In Table and Table 8, we present the extended version of our CIFARIO and CIFAR100 single-client experiments, first presented in Sec. 5. We observe similar trends in the original experiments. For example, we observe that CIFAR10 and CIFAR100 00 performances are similar and that there is no major difference in performance between the most bright and most red image properties.
F.2 EXTENDED SECURE AGGREGATION EXPERIMENTS
In Table 9, we present extended version of our CIFAR10 10 multi-client experiments first presented in Sec. 5 containing results for both the bright and dark properties. While we see the dark property reconstructions
generally slightly better than the bright ones, observe similar trends between the two sets of the experiments. This suggests our method for attacking secure aggregation federated updates is effective regardless the property used.
17

---

Published as a conference paper at ICLR 2024
Table 9: Reconstructions on securely aggregated batches on the bright and dark properties with different numbers of clients C' on CIFARIO, for different total numbers of images. We report the percentage of correctly reconstructed images (Rec) and the average PSNR across the top 37% images (PSNR-Top).
C = 4, Dark
C' = 4, Bright
C = 8, Dark
C' = 8, Bright
#lmgs
Rec (%)
PSNR-Top
Rec (%)
PSNR-Top
Rec (%)
PSNR-Top
Rec (%)
PSNR-Top
64
50.2
27.5 + 3.0
41.4
27.3 + 3.1
43.0
27.4 + 3.3
41.3
26.6 = 3.7
128
51.3
28.8 + 2.6
44.2
26.8 3.0
43.4
27.6 + 3.5
40.6
27.3 :: 3.3
256
50.9
29.8 1: 2.3
51.9
27.3 + 2.5
51.7
27.0
1: 2.9
41.9
25.4 = 3.1
512
61.3
30.2
+ 2.4
52.9
25.7 = 2.4
56.3
28.7
2.9
51.7
25.9
Table 10: Large batch reconstruction the bright property CIFARIO on a network trained with batch size = 128 and tested for various client batch sizes Bust- We report the percentage of well- reconstructed images (Rec), the average PSNR and its standard deviation on all reconstructions (PSNR- All), and across the top 37% images (PSNR-Top)
Bieat
Rec (%)
PSNR-Top
PSNR-AII f
64
42.0%
21.51 + 1.83
18.51 + 2.93
96
87.4%
30.56 + 1.31
26.28 + 4.92
128
94.2%
31.91 1.73
28.15 + 4.34
192
86.3%
30.78 2.37
25.77 + 5.14
256
67.5%
28.02 + 2.79
22.42 + 5.14
F.3 ROBUSTNESS TO B
In this section, demonstrate that attack parameters Oy generated by SEER for particular client batch size B can work to a large extent for batch sizes close to the original one, thus relaxing the requirement that the exact client batch size B is known during the crafting of the malicious model In particular, in Table 10, we show the effect of applying our single-client attack trained on B 128 on CIFAR10 using the Bright image property for clients with varying batch sizes Brest. We observe that while, as expected, SEER performs best when Bie = B, both the success rate and the quality of reconstruction on clients with batch sizes even 2x larger than the trained one remain very good. We note that Table 10 suggests that underestimating the client batch size Byea during the training of better than overestimating it, as the reconstruction performance when B 64 significantly worse than when Bres 256. This is mostly caused by d filtering out all images in the client batches resulting the removal of all the client data.
F,4 ROBUSTNESS TO IMAGE CORRUPTIONS
In this section, we show that SEER is robust to distributional shifts caused by common image corruptions. To this end, we use a SEER model trained on the CIFARIO trainset with the red property, batch size B 128, and without secure aggregation to attack batches sampled from the CIFAR 10-C dataset [Hendrycks & Dietterich, 2018], where 19 different image corruptions are applied at different levels of severity (1-5) the original testset of CIFAR1O. We show the results in Table 11.
We see that for all image corruptions but Fog and Contrast, even at severity 5, we recover images from more than 85% of client batches with good quality (average PSNR > 23 in most cases). For Fog and, to even greater extend, Contrast, however, we observed that reconstructions, while preserving the image semantic, became too bright resulting a very low PSNR numbers. To this end, we created modified version of our attack that approximates and applies multiplicative factor 3 by which one needs to multiply the recovered normalized images such that 90% the recovered images after denormalization inside the range [0, 1]. We use 90% to account for the fact that not all images will be correctly recovered, and we don't want these to affect our estimations. The results are shown under Fog Fixed and Contrast Fixed, where we get even better results than the original data.
18

---

Published as a conference paper at ICLR 2024
Table 11: Large reconstruction in the presence of different common corruptions applied on CIFARIO network trained on B = 128 using the red property. As baseline, our attack achieves Rec: 93.5%. PSNR-Top: 31.1 + 1.2, and PSNR-All: 27.8 + 4.1 We report the percentage of well-reconstructed images (Rec), the average PSNR and its standard deviation across all reconstructions (PSNR-AlI) and on the top 37% images (PSNR-Top) for three different degrees of severity of the corruption (Severity).
Severity
Severity = 3
Severity =
Compuptio
Rec (1)
PSNR-Top
PSNR-AII
Ree (%)
PSNR-Top
PSNR-AII 1
Roc (1)
PSNR-
PSNR-AII
Brightness
94.7
31.4 + 1.1
28.0 4.1
92.3
31.6 1.1
27.7 4.6
94.0
31.0 =1.0=10 1.4
27.3 t: 4.3
Contrast
96.2
21.6 = 1.0
22.9
2.8
17.8 0.9
16.6
15.1 = 1.3
13.6
Contrast Fixed
97.5
34.4
30.6 4.5
99.0
32.2
3.29
98.0
28.2
Defocus Blur
27.3
94.9
27.0
26.1 2.
94.8
25.2
23.2
Clastic Trunsform
85.0
29.0
26.3 3.3
94.7
27.7
25.2 4.
95.3
28.1
25.5 3.0
Fog
95.3
26.1 1.1
24.0
68.6
21.4 + 1.1
19.8
1.6
23.0
19.5 + 0.9
18.1 1.3
Fog Fixed
95.4
33.6
1.4
29.4
98.0
34.6
1.2
31.0
97.6
35.0
1.0
31.14:7
Frost
94.4
30.2 0.9
27.0
94.8
28.1 11 1.0
25.4 4.3
94.2
26.5
1. 0.7
24.3 2.6
Gaussian Blur
91.2
30.3
27.3
95.0
26.5 +
21.3
95.0
21.3
1.0
22.3
Gaussian Noise
27.2
25.2
90.9
26.3
Glass Blur
83.3
31.2 1.3
27.7
95.0
29.6
26.8 3.5
94,9
29.8
27.0 3.4
Impulse Noisc
83.0
29.5
0.8
26.0
90.9
26.1
21.0
85.8
22.5 - 0.5
2.1
Jpeg Compression
91.6
30.9 20 1.1
27.7 3.9
95.3
30.8 11 1.1
27.7 3.8
91.5
30.6 6 1.1
3.8
Motion Blur
96.3
28.6 1.0
26.1 3.1
95.0
26.2 +: 1.1
24.1 2.5
94.6
25.3 + 1.2
23.3 2.3
Pixelate
93.3
30.8 1.1
27.6 3.9
94.2
30.4
3.8
94.1
29.1 + 1.0
334
Saturate
29.1
26.1
95.0
28.0
91.4
24.7 +
Shot Noise
30.7
27.6
29.0
26.2 3.6
91.5
27.0
+33
Snow
83.3
31.3
27.9
92.6
30.8
4.4
91.8
29.3
25.9 4.0
Spatter
93.3
31.3
1.1
27.9
4.2
92.5
31.1 4: 1.0
27.5 4.4
93.0
30.6
+ 1.1
27.1 4:2
Speckle Noise
93.8
30.8
12 1.0
27.0
3.9
92.5
29.5
12 0.8
2.9
91.2
27.0
1 0.8
21. 3.3
Zoom Blur
83.6
27.9
1.1
25.3
11
93.6
26.6
2.
95.4
25.5 11
1.0
23.5 2.3
Table 12: Large batch reconstruction the bright and dark properties from batches of size B 128 on CIFARIO using local (Local) and global properties P (Global). We report the percentage of well-reconstructtd images (Rec), the average PSNR and its standard deviation across all reconstruc- tions (PSNR-AlI), and across the top 37% images (PSNR-Top).
CO Bright
CIFARIO, Dark
P
Rec (%)
PSNR-Top
PSNR-AlI
Rec (%)
OSNR-Top
PSNR-All f
Global
54.4
27.0 1.8
20.6 1 5.8
61.9
27.7 2.2
21.1
Local
94.2
31.9 + 1.7
28.2 + 4.3
81.3
33.6 + 1.4
27.4 7.3
We conjecture this is due to the fact that most of the corrupted images have narrower range of pixel values making obtaining high PSNR numbers easier. All in all, our experiments show that SEER is very robust to image corruptions, and that even the most severe changes like the ones in CIFAR 10-C, sometimes resulting in hard to recognize images, be handled well by our algorithm.
F.5 COMPARISON BETWEEN OUR LARGE BATCH AND SECURE AGGREGATION PROPERTIES
In this section, compare our two SEER variants-one based purely on the local distribution of m for the large batch setting, referred to as local P. and the other for secure aggregation setting, described in App. C, which based on a mix of the global and local distributions of m and referred to as global P. We mount both variants of SEER on gradients coming from single client with batch size B- 128 on CIFARIO 10. We note that both methods are well-defined in this setting, and either one can successfully reconstruct data from the client batches.
The results are depicted in Table 12. While both methods successfully reconstruct the majority of client batches, we clearly see the benefits of using the local property P. In particular, the results in Table 12 suggest that the local distribution approach reconstructs up to 1.75 times more images, while also producing higher PSNR values not only the full set of reconstructed batches but also the top 37% of them. This motivates the need for our single-client attack variant, and demonstrates that secure aggregation provides additional protection individual clients.
19

---

Published as a conference paper at ICLR 2024
Table 13: Large batch reconstruction on CIFARIO with 128 with different properties P. We report the percentage of well-reconstruuted images (Rec), the average PSNR and its standard deviation on all reconstructions (PSNR-AlI), and across the top 37% images (PSNR-Top).
Property
Rec (%)
PSNR-Tow Top
PSNR-AII
Bright
94.2
31.9
28.2 1
Dark
81.3
33.6 + 1.4
27.4 7.3
Red
93.5
31.1
27.8 4.1
Blue
97.2
0.9
28.6
Green
96.7
32.8
29.6 1 4.0
Edge
80.1
2..0+
24.4 5.5
V Edge
85.8
20.6
25.5
Green V Edge
95.1
3.
28.6 4.5
Rand
97.5
32.8
29.4
F.6 ADDITIONAL TYPES OF PROPERTY METRICS m
Bright
Dark
Red
Green
Blue
H Edge
V Edge
Green V Edge
Rand
C-1
GT
C-1
Figure 5: Example reconstructions of SEER trained on CIFAR10 with C 1, B 128 and different properties.
In this section, we demonstrate that our method works well for variety of properties P based on different metrics m.. In particular, we look at local properties P based on: (i) the image brightness- the most bright (Bright) and the most dark (Dark) image in batch: (ii) the image color the most red (Red), blue (Blue), or green (Green) image in a batch; (iii) edges in the image- the image with the strongest horizontal (H Edge), vertical (V Edge) edges in batch; (iv) combination of image color and edges-the most green image with vertical edges (Green V Edge); and, finally, (iv) based on random property (Rand). For the color properties we rank the batch images based on the difference between two times the average color channel response for the chosen color and the sum of the other two average color channel responses. For the edge properties, we ranked the batch images based on the average response with the [1. 1] edge filter (either in horizontal or vertical direction) grayscale version of the image. Further, for the combination filter we ranked images based on sum of the color and edge property scores. Finally, for the random property we ranked the batch images based on the average response to a random 3 convolution filter that was normalized. The results for CIFAR10 for the large batch size setting for B 128 is shown in Table 13. Further, example reconstructions are given in Fig. 5.
We observe that for all properties SEER successfully recovers data from large portion of the client batches (>80%), with good quality (PSNR=24). Yet, we still observe some variability across the properties with the Random property being the casiest attack, as demonstrated by the percentage of recovered images and the very good PSNR metrics. This is in line with the observations Fowl et al. [2022b]. We also observed that the Dark property produces the best image reconstructions, as shown
20

---

Published as a conference paper at ICLR 2024
Table 14: Large batch reconstruction on CIFARIO model trained with the Bright property on B 128 after several rounds of federated training (#Rounds) with different learning rates (Learning Rate) using 8 clients. We report the percentage of well-reconstruutee images (Rec), the average PSNR and its standard deviation across all reconstructions (PSNR-AlI), and on the top 37% images (PSNR-Top).
#Rounds
Learning Rate
Rec (%)
PSNR-Top 1 PSNR-AII
0
1 10
91.2
31.9 1.7
28.2 + 4.3
1
1 10
94.6
31.8 1.7
28.2 4.3
2
1 10
94.4
1.6
27.3
3
1 x 10
89.2
23.6 1.7
x 10-4
25.8
20.1 1.5
17.7+23
5 1 10
6.1
17.5
2.3
1 x 10
91.4
29.0 1.7
25.1 + 4.0
by PSNR-Top, but fails to reconstruct often. practice, we observed this happens due SEER recovering completely black images. We conjecture this is due to lack of diversity in the images SEER sees during training as most images CIFARIO with the Dark property have completely black background. Finally, we observe that color properties are easier to attack compared to edge ones but, interestingly, when combined, like in Green Edge, the results become much closer to the color version. We conjecture this is due images with very pronounced colors being more similar each other and, thus, casier to distinguish from the rest of the images in the batch.
F.7 RESULTS UNDER TRAINED ENCODER
Figure 6: Example reconstructions of SEER after training the client model for different number of FL rounds using clients without secure aggregation with learning rate x 10 Left to right: Ground truth, 1 round, rounds, 3 rounds, 4 rounds, 5 rounds.
Throughout this paper, we assumed that our attack is mounted at the beginning the training procedure. In this section instead, show the results of using SEER's decoder reconstruct images from gradients computed on models that were trained for several federated learning rounds from the malicious state chosen by SEER. We show the results quantitatively Table 14 and qualitatively in Fig. 6 for different number of rounds (#Rounds) and learning rates Learning Rate when training with clients without using secure aggregation.
We observe that our decoder works even after several rounds of training of the client model. Further, we observe that our decoder more robust single large step update (One round with learning rate 5 x 10-4) than many smaller updates applied sequentially (Five rounds with learning ratel x 10-1) and that each additional communication round results small additional drop in quality of the reconstruction. We believe further improvements over these results are achievable if one finetunes our decoder model each FL round to match the changes of the client encoder, but we leave this as future work.
F.8 EXPERIMENTS WITH AUXILIARY DATASETS DIFFERENT SIZES
In this section we investigate how the size of the auxiliary dataset used for training SEER affects its results. In particular, in Table 15, we show the results when we used only percent of the data points in CI'AR10's train set auxiliary data. As expected of any algorithm based training, results generally improve with the number of datapoints available to the attacker. Despite this, we that method is very sample efficient,
successfully reconstruct data from 80% of
21

---

Published as a conference paper at ICLR 2024
Table 15: Reconstructions on models trained with the Red property and B 128 on different percentage of data in the CIFARIO trainset. We report the percentage of well-reconstructed images (Rec), the average PSNR and its standard deviation across all reconstructions (PSNR-AlI), and on the top 37% images (PSNR-Top).
Rec (%)
PSNR-Top
PSNR-AII 1
5
80.6
24.5 + 1.6
21.5
27.9
24.7
20
86.9
30.2
26.4 4.9
87.5
31.7
27.2
50
96.8
31.
28.3 3.8
100
93.5
31.112
27.8
Figure 7: Example reconstructions SEER trained on different percentage p CIFAR1O train set. Left to right: Ground truth, 100%, 50%, 33%, 20% 10%, and 5%.
client batches even when training on mere 2500 training data samples (p = 5%). We further show qualitative comparison between the models in Fig. 7, where confirm the quality of reconstructions degrades when less data available, especially for very small p, yet for all the images remain recognizable regardless.
F.9 RESULTS UNDER CLIENT HETEROGENEITY
Next, we look at how our method is affected by the level of client data heterogeneity. In particular, take our CIFAR 10 model trained with B 128 and the Red property, and we evaluate its performance on clients with different level of noD-IID data. To simulate non-lID data, sample from Dirichlet distribution with parameter a, to determine each client's label distribution and randomly sample the client data according this. In this setting, close to 0 means that each client only holds data from few classes. We show the results Table 16, where we show that while heterogeneity slightly degrades our performance, SEER is very robust heterogeneity, even severe heterogeneity levels like a 0.1 produce above 90% attack success rate with average PSNR > 25.
F.10 RESULTS ON THE RESIMAGENET DATASET
this section, we show quantitative and qualitative results from applying SEER on the ResImageNet dataset (Restricted ImageNet [Tsipras et al., 2018], subset of ImageNet with 9 superclasses). Our setup is similar to the experiments presented for ImageNet in the main paper, i.e., we use batch size B 64, the Bright property, and our U-Net decoder architecture (App. G). Further, same hyperparameters (App. H), except for two small changes: (i) we execute the pretraining stage on the images in CIFAR 10, instead of the downsized version of the images in ImageNet, avoiding
also the
22

---

Published as a conference paper at ICLR 2024
Table 16: Reconstructions for clients with different data heterogeneity levels on CIFARIO model trained with the Red property on B = 128. We report the percentage of well-reconstructed images (Rec), the average PSNR and its standard deviation across all reconstructions (PSNR-AlI), and on the top 37% images (PSNR-Top).
Rec (%)
PSNR 1BNR-Top
PSNR-AII f
0.1
93.9
28.1 + 1.0
25.3 + 3.3
0.2
94.4
28.7 + 1.1
2.3
0.3
94.3
29.1
2.42
0.4
95.1
2.12
26.4
0.5
95.0
1.0
26.6 + 3.4
0.6
96.2
29.5
26.7 + 3.2
0.7
95.8
2.
26.6 3.3
0.8
95.8
29.7
27.0 + 3.3
0.9
95.8
29.6
26.9 3.3
1.0
96.1
29.8 - 1.1
27.0 + 3.3
Ground Truth
C=1
Figure 8: Example reconstructions of SEER ResImageNet for B 64 and the Bright property.
pretraining the full 1000 classes, (ii) train for 370 epochs instead, with 400 gradient descent steps per epoch.
Under these settings, SEER is able recover 77% of images with average PSNR of 20.6 + 3.7 and PSNR Top of 23.8 + 1.4. Examples are shown in Fig. 8, where we see that images are clearly recognizable and accurate in terms of object positions.
F.11 RESULTS UNDER DATA DOMAIN SHIFTS
Ground Truth
C=1
Figure 9: Example reconstructions of SEER trained on ImageNet with B and the Bright property and applied ISIC2019.
23

---

Published as a conference paper at ICLR 2024
[+pove "+2000" :) Onnm
TreppeCCee221ES 138
1533
1286543
Sbst0tsto1
S. So
g
000
300
500
S00
1000
2000
Figure 10:
The architecture of our U-Net-based reconstructor used in our ImageNet experiments. Here g is the randomly subsampled model gradient and the output is the resulting reconstructed image.
In this section, we first describe some implementation details for our ISIC2019 domain shift experi- ment originally presented in Table 3 that were omitted from the main text for brevity. In particular, to apply our CIFARIO network trained with the Red property on B 128 to the ISIC2019 dataset, we resized the ISIC2019 images that their larger side is 350 pixels, then applied random 224 x 224 crop, followed by another downsizing of the image to 32 x 32. Since the domain naturally includes a lot of red images, the data represents pictures of skin conditions, too brightly red images. To avoid this issue, we applied the fix presented in App. F.4 for the Fog and Contrast corruptions. Note that before, the fix requires additional knowledge about the color distributions of the images in the ISIC2019 dataset.
that method generated
Next, we explore a high-resolution version of our domain shift experiment presented in Table In particular, we use the ImagetNet network trained in App. F. 10 and apply it to the ISIC2019 dataset prepared like above but with final resolution 224 224 and only data from 76.3% of client batches with average PSNR 20.0 + 1.8 and PSNR Top of 21.7 + 0.8. obtain We further show example reconstructions of the images Fig. 9. Similarly to Table slightly worse reconstructions compared to the dataset used for the training, but we see that the skin conditions in Fig. are clearly identifiable from our reconstructed images.
64. We successfully recover
G U-NET-BASED IMAGE RECONSTRUCTOR
We explain the architecture of image reconstructor used in ImageNet experiments in Sec. 5. Our architecture is inspired by the decoder portion U-Net Ronneberger et al. [2015], which has been demonstrated to be memory- effirretefficient architecture for generating images.
We show our architecture in Fig. 10. In the figure, depicts our model's gradient subsampled randomly that only 3% of its entries kept (See App. I.1). There two main differences
24

---

Published as a conference paper at ICLR 2024
between in Fig. 10 and the original 6-layer U-Net architecture. First, functions, thus creating a (sparse) linear reconstructor function т. This allows us, similarly to our CIFAR10/100 experiments, to combine and d into single linear layer, whose bias becomes the target for our filtered-out inputs mut- Second, as our architecture does not include U-Net-style encoder, the U-connections of reconstructor are substituted by pairs of linear layers with bottleneck in the middle applied on g. In Fig. 10, we depict the bottleneck sizes and transposed convolution sizes, as well as the intermediate output sizes of the different layers. The bottlenecks ensure that the memory efficiency of our method is preserved and are inspired by the intuition that the U-connections only need to provide fighffreyeeccy content which can live in much lower- dimensional subspace. Finally, we note that for the purpose of pretraining, we used the first 3 channels of the third transposed convolution layer as our downsized image output and that our transposed convolution stack produces images of size 264 x 264, which we then center-crop to produce our ImageNet-sized final output.
no activation
H HYPERPARAMETERS
In this section, we provide more details about the exact hyperparameters used in our experiments in Sec. 5. We implemented SEER in Pytorch 1.13. Throughout experiments, we used the Adam optimizer with a learning rate of 0.0001. For CIFARION 10/100, we trained between 500 and 1000 epochs, where epoch is defined to be 1000 sampled batches from our trainset. To stabilize our training convergence, we adopted gradient accumulation and, thus, updated our modules' parameters only once every 10 gradient steps amounting 100 gradient descent steps per epoch for those datasets. For ImageNet, we additionally execute pretraining stage downsized version of the training images allowing the network to first learn to recover large details in the image before recovering finer details. For ImageNet, we train with the original fz-based loss Liee for the first 200 epochs, followed by 300 epochs of using & version of it, resulting in better visual quality of the reconstruction. At each epoch we only use 4000 randomly sampled batches instead of the full dataset to reduce computational complexity.
For faster convergence and better balance in the optimized objective L Lree + a Lmul, adopted a schedule for the hyperparameter a, following an exponential curve of the epoch K.
The schedule is defined as: a(k) mén( [), 2*(*)), ), where ((x) = [K (K-x))a 80 linearly inter- polates between in and 81 across the total number of epochs K with (5n, 31) set to (-2, loga B)). For ImageNet, we set (Bo, 31) to (-5, 5.3) to allow for better reconstruction earlier in the training process. Finally, we point out that in the multi-client setting, we estimate the cumulative density functions before training for the first and -eoond-highest brightness in batch 20000 randomly sampled batches from the trainset (see App. C).
I IMPLEMENTATION DETAILS
1.1 SUBSAMPLED GRADIENTS
In order to save memory and computation, we use only part of the entries in our full model gradient g to construct our intermediate disaggregation space 124 In particular, we randomly sample 0.1% of the gradient entries of each of the model's parameters while ensuring that at least 8400 entries per parameter are sampled for our CIFAR10/100 experiments, and 2% and at least 9800 entries for our ResImageNet experiments. This results in 1.6% of the total gradient entries for CIFAR10/100 and 3.0% for ResImageNet.
theorize that we are able to reconstruct nearly perfectly with such a small percent of the gradient entries because there is large redundancy in the information different gradient entries provide.
I.2 EFFICIENTLY COMPUTING THE DISAGGREGATION LOSs LNLLL.
Computing Lrull directly for large batch sizes B takes lot of memory due the need to store g; for all i in the large set I Note that the reason for this is that we want to enforce all of the individual gradient g; to fall in the null space of 0₂ separately. In practice, save space, we enforce the same
25

---

Published as a conference paper at ICLR 2024
Table 17: Large batch reconstruction on the bright and red properties from batches of different sizes B on CIFARIO. We report several additional quality of reconstruction metrics- the average MSE and its standard deviation across all reconstructions (MSE All), and on the top 37% images (MSE Top), as well as. the average LPIPS and its standard deviation across all reconstructions (LPIPS AlD,, and on the top 37% images (LPIPS Top).
CIFARIO, Rel
CI Bright
15
MSE Top J
MSE AIL
LPIPS Top
LPIPS AIL
MSE Top 1
MSE AlL
LPIPS Top
LPIPS All
0.0009 0.0022
0,0088
0.039 0.012
0/0007
0.0060
0.046
0.104 0.090
128
0.0008 1 0.0002
0,0002 0,0072
0,050 1 0,014
0.006 0.089
0.0007 1. 0.0002
0.0031 1. 0.0060
0.057 1. 0.025
0,104 1. 0.081
256
0.0008
0.095
0.0006 0.0002
0.0075
0.054
0.087
512
0.0010 + 000002
0,0039 0.0051
0.080 0.025
0.341 + 0.01
0.0024 + 0.0007
0,0070 = 0.0091
0.168 + 0.058
0.304 0.089
condition by computing the surrogate:
Σ d(g.) 112+ || ((d 12,
where the first part of the equation enforces the mean gradient, and the second part enforces different randomly chosen gradient at every SGD to both approach 0. This, in practice, has a similar result to the original loss Lulll in that, over time, all gradients in Lnul go to O.
I.3 TRAINSET DATA AUGMENTATION
For the purpose of training our encoder-decoder framework, we observed data augmentation of our auxiliary dataset is crucial, especially for large batch sizes B. We theorize that the reason for this is the lack of diversity in the reconstruction samples X particular, as grows, an increasingly smaller set of images are selected to be the brightest or darkest of any batch sampled from the training set. To this end, when sampling our training batches for CIFAR10/100, we first apply random ColorJitter with brightness, contrast, saturation, and hue parameters 0.2, 0.1, 0.1, and 0.05, respectively, followed by random horizontal and vertical flips, and random rotation at N * 90 + c degrees, where N is a random integer and is chosen uniformly at random on [-5,5]. For ResImageNet, we additionally do a random cropping of the original image to the desired size of 224 x 224 before the other augmentations.
1.4 TRAINSET BATCH AUGMENTATION
As detailed in App. C, when mounting SEER the secure aggregation setting we select r based on a mix of the global and local distributions of m. As noted in Sec. 4.1, the probability of attack success with an optimal threshold T based on the global distribution of m is 1 in the limit of the number of images being aggregated. We expect for large B, therefore, SEER to also successfully reconstruct only for ~ 1 of the securely-aggregate batches, forcing us to avoid training on the rest 1 > 1 of the securely-aggregated batches, which act as strong noise during training and prevent convergence. This, in turn, results in rejecting training on a big portion of our sampled client batches.
2
To address this sample inefficiency, we use batch augmentation during training transform the client batches to ones with a desired brightness distribution. The batch augmentation simply consists of adjusting the brightnesses of individual images within each batch. We do two types of batch augmentations based on two different distributions one where it contains precisely one image in the batch with brightness above the threshold 7 and another where precisely zero images in the batch have brightness above the threshold r. We alternate the two augmentations at each step of the training procedure, To achieve the distributions, we adjust all image brightness within a batch using a heuristic method. The method first adjusts the brightness of the most bright image (the least bright image in the case of the dark image property) such that it lands on the desired side of the threshold т. However, as after adjusting the image, the brightnesses within the batch are longer normalized, we then need to renormalize the batch, resulting in a new batch brightness distribution. If our new distribution is desired, we stop. Otherwise, we iterate the process until convergence.
26

---

Published as a conference paper at ICLR 2024
Table 18: Large batch reconstruction on the bright and red properties from batches of different sizes B on CIFAR 100. We report several additional reconstruction quality metrics- the average MSE and its standard deviation across all reconstructions (MSE All), and on the top 37% images (MSE Top), as well as, the average LPIPS and its standard deviation across all reconstructions (LPIPS All), and on the top 37% images (LPIPS Top).
CIFARIOG, Red
CIFARIO0, Bright
15
MSE Top
MSE AlL 1
LPIPS Top
LPIPS AIL
MSE Top 1
MSE AIL
LPIPS Top
LPIPS All
0.0007 0,0022
0.0016
0.010
0.66 0.099
0/0006
0.0008 0.0047
0.050
0.001
128
0.0007 1 0.0001
0,0019 1. 0,0030
0.031 1 0.011
0.051 1: 0.061
0,0010 1 0.0003
0,0005
0,0045
0,112 11121100 0.035
0.164 0.078
256
0.0077
0.014
0.044
0.0003 0.0001
0.0017
0.0031
0.019
512
0.0005 = 0.0001
0.0014 + 0.0025
0.054 + 0.000
0.057 0.019
0,0006 + 0.0002
0.0032 = 0.0041
0.077 + 0.032
0.148 0.085
Table 19: Reconstruction from securely aggregated updates on the bright and dark properties using different numbers of clients C' on CIFARIO, for different total numbers of images. We report additional reconstruction quality measures the average MSE (MSE Top) and LPIPS (LPIPS Top) and their respective standard deviations on the top 37% images.
C- 4, Dark
Rright
C-
C- 8, Bright
#ings
MSR Top |
LPIS Top.
MSR Top
LMMSS Top
MSF Top
LPIPS Top
MSR Top 1
L Top
64
0.0020 + 0001
0,110 4 0,052
0.0027 + 0.0022
0.091 0.060
0.0031 + 0.005
0,111
0.059
0.0026 0.0021
0.123 + 0.073
00003
0,0025 SUDOLS
0,0225
0.002
0.118
0.0001 0.0206
0.111
256
0.0012 # 0.0008
0.130 à 0.045
0.0022 # 0.0012
0.118 4: 0.040
0,0023
: 0.0014
0,130 4: 1.000
0.0085
0.0022
0.164
0.0012
DU00L3
0
0.0015 1: 0.0010
0,120 0,004
0,0029 000077
0.074
1.5 DETAILS ON PRIOR WORK COMPARISON
In this section, we provide further details about the exact setting in which we do the comparison against prior work in Table 4 and Fig. 1 in the main text. We focus on the feature fishing variant of Wen et al. [2022], that targets batches containing a large percentage repeated labels, as this setting was shown in prior work [Yin et al., 2021; Geng et al., 2021] to be significantly harder to solve. For the comparison itself, we evaluate both the Fishing and SEER models on client batches of the same class. This setting favors Wen et al. [2022] over SEER, as the Fishing's ' weights are specifically adapted and the SEER model was only trained on batches with randomly selected mixed labels, as in the rest of the paper. We select D-SNR threshold of 5 for the experiment in Table 4 based on Fig. 1. For the comparison against 2hang23 [Zhang et al., 2023] and LOKI [Zhao et al., 2023], we observe they have practically infinite T-SNR, making them trivially detectable.
J ADDITIONAL MEASUREMENTS OF THE QUALITY OF OUR RECONSTRUCTED IMAGES
In Sec. 5 and App. F.I, we focussed on reporting the quality of our image reconstructions in terms of the popular PSNR image quality metric. In this section, we provide additional image quality measurements terms of the mean square error of the individual pixels (MSE) and the learned perceptual image patch similarity (LPIPS) metrics. We present the additional measurements for all large batch experiments from App. F.1 in Table 17 and Table 18 for CIFARIO 10 and CIFAR 100, respectively. Further, in Table 19, we present the additional measurements for the multi-client experiments originally presented in Sec. 5. The additional measurements reinforce our observations from See. that SEER consistently reconstructs client data well.
27


Reviews:
[{"review_id": "gOmfllm1mL", "reviewer": "Reviewer_Lg7w", "summary": "The paper found that existing malicious server attacks in FL are detectable on the client side using a metric, D-SNR. The authors then propose a new attack called SEER which can bypass the detection by co-optimizing the disaggregator and reconstructor.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. D-SNR is a useful metric to detect disaggregation by malicious server\n2. SEER is an attack that can efficiently extract information and bypass D-SNR detection.", "weaknesses": "1. The performance of SEER will depend on the task and dataset. For some dataset, SEER might lose effect since it's not always possible to disaggregate a value from the mean.\n2. The evaluation is on CIFAR10/100. It'd better to demonstrate the effectiveness of SEER on more tasks closer to real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqhdfhb8i4", "reviewer": "Reviewer_dXt6", "summary": "This paper studies data-stealing attacks in federated learning with malicious servers. The authors first demonstrate the client-side detectability of existing attacks by introducing a simple vulnerability metric called D-SNR, and then reveal the limitation of their attack design which is their dependence on the underlying honest attack. Based on this analysis, the authors further propose a new attack framework SEER using a server-side decoder that is jointly optimized with the shared model to reduce the chances of being detected. Experiments on three image datasets show that the proposed attack can successfully extract user data of large batch sizes (up to 512) even under secure aggregation.", "questions": "1. Could you provide some further explanation on how SEER utilizes batch normalization to choose the local property? In cases where this is not feasible (e.g., secure aggregation or no batch norm layer), would it be possible for the attacker to simply run a linear search to find the optimal property for each batch?\n\n2. Would it be possible to replace the learned disaggregating mapping with some hand-crafted criteria such as simply recovering the image with the largest loss within the batch?\n\n3. Does the design of the local properties take into account the potential non-IIDness of the data? What’s the success rate if the clients’ data are locally correlated?\n\n4. Could you share some insights on designing potential defense and mitigation strategies for the proposed attack besides standard options like DP and HE?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is in general well-written and self-contained and provides a good summarization of prior studies in this field.\n\n- The proposed method addresses several limitations of existing attacks including client-side detectability and the assumptions on BN statistics and labels.\n\n- The idea of jointly learning a disaggregator and a reconstructor with the shared model is novel.\n\n- The empirical results are in favor of the proposed method in terms of stealthiness and reconstruction quality.", "weaknesses": "- The work lacks a more nuanced discussion on the threat model. Despite having relaxed some of the assumptions of previous work, the proposed attack is still somewhat restrictive in the sense that it requires multiple steps of offline training and an auxiliary dataset of sufficient size, and it is not very clear how to deploy such an attack during the federated learning process in practice. In particular, the joint optimization of the shared model seems to ignore the classification loss of the federated task, which might be leveraged to detect the attack in a retrospective way (e.g., a client may evaluate the local training loss before and after each update and choose to opt out of training if the loss is not reduced after several rounds). One reasonable strategy is to deploy the attack in the first round of training, i.e., substitute the random initialization with the optimized model weights, but this may still raise the alarm if it’s beyond normal weight distributions. It would be better to add corresponding discussions and limitations.\n\n- It is not quite clear how to utilize the batch normalization statistics to design the local selection strategy of the property.\n\n- The proposed attack algorithm appears to be quite computationally expensive (~14 GPU days to train for ResImageNet). It also implicitly assumes knowledge of the private data distribution as is for all learning-based attack approaches that rely on a large auxiliary dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xu73axHqPx", "reviewer": "Reviewer_SRu7", "summary": "1. The paper focuses on the issue of client-side detectability of malicious server (MS) attacks in federated learning. It discusses prior work on gradient leakage attacks in federated learning, including honest server attacks, malicious server attacks, and the limitations of existing attacks.\n\n2. To ensure reproducibility, the authors provide the source code of SEER in the supplementary material and detail how to install the code prerequisites and reproduce the experiments presented in the paper.\n\n3. Overall, the paper contributes a novel attack strategy that avoids detection in federated learning, highlights the importance of attack detectability, provides insights into reconstructing data from securely aggregated gradient updates, and addresses the ethical implications of their attack.", "questions": "See weakness part", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a novel attack strategy called SEER that avoids detection while effectively stealing data despite aggregation. They demonstrate that all prior attacks are detectable in a principled way and highlight the importance of studying attack detectability.\n\n2. SEER is designed to reconstruct data from securely aggregated gradient updates. The authors describe in detail how they combine local and global distribution information about the client data to achieve this reconstruction.\n\n3. The paper also discusses the ethical implications of their attack and acknowledges the potential disparate impact and privacy risks. However, they argue that their investigation of detection and emphasis on realistic scenarios have a positive impact by enabling further studies of defenses and helping practitioners better estimate privacy risks.", "weaknesses": "1. Would the author have some initial idea for defensing the proposed attacks？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnSy0SOjuT", "reviewer": "Reviewer_sMvX", "summary": "This paper proposes the disaggregation signal-to-noise ratio (D-SNR) metric, designed to detect the vulnerability of local clients to malicious attacks (MS). Subsequently, the authors outline a set of requirements for future realistic MS attacks. They then present a novel attack framework, SEER, which operates by disaggregating the gradients in a hidden space. Specifically, the SEER framework encompasses a shared model dispersed among clients for encoding gradients, a server-side disaggregator to nullify the contributions of images not satisfying the pre-defined property and a server-side reconstructor to reconstruct images that comply with the said property. The framework facilitates end-to-end training and presents a challenge for detection using D-SNR, in contrast to previous MS methods. Experiments also show the effectiveness of the proposed method.\n\nThe main contributions of this work include a pioneering study of client-side detectability, the introduction of a novel detection metric, highlighting potential concerns for future MS, and the unveiling of a groundbreaking realistic MS approach.", "questions": "- How does the proposed SEER MS fare when differential privacy mechanisms are implemented?\n- How does SEER perform in non-iid settings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-written and easy to follow, making for an enjoyable read.\n- The motivation is clear and the study problem is significant.\n- The proposed detection metric and attack framework are innovative and intriguing, offering a fresh viewpoint on the privacy issue in FL.", "weaknesses": "The main concern is about the auxiliary data used to train the SEER framework. The key point is that the pre-defined property could separate examples.\nThe paper details experiments where CIFAR10 serves as the auxiliary data for other client datasets (CIFAR10.1v6, CIFAR10.2, and TinyImageNet). While the results affirm its robustness to distributional disparities between the auxiliary and client data, there may be scenarios where this approach is less effective. For instance, when the clients possess specialized data such as medical records, the server have no prior knowledge about the data and use generic dataset like CIFAR10 as the auxiliary data, the framework might encounter challenges, because the pre-defined property might significantly differ across such diverse datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3AVpJBhs7S", "reviewer": "Reviewer_ZL2M", "summary": "This paper points out that malicious attacks for data stealing of FL clients are easily detectable, and propose a novel data stealing attack named SEER that can evade defenses and achieve good data reconstruction performance.", "questions": "1.\tDoes a distinguishing property for each sample in a batch always exist? What if it does not exist? How does it affect the performance of SEER?\n2.\tAre all the experiments under any kinds of defenses or nor? How do the experiment results show that SEER is undetectable?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "Propose to design defense-aware data stealing attacks, making the attacks potentially stealthier.", "weaknesses": "1.\tThreat model is missing. For example, it is until the description of the SEER attack that the authors mention the availability of auxiliary data at the server, which is not always required by other attacks and can be a strong assumption. \n2.\tLack of details: it is not clear what are the exact client-side checks the malicious server needs to evade; it is also not clear what do the authors mean by “handcrafted modifications”. Why is SEER’s mode not handcrafted? The descriptions of the necessary conditions at the end of section 3 are too vague, not to mention providing any theoretical or experimental proofs for those.\n3.\tIt is not clear how SEER exactly breaks through secure aggregation.\n4.\tQuality and quantity of auxiliary data are key to the performance of SEER. More discussions and ablation studies are needed on auxiliary data.\n5.\tComplexity issue. Algorithm 1 needs to be repeated for each sample in the batch (i.e., retrain a set of models for each property). This incurs high computational complexity, and should be evaluated carefully and compared with baselines.\n6.\tNo evaluations of SEER and baselines under defenses, which is supposed to be the main motivation of the paper.\n7.\tThe results in Table 3 are quite counter-intuitive and curious. This needs more exploration and explanation.\n8.\tOnly comparison with one baseline in Table 4, which is insufficient. The reviewer would like to see more evaluations and comparisons with the malicious server attacks mentioned in the \"gradient leakage attacks\" subsection in introduction, and \"malicious server attacks\" sub-section in related work. Specifically, comparisons with the following recent attacks are needed:\n\n1) Joshua C. Zhao, Atul Sharma, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Salman Avestimehr, and Saurabh Bagchi. Secure aggregation in federated learning is not private: Leaking user data at large scale through model modification. arXiv, 2023. (Just accepted to S&P 2024 with the title \"LOKI: Large-scale Data Reconstruction Attack against Federated Learning through Model Manipulation\").\n\n2) Shuaishuai Zhang, Jie Huang, Zeping Zhang, and Chunyang Qi. Compromise privacy in large-batch federated learning via malicious model parameters. In ICA3PP, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)