CAFE: Catastrophic Data Leakage in Federated Learning
Authors: Xiao Jin
Ruijie Du
Pin-Yu Chen
Tianyi Chen
Venue:  (2021)

Abstract: Private training data can be leaked through the gradient sharing mechanism deployed in machine learning systems, such as federated learning (FL).
Increasing batch size is often viewed as a promising defense strategy against data leakage. In this paper, we revisit this defense premise and propose an advanced data leakage attack to efficiently recover batch data from the shared aggregated gradients. 
We name our proposed method as \textit{\underline{c}atastrophic d\underline{a}ta leakage in \underline{f}ederated l\underline{e}arning (CAFE)}.
Comparing to existing data leakage attacks, CAFE demonstrates the ability to perform large-batch data leakage attack with high data recovery quality. 
Experimental results on vertical and horizontal FL settings have validated the effectiveness of CAFE in recovering private data from the shared aggregated gradients. 
Our results suggest that data participated in FL, especially the vertical case, have a high risk of being leaked from the training gradients. Our analysis implies unprecedented and practical data leakage risks in those learning settings.

Under review as a conference paper at ICLR 2021
CAFE: CATASTROPHIC DATA LEAKAGE IN FEDER- ATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Private training data can be leaked through the gradient sharing mechanism de- ployed in machine learning systems, such as federated learning (FL). Increasing batch size is often viewed as promising defense strategy against data leakage. In this paper, we revisit this defense premise and propose an advanced data leak- age attack to efficiently recover batch data from the shared aggregated gradients. We name our proposed method as catastrophic data leakage in federated learning (CAFE). Comparing to existing data leakage attacks, CAFE demonstrates the abil- ity to perform large-batch data leakage attack with high data recovery quality, Experimental results on vertical and horizontal FL settings have validated the ef- fectiveness of CAFE in recovering private data from the shared aggregated gradi- ents. Our results suggest that data participated in FL, especially the vertical case, have a high risk of being leaked from the training gradients. Our analysis implies unprecedented and practical data leakage risks in those learning settings.
INTRODUCTION
Federated learning (FL) (Chilimbi et al., 2014; Shokri & Shmatikov, 2015) is an emerging machine learning framework where a central server and multiple workers collaboratively train machine learning model. Most of existing FL methods consider the setting where each worker has data of a different set of subjects but their data share many common features. This setting is also referred to data partitioned or horizontal FL (HFL). Unlike the HFL setting, in many learning scenarios, multiple workers handle data about the same set of subjects, but each has a different set of features. This case arises in financial and healtheare applications (Chen et al.. 2020). In these examples, data owners (e.g., financial institutions and hospitals) have different records of those users their joint user base, so, by combining their features, they can establish a more accurate model. We refer to this setting as reature-partitionel or vertical FL (VFL).
Compared with existing distributed learning paradigms, FL raises new challenges including the heterogeneity of data and the privacy of data (McMahan et al., 2017). To protect data privacy, only model parameters and the change of parameters (e.g., gradients) are exchanged between server and workers (Li, 2014; landola et al., 2015). Recent works have studied how malicious worker can embed backdoors or replace the global model in FL (Bagdasaryan Xie et al., 2020). As exchanging gradients is often viewed as srivacy-preservigg protocols, little attention has been paid to information leakage from public shared gradients and batch identities.
al., 2018; Bhagoji al., 2019;
In this context, inferring private user data from the gradients has received growing interests (Fredrik- son et al., 2015; Hitaj et al., 2017; Melis et al., 2018). A popular method that was termed deep leakage from gradients (DLG) has been developed in (Zhu et al., 2019) that infers training data in an efficient way without using any generative models or prior information. However, DLG lacks generalizability on model architecture and weight distribution initialization (Geiping et al., 2020). In Zhao et al. (2020), an analytical approach has been developed to extract accurate labels from the gradients. Wang al. (2020) proposed novel gradient difference distance measure to improve recovery accuracy. However, all of them cannot scale up to the large-batch data leakage setting.
The contributions of this paper are summarized in the following.
1) We develop an advanced data leakage attack that we term CAFE to overcome the limitation of current data leakage attacks FL.. CAFE is able to recover large-scale data both in VFL and HFL.

---

Under review as a conference paper at ICLR 2021
(a) Original
(b) DLG (batch size 40)
(c) CAFE (batch size 10
Figure 1: Illustration of large-batch data leakage on CIFAR-10 from shared gradients in FL
2) Our large-batch data recovery is based on the novel use of data index alignment and internal representation alignment in FI., which can significantly improve the recovery performance.
3) The effectiveness and practical risk induced from our data leakage algorithm justified in the dynamic FL training setting when all parameters in the model are updated every iteration.
2
PRELIMINARY
FL can be categorized into horizontal and vertical FL settings (Kairouz al., 2019). In this section, provide necessary background of FL in this section.
Horizontal FL. HFL, data are distributed among local workers holding the same feature space. Suppose that there are M workers participating in the FL process and the number of samples in the dataset X is N. The dataset is denoted as == .m. ......... XM]", where X,, E R Nm Xp the local data partitioned to worker m, and p is the dimension of data feature space, Nm is the number of data samples partitioned to local worker m, and ΣM Nm N. Since all local data share the same feature space, each local worker computes the gradients independently and uploads them the server. The server receives all gradients from each local worker and uses gradient aggregation methods such as FedAvg (Konečný et al., 2016). Let the parameters of the model as and the loss function as L. Then the objective function of HFL can be defined as:
M
min
1
(():Xm) with L(0:Xm)
Σ x.)x,,)
Vertical FL. Different from HFL, in VFL, each local worker is associated with unique set of features. Each data sample x,, in dataset X' can be written as
X
(2)
where Xmm E Pm is the data partitioned worker and pm is the data dimension in local worker m. The label space [ can be regarded as a special feature and is partitioned to the server or certain local worker. Similar (1), the objective function of VFL can be written as:
N
min
C(0:x,1i.... ... ;.,
(3)
N
CATASTROPHIC DATA LEAKAGE FROM BATCH GRADIENTS
To realize large-scale data recovery from aggregated gradients, propose algorithm named as CAFE: Catastrophic dAta leakage in Federated IEarning. While CAFE can be applied to any type of data, without loss of generality, we image datasets throughout the paper.
2

---

Under review as a conference paper at ICLR 2021
Local worker
Upload gradiient
Iimiaeaa Model
Brondown Model
Kesu Liits
Upined gradiess
Dummy
Inkmooaas
Local
Eptial grediess
Server
Baraskont Mindel
gradienes
Broadtont Mole!
Dumony
Late
Upicad gradinsee
gradions
Uptosd Take
Raume lax
Local warker M
Real Aggreagted Gradients
Fake Aggreagted Grodients
Try match
Figure 2: Overview of CAFE in VEL
3.1 WIIY LARGE-BATCII DATA LEAKAGE ATTACK IS DIFFICULT?
We start by providing some intuition on the difficulty of performing large-batch data leakage from aggregated gradients based on the formulation of DLG (Zhu et al., 2019). Assume that N images are selected as the input for a certain learning iteration. We define the data batch as X' {x,, In Xxx E .,,, where H. W, represents the height, the width and the channel number of each image. Likewise, the batched 'recovered data" is denoted by X { Ihe & xH XW xC n ,, NJ. which have the same dimension as X. Then the objective function is
N
N
X arg min
Σ
VaL(O, Xr. Va.)
VoL(O, Un)
(4)
X
N
N
M=1
Note that in (4), the dimensions of the aggregated gradients is fixed. However, as the increases, the dimension of & and rise. When N sufficiently large, it will be more challenging to find the "right" solution & of (4) corresponding to the ground-truth dataset X' On the other hand, CAFE addresses this large-batch issue by data index alignment for batch data recovery, which can effectively exclude undesired solutions. We discuss a specific example in Appendix A.
As motivating example, Figure compares our proposed attack with DLG on batch of 40 images. The recovery quality of DLG is far from satisfactory. while CAFE can successfully recover all images the batch. is worth noting that because DLG is not effective on large-batch recovery, is suggested in Zhu et al. (2019) that increasing batch size could be a promising defense. However, the successful recovery of CAFE shows that such defense premise gives a false sense of security in data leakage and the current FL. is at risk, as large-batch data recovery can be accomplished.
3.2 CAFE IN VFL
In VFL. the server sends public key to local workers and decides the data index in each iteration of training and evaluation (Yang et al., 2019; Cheng et al., 2019). During the training process, local workers exchange their intermediate results with others to compute gradients and upload them. Therefore, the server has the access to both the model parameters and their gradients. Notably, CAFE can be readily applied to existing VFL protocols where the batch data index is assigned.
Figure 2 gives an overview of CAFE in the VFL setting. The blue part represents a normal VFL paradigm and the red part represents the CAFE attack. Since data are vertically partitioned among different workers, data index alignment turns out to be an inevitable step in the vertical training process, which provides the server (the attacker) opportunity control the selected batch data index. Suppose that there are M workers participating FL and the batch size N. The aggregated gradients can be denoted by
No. VoL(O, with X (x ......., X.x))
VoL(0.X*)
(5)
No
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 CAFE in VFL regullar VFL protocol and CAFE protocol
1: Initialize model parameters and generate fake data A for .,2,..., do
Server broadcasts global model to local workers (a of workers)
4:
for 2,..., .V do
5: Worker m takes real batch data
6: Worker m computes the intermediate results and exchanges them with other workers
7: Worker uses the exchanged intermediate results to compute local aggregated gradients
8: Worker uploads real local aggregated gradients the server.
end
10:
Server computes real global aggregated gradients VoL(0,
Server computes fake global aggregated gradients VoL(O, X²)
12:
Server computes CAFE loss: D(X'; X*) and V (xX"
13:
Server updates the batch data X' with V D=="
Server updates model parameters with VoL(0,XX)
15:
end for
A benign server will perform legitimate computations designed by FL protocol. However, as shown in Figure 2, a curious server can provide the same legitimate computation as henign server while simultaneously perform data recovery in stealthy manner. The server symmetrically generates fake images corresponding to the real ones. Once a batch of original data is selected, the server takes the corresponding fake batch and obtains the fake gradients
Nº No Σ VoL(O, with X [2 Sh2..... ... x.M...
VoL(O,2))
(6)
Algorithm 1 gives pseudo code that implements our CAFE attack in VFL cases. The key part in our algorithm is aligning the real data batch indices with the fake ones. We define the squared f2-norm of the difference between the real and fake aggregated gradients (7), Since the server has the access to the model parameters, the attacker is able to compute the gradient of fake data from the loss (7) and optimize the fake data for the purpose of recovering real data.
D(X"; VoL(O. "
3.3 AUXILIARY REGULARIZERS
In addition to the gradient matching loss in (7), we further introduce two regularization terms internal representation regularization and total variance (TV) norm. Motivated by (Geiping al., 2020), the input vectors of the first fully connected layer can be directly derived from the gradients, we define the real/fake inputs of the first fully connected layer at the /th iteration as 2'12¹ E N and we use & norm of their difference as what we call internal representation regularization.
To promote the smoothness of the fake images, we assume the TV norm of the real images constant, and compare it with the TV norm of the fake ones, TV(X)). For each image E If x W xC in data batch X its TV norm is denoted by TV(x) = Σ. En. [|x+1,,,, X1,w,el +
[ xx,,wel) As the result, the loss at the tth iteration ((("",2"") can be rewritten as:
1 ||VoL x*) VoL(O, 20) ( (TV( + +7||2' -201p (8)
where and 7 coefficients and 1 ((TV(&))=22 is the indicator function. We will provide ablation study in Section 4.5 to demonstrate the utility of these regularizers.
4

---

Under review as a conference paper at ICLR 2021
(50)
(100)
(200)
(300)
(450)
(600)
Original data
Figure 3: CAFE on Linnaeus
(Epoch: 50, 100, 200, 300, 450, 600, Original data)
3.4 CAFE HFL
Similarly, we can apply our CAFE algorithm to HFL as well. Let 2 denote the original batch data taken by local worker at the ith iteration. The gradients the parameters the fth iteration is
M
VoL(O,
Σ VoL(O,X M
1
x {27
XM
(9)
Symmetrically, we define the batch fake data and fake aggregated gradients
M
VoL(O, M VoL
X' = f&i
(10)
Due to space limitation, will provide the CAFE algorithm for HFL in Appendix B.
4 PERFORMANCE EVALUATION
4.1 EXPERIMENT SETUPS AND DATASETS
conduct experiments on CIFAR-10, CIFAR- 100 and Linnaeus 5 datasets in both HFL and VFL settings. All the fake data initialized uniformly and optimized by the normalized gradient descent method. Our algorithm can recover all the data participating in FL with a relative large batch size (more than 40). Scaling up VFL setting including workers.
hardware limits, CAFE leak many 2000 images in the
Evaluation metrics. To measure the data leakage performance, introduce peak signal-to-noise ratio (PSNR) value with mean squared error (MSE) defined in (11) and (12). Higher PSNR value of leaked data represents better performance of data recovery.
H
W
MSE.(X, x)
X iijc
&tjel?
(11)
HW
i=1
1=
PSNR = C
20 logio (maxx4jc) - 10 logio 10 (MSS,(xx)) x))
(12)
1.j
Baseline methods for comparison. We compare CAFE with three other baselines, (i) DLG (Zhu al., 2019), (ii) DLG given labels (iDLG) (Zhao al., 2020), and (iii) using cosine similarity to compare the real and fake gradients (Geiping et al., 2020). We implement the original DLG and our CAFE under the same model and optimization methods. We run the DLG on 50 single images respectively and compute the average iterations required to make the PSNR value of a single leaked image above 30. We also compute the expected iteration number per image leakage for our CAFE algorithm. Furthermore, we fix the batch size, and compare the PSNR value obtained by CAFE with that of DLG. We also test the impact of given labels on CAFE by using the techniques in (Zhao et al., 2020). Moreover, replacing the squared b norm term with the cosine similarity of two gradients (CAFE with cosine similarity) ii) loss proposed in (Geiping al., 2020), which only contains the TVV norm regularizer.
compare the performance of CAFE under different loss functions: i)
5

---

Under review as a conference paper at ICLR 2021
10
35
30
10
CIFAR-10
PSW
25
CIFAR-10
10
CIFAR-100 100
Linnaeus
20
Linnaeus
10
15
10
10
0
1000
2000
3000 4000
5000 6000 7000
1000
2000
3000 4000
5000 6000 7000
Iteration
lteration
(a) HFL Loss
(b) PSNR
80
10
50
L/LD
10
30
10
CIFAR-10
CIFAR-10
CIFAR-100
20
CIFAR-100
Linnaeus
Linnaeus
10
1000 2000 3000 4000 5000 8000 7000
1000 2000 3000 4000
5000
6000 7000
Iteration
Iteration
(c) VFL Loss
(d) VFL PSNR
Figure 4: CAFE loss ratio and PSNR curves
Table 1: CAFE vs DLG
Iterations
Datasets
CECAR-IOCIFAR-IO 100ERR--00LLAA Linnaeus
PSNR
Datasets
Batch size
I(DLG)
284.4
266.9
366.7
www.100OLiiinaann Linnaeus
10 4
9.50
6.00
9.50
Algorithm
20 x 4
6.75
3,86
4.75
CAFE
35.03
36.90
36.37
30 x 4
4.83
3.41
3.17
DLG
10.09
10.79
10.10
40 4
3.75
3.75
2.375
(b) Comparison of leakage performance. Higher
(a) Comparison of data leakage speed. Lower iteration count is faster.
PSNR is better. Batch size = 40.
Table 2: PSNR via Loss
PSNR
Datasets
PSNR
Datasets
CIFAR- LUCIFAAR-10LLimaqu
CIFAR- DOCIFAAR-OOLLnnaane
Loss
Loss
CAFE (())
35.03
36.90
36.37
CAFE ((8)
43.31
48.10
35.06
CAFE with cosine similarity
CAFE with cosine similarity
30.15
31.38
30.76
30.96
43.68
34.90
Loss in
Loss in (Geiping et al., 2020)
16.95
19.74
16.42
12.76
10.85
10.46
(Geiping et al., 2020)
(a) HFL
(b) VFL
(4 workers, batch ratio 0.1, batch size 10 x 4)
(4 workers, batch ratio 0.1, batch size 40)
4.2 CAFE HFL SETTINGS
In the HFL setting, we use neural network consisting of 2 convolutional layers and 3 fully con- nected layers. The number of output channels of the convolutional layers are and 128 respectively. The number of nodes of the first two fully connected layers are 512 and 256. The last layer is the softmax classification layer. We assume that 4 workers are involved HFL and each of them holds dataset including 100 images. The batch size of each worker in the training is 10, there 40 (10 x 4) images total participating per iteration. For each experiment, we initialize the fake data using uniform distribution and optimize them for 800 epochs.

---

Under review as a conference paper at ICLR 2021
Figures 4a and 4b show the CAFE loss curves and the PSNR curves the three datasets in HFL cases. In the loss ratio curve, we set the ratio of current CAFE loss and the initial CAFE loss <(0,x*)
The PSNR values are always above 35 at the end of each CAFE attacking process, suggesting high data recovery quality (see Figure 1 as an example). Figure shows the attacking process of CAFE on Linnaeus. Under CAFE, PSNR reaches 35 at the 450th epoch where the private training data are completely leaked visually.
2(0,x²) as label
Comparison with DLG baseline. Table la, we set the batch ratio in CAFE as 0.1 and compare it with DLG under different batch sizes. Clearly, CAFE outperforms DLG thanks to our novel design of large-batch data leakage attack. As shown in Table lb, DLG cannot obtain satisfactory results when the batch size increases to 40, while CAFE successfully recovers all the images. Due to similarity between iDLG and DLG, the results are in Appendix C.
Comparison with cosine similarity. Table 2a shows that the PSNR values are still above 30 if we use cosine similarity instead of lg norm. The slight drop in PSNR value may result from scaling ambiguity in cosine similarity. There is performance gap between the loss of CAFE and the loss in Geiping et al. (2020), which validates the importance of our proposed auxiliary regularizers.
Table 3: PSNR via Batch size
PSNR
Datasets
PSNR
Dataseis
CAAAR-OCIFAR-III 10CIFAR-100Linnaauu
Batch size
Batch size
10 per worker
35.03
36.90
36.37
8
41.80
44.42
39,96
20 per worker
33.14
33,99
36.32
40
59.51
65.00
41.37
30 per worker
32.31
33.21
35.96
80
$7.20
63.10
43.66
40 per worker
30.59
30.70
33,49
160
54.74
64.75
38.72
(a) HFL (4 workers, batch ratio 0.1)
(b) VFL (4 workers, batch ratio = 0.2)
30
Table 4: PSNR via Batch ratio
25
10
Training boss
PSNR
Datasets
ESNR
CIFAR-10 10
Linnaeus
CIFAR- 10 (VFL)
20
(HFL)
10
Batch ratio
PSNR
0.1
34.10
35.38
48.78
Training
0.05
0.02
37.96
34.49
35.66
32,92
55.46
48.45
10
102
10
10
20
Iteration on
0.01
35.39
36.56
46.46
Figure 5: PSNR and training loss curves
4.3 CAFE IN VFL SETTINGS
Since DLG cannot be applied in VFL protocol, we test the performance of CAFE on various factors. We slice one image into 4 small pieces. Each worker holds one piece and the feature space dimension of each piece is 16 16 3. The model is composed of 2 parts. The first part consists of convolutional layers and 3 fully connected layers for each worker. The second part only consists of the softmax layer. In the training process, the pieces are sent into the first part respectively and turn to vectors as intermediate results. Local workers then exchange their intermediate results, concatenate them and put them into the second part. We set the batch size as 40 in VFI.. Figure 4c and 4d show the CAFE loss curves and the PSNR curves on the three datascts in VFL cases. The data recovery is even better than the results in HFL. The PSNR values of CIFAR- 10 and CIFAR- 100 rise higher than 40. Same as the part in HFL, we put comparison with iDLG in Appendix C.
Comparison with cosine similarity. From Table 2b, we can conclude that the PSNR values still keep close to the ones by using CAFE. Scaling ambiguity in cosine similarity may also cause the drop in PSNR value. The performance gap between the loss of CAFE and the loss in Geiping et al. (2020) is much larger than the one in VFI., which indicates the utility of our auxiliary regularizers.
7

---

Under review as a conference paper at ICLR 2021
CIFAR-I0
CIFAR-100
Linnaeus
CAFE
8 =
==0
==0
8
Original data
Figure 6: Effect of auxiliary regularizers
4.4 ATTACKING WHILE FL
Previous works have shown that DLG performs better on an untrained model than trained one (Geiping et al., 2020). We also implement CAFE in the "attacking while learning' mode, in which the FL. process is ongoing. When the network is training, the selected batch data and the parameters of the model change every iteration, which may cause the attack loss to diverge. To address this issue, for each real data batch, we compute the real gradients and optimize the corresponding fake data à times. We demonstrate (100 epochs). Figure 5 gives the curves of the training loss and the corresponding PSNR value. The PSNR value still can be raised to relatively high value. It indicates that CAFE can be practical data leakage attack in a dynamic training environment of FL.
Linnaeus dataset, set k: 10 and stop CAFE after 1000 iterations
4.5 ABLATION STUDY
We test CAFE under different batch size, batch ratio, and with (without) auxiliary regularizers.
PSNR via Batch size. Table 3 shows that the PSNR values still keep above 30 when the batch size increases with fixed number of workers and batch ratio. The result implies that the increasing batch size has little influence on data leakage performance of CAFE.
PSNR via Batch ratio. In HFI,, workers participate in the learning setting and we the amount of data held by each worker as 500. In the VFL case, implement CAFE total of 800 images. In Table 4, we change the batch ratio from 0.1 to 0.01 while keeping the trained epochs as 800. For both settings, the data leakage performance keeps the same level.
Impact of auxiliary regularizers Table 6 Appendix D demonstrates the impact of auxiliary regularizers. From Figure adjusting the threshold & prevents images from being over blurred during the reconstruction process. TV norm can eliminate the noisy patterns on the recovered images and increase the PSNR. Images leaked without regularizing the Frobenius norm of the difference between the internal representations Z and Z may lose some details and causes the drop PSNR.
CONCLUSIONS
this paper, we uncover the risk çatastrophic data leakage in federatee learning (CAFE) through an algorithm that can perform large-batch data leakage with high data recovery quality. Extensive experimental results demonstrate that CAFE can recover large-scale private data from the shared aggregated gradients on both vertical and horizontal FL settings, overcoming the batch limitation problem in current data leakage attacks. Our advanced data leakage attack and its stealthy nature suggests practical data privacy concerns in FL and poses challenges on future defenses.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. arXiv, p 2018.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634- 643, Long Beach, California, 2019.
Tianyi Chen, Xiao Jin, Yuejiao Sun, and Wotao Yin. VAFL: method of vertical asynchronous federated learning. arXiv preprint:2007.06600 2020.
K. Cheng, T. Fan, Yilun Jin, Yang Liu, Tianjian Chen, and Qiang Yang. Securcboost: lossless federated learning framework. arXiv, eprint:1901.0877 2019.
Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium Operating Systems Design and Implementation (OSDI 14), pp. 571-582, Broomfield, CO, October 2014. USENIX Association.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Confer- ence on Computer and Communications Security, pp. 1322-1133,, New York, NY, USA, 10 2015. Association for Computing Machinery.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy to break privacy in federated learning? arXiv, eprint:2003. 14055,, 2020.
Briland Hitaj, Giuseppe Ateniese, and Fernando Pérez-Cruz. Deep models under the GAN: infor- mation leakage from collaborative deep learning. arXiv, 1rrint:/002.0777
2017.
Forrest N. landola, Khalid Ashraf, Matthew W. Moskewicz, and Kurt Keutzer. Firecaffe; near-linear acceleration of deep neural network training on compute clusters. arXiv, eprint: 2015.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint: 1912.
191: 1912.04977, 04997, December 2019.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv 10e.:1n::1610.02527,, October 2016.
Mu Li. Scaling distributed machine learning with the parameter server. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, pp. 1-1, USA, 08 2014. USENIX Association.
H. McMahan, Eider Moore, D. Ramage, S. Hampson, and Blaise Agúera Arcas. Communication- efficient learning of deep networks from decentralized data. In AISTATS, Ft. Lauderdale, FL, USA, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Inference attacks against collaborative learning. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148-162, New York, NY, USA, 2018. Association for Computing Machinery.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS $15, pp. 1310-1321, New York, NY, USA, 2015. Association for Computing Machinery.
Yijue Wang, Jieren Deng, Dan Guo, Chenghong Wang, Xianrui Meng, Hang Liu, Caiwen Ding, and Sanguthevar Rajasekaran. Sapag: A self-adaptive privacy attack from gradients. arXiv, eprint:2009, 2prin::2009006228, 2020.
Chulin Xie, Keli Huang, Pin-Yt Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against feder- ated learning. In International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.

---

Under review as a conference paper at ICLR 2021
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. 10(2). January 2019.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. arXiv, eprint:2001, 020600, 2020.
idlg: Improved deep leakage from gradients.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems 32, pp. 14774-144 14784, Vancouver, Canada, 2019.
10


Reviews:
[{"review_id": "lLnPaIM72P3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an attack to extract information about training data from gradient updates sent as part of a federated learning setting.\n\nThe description of the attack setting and the attack algorithm is provided at a high level and detailed description is missing, making it hard to understand the novelty of the contribution. Experimental results do show that the attack is stronger than previous work. However, the overall presentation of the paper could be improved to be ready for a publication. Some suggestions listed below.\n\n[Attack setting] It seems that the paper departs from some of the related work on information leakage by considering an attacker that can tamper with the federated learning process. Hence, the attacker is malicious and not benign. The authors should make this distinction clear if it is indeed the case. Commenting on why this malicious activity will not be noticed by the workers is important. For example, Figure 5 indeed shows that training accuracy is impacted by the attack.\nHow does this attack compare to the active attacker in the work by Melis et al?\n\nThe paper considers only 4 workers in most experiments. Federated learning usually has many more participants. Is this a problem? Do same workers need to be contacted at every iteration? Are there any assumptions on same data being used in each iteration.\n\n[Attack algorithm]\nThe algorithm makes use of \"data index alignment\". Some guidance on what it means would facilitate the reading and understanding of the algorithm. Red part of Algorithm 1 should be expanded. How do these values get computed? Do they replace the blue parts or complement them?\n\n[Presentation]\n“As a motivating example, Figure 1 compares”: it would be best to motivate the algorithm key insight and not its improved performance over previous work that was already mentioned.\n“However, as shown in Figure 2, a curious server can provide the same legitimate computation as a benign server while simultaneously perform data recovery in a stealthy manner. The server symmetrically generates fake” The figure does not describe how fake parameters are computed and it is not clear how pictorial representation shows this.\n\nMinor details\nFigure 1 captions: why 40 vs 10 x 4; “workers participating FL” -> workers participating in FL; please consider a better title for “4.4 ATTACKING WHILE FL”                                         ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BYFnLabnNEs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission considers the problem of reconstructing private data from gradients in a Federated Learning system, which has been recently shown to a threat in distributed learning systems. Two types of federated learning systems are considered. Vertical federated learning (VFL) refers to the case where different agents hold different features of the same data points while  Horizontal federated learning (HFL) refers to the case where different agents how all the features of different subsets of the data.\n\nPrevious attacks solve an optimization problem that aims to infer the data by minimizing the mismatch between real gradients and fake gradients. This method suffers difficulty when the number of samples in one round is large. The paper proposes CAFE, which takes advantage of the fact that in vertical federated learning (VFL) systems, the server can identify the indices of the samples that are selected in each round. This extra information help reduce unwanted solutions in the optimization problem and help improve the reconstruction performance. The authors conduct experiments to show that the proposed algorithm outperforms previous works.\n\nThe paper shows that data leakage from gradients is a potential threat in VFL systems even when the batch size is large. However, I have the following concerns about the paper.\n\n1. The paper claims that the attack also works for the HFL setting. However, this is no well justified for the following reasons:\n(1) The assumption that the server knows the indices of the samples that are selected in each round is not valid in general for the HFL setting since each agent can sample a batch locally.\n(2) In HFL settings, it is generally assumed that the number of agents is large and each agent only participates in a few rounds, which is not considered in the experiments in the submission.\n2. In the experiments, it is shown that as the number of training epochs grows, better inference on the private data can be made. It would be better if the authors can also include the training error on each epoch in the same plot. It is believable that if the training goes on forever,  enough information can be inferred about the training samples. However, it might be good to see whether the server can infer the train samples before the model has already converged.\n\nI hope the authors can address my above concerns in the response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "17fnXB-rVHO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the data leakage issue in the federated learning. More precisely, when the servers have access to model parameters and gradients. It can recover the input data via gradient matching, and the authors claim that their method performs well even with large training batch sizes, e.g. over 40. Finally, the author also studies the possibility of attacking during learning, where they suggest that multiple updates of fake data helps. However, their contribution seems incremental, gradient matching is used in previous literature [zhu et al 2019], and their main modification is extra two regularization terms: total variation and internal representation regularization, and a data index alignment technique (whose exact meaning is unclear in the paper).\n\nThe following are some questions:\n\nWhat does index alignment mean? Is that the server controls the indexes of samples chosen at each iteration? This seems to be very restrictive in practice, especially for horizontal federated learning.\n\nDoes the server have access to the aggregated grads from each worker separately or the workers aggregate all the gradients before sending them back to the server? The second scenario cab be achieved while secure aggregation technique.\n\nIn vertical federated learning, the gradients of part 1 of the network does not need to be exchanged with the server, as there is no average  operation needed, even the parameter itself does not need to be transferred to the server for the same reason, will your method work under this setting?\n\nSome terms are not properly defined, such as normalized gradient descent, batch ratio, et. al.\n\nOther questions:\n\nWhat does the iterations represent in table 1a? Is that the number of iterations need to reach a 35 PSNR?\n\nUsing cosine dissimilarity decreases the PSNR, I assume this is because PSNR penalize the scale, is there noticeable degradation visually when using cosine dissimilarity?\n\nIn the attack during learning scenario, is there any intuition why optimizing fake data multiple times works better?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-b7PWDT8np", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces CAFE, a novel training algorithm to leak training data in a federated learning setup. Extending from \"deep leakage from gradient\" fake images are optimised with respect to the difference observed from the client gradients (i.e. with the real images) and the one observed with the current version of the fake image. However, DLG does not work when the mini-batch size increases due to a messy gradient representation. In this work, the authors propose to keep track of the batch index. Indeed, it may happen that the server decides of the batch index corresponding to the training data that will be used by the client during the local training. Within such conditions, a malicious server can easily store fake images corresponding to specific indices and therefore optimise correctly each fake images w.r.t the corresponding real image. \n\nIt is clear from the obtained results that this method works, and that images are recovered. However, I am unsure about the relevance of the experimental protocol. 1. If the server does not ask for specific indices (and it is pretty common), the method is equivalent to DLG (i.e. does not work well) with large batches. 2. What if we don't have the gradients ? A common way of doing FL is to simply communicate the locally trained weights (with multiple local epochs). As specified in the introduction (point 3), the proposed method wouldn't work in this realistic scenario. \n\nThen, I found Section 3.3 unclear is some aspects. Are the two proposed regularisation methods relying on the \"real\" image ? If so, isn't this a strong bias ? (we are not expected to have the input images). I suppose that this comes from the citation to the work of Geipinget al., 2020. However, these two paragraph should be re-written to clearly explain how we can extract the input vector and how it relates to eq. 8. \"To promote the smoothness of the fake images, we assume the TV norm of the real images as a constant,\" -> We can't use the real image here, so it is not valid. \n\nPros: \n+ In the given conditions, CAFE clearly outperforms the other approach to leak training data from gradients during FL.\n+ Very simple attack to implement.\n\nCons:\n- The conditions necessary to the success of the proposed methods seem to be quite strong and not really connected to a realistic FL framework. \n- Small ideas can lead to drastic changes in the field, but the core idea of the paper is to solely store batch indices. \n\nRemarks:\n- \"In this section, we provide necessary background of FL [in this section].\"\n- Figure 2 should be checked. \"Aggreaged\", \"upload fake gradient\" only once .\n- What are t and b in Eq. 5. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses attacks on federated learning. The reviewers had the following concerns:
- The assumption of knowledge of batch indices is unrealistic in an HFL setting
- The setup only works when doing a single epoch (I believe the authors claim that it is applicable in more general settings, but evidence to that effect has not been provided)
- The novelty of the approach is somewhat limited.
- The description of the algorithm and comparison to prior work could be clearer.

I raised the question of whether the reviewers would be more positive if there were no claimed results on HFL. They still did not seem positive enough to justify acceptance (due to the other reasons mentioned above).
Decision: Reject