Year: 2020

Abstract:
Private training data can be leaked through the gradient sharing mechanism deployed in machine learning systems, such as federated learning (FL). Increasing batch size is often viewed as a promising defense strategy against data leakage. In this paper, we revisit this defense premise and propose an advanced data leakage attack to efficiently recover batch data from the shared aggregated gradients. We name our proposed method as catastrophic data leakage in federated learning (CAFE). Comparing to existing data leakage attacks, CAFE demonstrates the ability to perform large-batch data leakage attack with high data recovery quality. Experimental results on vertical and horizontal FL settings have validated the effectiveness of CAFE in recovering private data from the shared aggregated gradients. Our results suggest that data participated in FL, especially the vertical case, have a high risk of being leaked from the training gradients. Our analysis implies unprecedented and practical data leakage risks in those learning settings.

heading: 1 INTRODUCTION
text: Federated learning (FL) (Chilimbi et al., 2014; Shokri & Shmatikov, 2015) is an emerging machine learning framework where a central server and multiple workers collaboratively train a machine learning model. Most of existing FL methods consider the setting where each worker has data of a different set of subjects but their data share many common features. This setting is also referred to data partitioned or horizontal FL (HFL). Unlike the HFL setting, in many learning scenarios, multiple workers handle data about the same set of subjects, but each has a different set of features. This case arises in financial and healthcare applications (Chen et al., 2020). In these examples, data owners (e.g., financial institutions and hospitals) have different records of those users in their joint user base, so, by combining their features, they can establish a more accurate model. We refer to this setting as feature-partitioned or vertical FL (VFL).
Compared with existing distributed learning paradigms, FL raises new challenges including the heterogeneity of data and the privacy of data (McMahan et al., 2017). To protect data privacy, only model parameters and the change of parameters (e.g., gradients) are exchanged between server and workers (Li, 2014; Iandola et al., 2015). Recent works have studied how a malicious worker can embed backdoors or replace the global model in FL (Bagdasaryan et al., 2018; Bhagoji et al., 2019; Xie et al., 2020). As exchanging gradients is often viewed as privacy-preserving protocols, little attention has been paid to information leakage from public shared gradients and batch identities.
In this context, inferring private user data from the gradients has received growing interests (Fredrikson et al., 2015; Hitaj et al., 2017; Melis et al., 2018). A popular method that was termed deep leakage from gradients (DLG) has been developed in (Zhu et al., 2019) that infers training data in an efficient way without using any generative models or prior information. However, DLG lacks generalizability on model architecture and weight distribution initialization (Geiping et al., 2020). In Zhao et al. (2020), an analytical approach has been developed to extract accurate labels from the gradients. Wang et al. (2020) proposed a novel gradient difference as a distance measure to improve recovery accuracy. However, all of them cannot scale up to the large-batch data leakage setting.
The contributions of this paper are summarized in the following.
1) We develop an advanced data leakage attack that we term CAFE to overcome the limitation of current data leakage attacks on FL. CAFE is able to recover large-scale data both in VFL and HFL.
2) Our large-batch data recovery is based on the novel use of data index alignment and internal representation alignment in FL, which can significantly improve the recovery performance.
3) The effectiveness and practical risk induced from our data leakage algorithm is justified in the dynamic FL training setting when all parameters in the model are updated every iteration.

heading: 2 PRELIMINARY
text: FL can be categorized into horizontal and vertical FL settings (Kairouz et al., 2019). In this section, we provide necessary background of FL in this section.
Horizontal FL. In HFL, data are distributed among local workers holding the same feature space. Suppose that there are M workers participating in the FL process and the number of samples in the dataset X is N . The dataset is denoted as X := [X1, . . . ,Xm, . . . ,XM ]T , where Xm ∈ RNm×p is the local data partitioned to worker m, and p is the dimension of data feature space, Nm is the number of data samples partitioned to local worker m, and ∑M m=1Nm = N . Since all local data share the same feature space, each local worker computes the gradients independently and uploads them to the server. The server receives all gradients from each local worker and uses gradient aggregation methods such as FedAvg (Konečnỳ et al., 2016). Let the parameters of the model as θ and the loss function as L. Then the objective function of HFL can be defined as:
min θ
1
N M∑ m=1 L(θ;Xm) with L(θ;Xm) := ∑ n∈Nm L(θ;xn) (1)
Vertical FL. Different from HFL, in VFL, each local worker m is associated with a unique set of features. Each data sample xn in dataset X can be written as
xn = [x T n1, . . . ,x T nm, . . . ,x T nM ] T (2) where xnm ∈ Rpm is the data partitioned to worker m and pm is the data dimension in local worker m. The label space {yn}Nn=1 can be regarded as a special feature and is partitioned to the server or a certain local worker. Similar to (1), the objective function of VFL can be written as:
min θ
1
N N∑ n=1 L(θ;xn1; . . . ;xnM ) (3)

heading: 3 CATASTROPHIC DATA LEAKAGE FROM BATCH GRADIENTS
text: To realize large-scale data recovery from aggregated gradients, we propose our algorithm named as CAFE: Catastrophic dAta leakage in Federated lEarning. While CAFE can be applied to any type of data, without loss of generality, we use image datasets throughout the paper.

heading: 3.1 WHY LARGE-BATCH DATA LEAKAGE ATTACK IS DIFFICULT?
text: We start by providing some intuition on the difficulty of performing large-batch data leakage from aggregated gradients based on the formulation of DLG (Zhu et al., 2019). Assume thatN images are selected as the input for a certain learning iteration. We define the data batch as X = {xn, yn|xn ∈ RH×W×C , n = 1, 2, . . . , N}, where H,W,C represents the height, the width and the channel number of each image. Likewise, the batched ‘recovered data’ is denoted by X̂ = {x̂n, ŷn|x̂n ∈ RH×W×C , n = 1, 2, . . . , N}, which have the same dimension as X . Then the objective function is
X̂ ∗ = argmin X̂ ∥∥∥∥∥ 1N N∑
n=1
∇θL(θ,xn, yn)− 1
N N∑ n=1 ∇θL(θ, x̂n, ŷn) ∥∥∥∥∥ 2
(4)
Note that in (4), the dimensions of the aggregated gradients is fixed. However, as the N increases, the dimension of X̂ and X rise. When N is sufficiently large, it will be more challenging to find the “right” solution X̂ of (4) corresponding to the ground-truth dataset X . On the other hand, CAFE addresses this large-batch issue by data index alignment for batch data recovery, which can effectively exclude undesired solutions. We discuss a specific example in Appendix A.
As a motivating example, Figure 1 compares our proposed attack with DLG on a batch of 40 images. The recovery quality of DLG is far from satisfactory, while CAFE can successfully recover all images in the batch. It is worth noting that because DLG is not effective on large-batch recovery, it is suggested in Zhu et al. (2019) that increasing batch size could be a promising defense. However, the successful recovery of CAFE shows that such defense premise gives a false sense of security in data leakage and the current FL is at risk, as large-batch data recovery can be accomplished.

heading: 3.2 CAFE IN VFL
text: In VFL, the server sends public key to local workers and decides the data index in each iteration of training and evaluation (Yang et al., 2019; Cheng et al., 2019). During the training process, local workers exchange their intermediate results with others to compute gradients and upload them. Therefore, the server has the access to both the model parameters and their gradients. Notably, CAFE can be readily applied to existing VFL protocols where the batch data index is assigned.
Figure 2 gives an overview of CAFE in the VFL setting. The blue part represents a normal VFL paradigm and the red part represents the CAFE attack. Since data are vertically partitioned among different workers, data index alignment turns out to be an inevitable step in the vertical training process, which provides the server (the attacker) an opportunity to control the selected batch data index. Suppose that there are M workers participating FL and the batch size is N . The aggregated gradients can be denoted by
∇θL(θ,X t) = 1
Nb Nb∑ n=1 ∇θL(θ,X tn) with X tn = [xtn1,xtn2, . . . ,xtnM ]. (5)
Algorithm 1 CAFE in VFL ( regular VFL protocol and CAFE protocol )
1: Initialize model parameters θ and generate fake data X̂ 2: for t = 1, 2, . . . , T do 3: Server broadcasts the global model to all local workers (a total of M workers)
4: for m = 1, 2, . . . ,M do 5: Worker m takes real batch data 6: Worker m computes the intermediate results and exchanges them with other workers 7: Worker m uses the exchanged intermediate results to compute local aggregated gradients 8: Worker m uploads real local aggregated gradients to the server. 9: end for
10: Server computes real global aggregated gradients∇θL(θ,X t)
11: Server computes the fake global aggregated gradients∇θL(θ, X̂ t) 12: Server computes CAFE loss: D(X t; X̂ t) and∇X̂ tD(X
t; X̂ t) 13: Server updates the batch data X̂ t with∇X̂ tD(X t; X̂ t)
14: Server updates the model parameters θ with∇θL(θ,X t) 15: end for
A benign server will perform legitimate computations designed by FL protocol. However, as shown in Figure 2, a curious server can provide the same legitimate computation as a benign server while simultaneously perform data recovery in a stealthy manner. The server symmetrically generates fake images corresponding to the real ones. Once a batch of original data is selected, the server takes the corresponding fake batch and obtains the fake gradients as
∇θL(θ, X̂ t) = 1
Nb Nb∑ n=1 ∇θL(θ, X̂ tn) with X̂ tn = [x̂tn1, x̂tn2, . . . , x̂tnM ]. (6)
Algorithm 1 gives a pseudo code that implements our CAFE attack in VFL cases. The key part in our algorithm is aligning the real data batch indices with the fake ones. We define the squared `2-norm of the difference between the real and fake aggregated gradients in (7). Since the server has the access to the model parameters, the attacker is able to compute the gradient of fake data from the loss in (7) and optimize the fake data for the purpose of recovering real data.
D(X t; X̂ t) = ∥∥∇θL(θ,X t)−∇θL(θ, X̂ t)∥∥2. (7)

heading: 3.3 AUXILIARY REGULARIZERS
text: In addition to the gradient matching loss in (7), we further introduce two regularization terms – internal representation regularization and total variance (TV) norm. Motivated by (Geiping et al., 2020), the input vectors of the first fully connected layer can be directly derived from the gradients, we define the real/fake inputs of the first fully connected layer at the tth iteration as Zt/Ẑt ∈ RN×P and we use `2 norm of their difference as what we call internal representation regularization.
To promote the smoothness of the fake images, we assume the TV norm of the real images as a constant, ξ, and compare it with the TV norm of the fake ones, TV(X̂ ). For each image x ∈ RH×W×C in data batch X t, its TV norm is denoted by TV(x) = ∑ c ∑ h,w [ |xh+1,w,c − xh,w,c|+
|xh,w+1,c − xh,w,c| ] . As the result, the loss at the tth iteration D(X t, X̂ t) can be rewritten as:
D(X t; X̂ t) = ∥∥∇θL(θ,X t)−∇θL(θ, X̂ t)∥∥2 + βTV (X̂ ) · 1{TV (X̂ )−ξ≥0} + γ∥∥Zt − Ẑt∥∥2F , (8)
where β and γ are coefficients and 1{TV (X̂ )−ξ≥0} is the indicator function. We will provide an ablation study in Section 4.5 to demonstrate the utility of these regularizers.
(Epoch: 50, 100, 200, 300, 450, 600, Original data)

heading: 3.4 CAFE IN HFL
text: Similarly, we can apply our CAFE algorithm to HFL as well. Let X tm denote the original batch data taken by local worker m at the tth iteration. The gradients of the parameters at the tth iteration is
∇θL(θ,X t) = 1
M M∑ m=1 ∇θL(θ,X tm) , X t = {X t1 ,X t2 , . . . ,X tm, . . . ,X tM}. (9)
Symmetrically, we define the batch fake data and fake aggregated gradients as
∇θL(θ, X̂ t) = 1
M M∑ m=1 ∇θL(θ, X̂ tm) , X̂ t = {X̂ t1 , X̂ t2 , . . . , X̂ tm, . . . , X̂ tM}. (10)
Due to space limitation, we will provide the CAFE algorithm for HFL in Appendix B.

heading: 4 PERFORMANCE EVALUATION

heading: 4.1 EXPERIMENT SETUPS AND DATASETS
text: We conduct experiments on CIFAR-10, CIFAR-100 and Linnaeus 5 datasets in both HFL and VFL settings. All the fake data are initialized uniformly and optimized by the normalized gradient descent method. Our algorithm can recover all the data participating in FL with a relative large batch size (more than 40). Scaling up to our hardware limits, CAFE can leak as many as 2000 images in the VFL setting including 4 workers.
Evaluation metrics. To measure the data leakage performance, we introduce peak signal-to-noise ratio (PSNR) value with mean squared error (MSE) defined in (11) and (12). Higher PSNR value of leaked data represents better performance of data recovery.
MSEc(x, x̂) = 1
HW H∑ i=1 W∑ j=1 [xijc − x̂ijc]2 (11)
PSNR(x, x̂) = 1
C C∑ c=1 [ 20 log10(max i,j xijc)− 10 log10(MSEc(x, x̂)) ] . (12)
Baseline methods for comparison. We compare CAFE with three other baselines, (i) DLG (Zhu et al., 2019), (ii) DLG given labels (iDLG) (Zhao et al., 2020), and (iii) using cosine similarity to compare the real and fake gradients (Geiping et al., 2020). We implement the original DLG and our CAFE under the same model and optimization methods. We run the DLG on 50 single images respectively and compute the average iterations required to make the PSNR value of a single leaked image above 30. We also compute the expected iteration number per image leakage for our CAFE algorithm. Furthermore, we fix the batch size, and compare the PSNR value obtained by CAFE with that of DLG. We also test the impact of given labels on CAFE by using the techniques in (Zhao et al., 2020). Moreover, we compare the performance of CAFE under different loss functions: i) replacing the squared `2 norm term with the cosine similarity of two gradients (CAFE with cosine similarity) ii) loss proposed in (Geiping et al., 2020), which only contains the TV norm regularizer.

heading: 4.2 CAFE IN HFL SETTINGS
text: In the HFL setting, we use a neural network consisting of 2 convolutional layers and 3 fully connected layers. The number of output channels of the convolutional layers are 64 and 128 respectively. The number of nodes of the first two fully connected layers are 512 and 256. The last layer is the softmax classification layer. We assume that 4 workers are involved in HFL and each of them holds a dataset including 100 images. The batch size of each worker in the training is 10, so there are 40 (10 × 4) images in total participating per iteration. For each experiment, we initialize the fake data using uniform distribution and optimize them for 800 epochs.
Figures 4a and 4b show the CAFE loss curves and the PSNR curves on the three datasets in HFL cases. In the loss ratio curve, we set the ratio of current CAFE loss and the initial CAFE loss L(θ,X t) L(θ,X 0) as label y. The PSNR values are always above 35 at the end of each CAFE attacking process, suggesting high data recovery quality (see Figure 1 as an example). Figure 3 shows the attacking process of CAFE on Linnaeus. Under CAFE, PSNR reaches 35 at the 450th epoch where the private training data are completely leaked visually.
Comparison with DLG baseline. In Table 1a, we set the batch ratio in CAFE as 0.1 and compare it with DLG under different batch sizes. Clearly, CAFE outperforms DLG thanks to our novel design of large-batch data leakage attack. As shown in Table 1b, DLG cannot obtain satisfactory results when the batch size increases to 40, while CAFE successfully recovers all the images. Due to similarity between iDLG and DLG, the results are in Appendix C.
Comparison with cosine similarity. Table 2a shows that the PSNR values are still above 30 if we use cosine similarity instead of `2 norm. The slight drop in PSNR value may result from scaling ambiguity in cosine similarity. There is a performance gap between the loss of CAFE and the loss in Geiping et al. (2020), which validates the importance of our proposed auxiliary regularizers.
Table 3: PSNR via Batch size
Batch size
PSNR Datasets CIFAR-10CIFAR-100Linnaeus
10 per worker 35.03 36.90 36.37 20 per worker 33.14 33.99 36.32 30 per worker 32.31 33.21 35.96 40 per worker 30.59 30.70 35.49
(a) HFL (4 workers, batch ratio = 0.1)
Batch size
PSNR Datasets CIFAR-10CIFAR-100Linnaeus
8 41.80 44.42 39.96 40 59.51 65.00 41.37 80 57.20 63.10 43.66 160 54.74 64.75 38.72 (b) VFL (4 workers, batch ratio = 0.2)
Table 4: PSNR via Batch ratio
Batch ratio
PSNR Datasets CIFAR-10
(HFL) Linnaeus CIFAR-10 (VFL)
0.1 34.10 35.38 48.78 0.05 34.49 32.92 55.46 0.02 37.96 35.66 48.45 0.01 35.39 36.56 46.46
10 1
10 2
10 3
10 4
Iteration
10
15
20
25
30
PS NR
10 −3
10 −2
10 −1
Tr ain
ing lo
ss
PSNR Training loss
Figure 5: PSNR and training loss curves

heading: 4.3 CAFE IN VFL SETTINGS
text: Since DLG cannot be applied in VFL protocol, we test the performance of CAFE on various factors. We slice one image into 4 small pieces. Each worker holds one piece and the feature space dimension of each piece is 16 × 16 × 3. The model is composed of 2 parts. The first part consists of 2 convolutional layers and 3 fully connected layers for each worker. The second part only consists of the softmax layer. In the training process, the pieces are sent into the first part respectively and turn to vectors as intermediate results. Local workers then exchange their intermediate results, concatenate them and put them into the second part. We set the batch size as 40 in VFL. Figure 4c and 4d show the CAFE loss curves and the PSNR curves on the three datasets in VFL cases. The data recovery is even better than the results in HFL. The PSNR values of CIFAR-10 and CIFAR-100 rise higher than 40. Same as the part in HFL, we put comparison with iDLG in Appendix C.
Comparison with cosine similarity. From Table 2b, we can conclude that the PSNR values still keep close to the ones by using CAFE. Scaling ambiguity in cosine similarity may also cause the drop in PSNR value. The performance gap between the loss of CAFE and the loss in Geiping et al. (2020) is much larger than the one in VFL, which indicates the utility of our auxiliary regularizers.

heading: 4.4 ATTACKING WHILE FL
text: Previous works have shown that DLG performs better on an untrained model than a trained one (Geiping et al., 2020). We also implement CAFE in the ‘attacking while learning’ mode, in which the FL process is ongoing. When the network is training, the selected batch data and the parameters of the model change every iteration, which may cause the attack loss to diverge. To address this issue, for each real data batch, we compute the real gradients and optimize the corresponding fake data k times. We demonstrate on Linnaeus dataset, set k = 10 and stop CAFE after 1000 iterations (100 epochs). Figure 5 gives the curves of the training loss and the corresponding PSNR value. The PSNR value still can be raised to a relatively high value. It indicates that CAFE can be a practical data leakage attack in a dynamic training environment of FL.

heading: 4.5 ABLATION STUDY
text: We test CAFE under different batch size, batch ratio, and with (without) auxiliary regularizers.
PSNR via Batch size. Table 3 shows that the PSNR values still keep above 30 when the batch size increases with fixed number of workers and batch ratio. The result implies that the increasing batch size has little influence on data leakage performance of CAFE.
PSNR via Batch ratio. In HFL, 4 workers participate in the learning setting and we fix the amount of data held by each worker as 500. In the VFL case, we implement CAFE on a total of 800 images. In Table 4, we change the batch ratio from 0.1 to 0.01 while keeping the trained epochs as 800. For both settings, the data leakage performance keeps at the same level.
Impact of auxiliary regularizers Table 6 in Appendix D demonstrates the impact of auxiliary regularizers. From Figure 6, adjusting the threshold ξ prevents images from being over blurred during the reconstruction process. TV norm can eliminate the noisy patterns on the recovered images and increase the PSNR. Images leaked without regularizing the Frobenius norm of the difference between the internal representations Z and Ẑ may lose some details and causes the drop of PSNR.

heading: 5 CONCLUSIONS
text: In this paper, we uncover the risk of catastrophic data leakage in federated learning (CAFE) through an algorithm that can perform large-batch data leakage with high data recovery quality. Extensive experimental results demonstrate that CAFE can recover large-scale private data from the shared aggregated gradients on both vertical and horizontal FL settings, overcoming the batch limitation problem in current data leakage attacks. Our advanced data leakage attack and its stealthy nature suggests practical data privacy concerns in FL and poses new challenges on future defenses.

Review summaries:
This work introduces CAFE, a novel training algorithm to leak training data in a federated learning setup. Extending from "deep leakage from gradient" fake images are optimised with respect to the difference observed from the client gradients (i.e. with the real images) and the one observed with the current version of the fake image. However, DLG does not work when the mini-batch size increases due to a messy gradient representation. In this work, the authors propose to keep track of the batch index. Indeed, it may happen that the server decides of the batch index corresponding to the training data that will be used by the client during the local training. Within such conditions, a malicious server can easily store fake images corresponding to specific indices and therefore optimise correctly each fake images w.r.t the corresponding real image. 

This paper studies the data leakage issue in the federated learning. More precisely, when the servers have access to model parameters and gradients. It can recover the input data via gradient matching, and the authors claim that their method performs well even with large training batch sizes, e.g. over 40. Finally, the author also studies the possibility of attacking during learning, where they suggest that multiple updates of fake data helps. However, their contribution seems incremental, gradient matching is used in previous literature [zhu et al 2019], and their main modification is extra two regularization terms: total variation and internal representation regularization, and a data index alignment technique (whose exact meaning is unclear in the paper).

The submission considers the problem of reconstructing private data from gradients in a Federated Learning system, which has been recently shown to a threat in distributed learning systems. Two types of federated learning systems are considered. Vertical federated learning (VFL) refers to the case where different agents hold different features of the same data points while  Horizontal federated learning (HFL) refers to the case where different agents how all the features of different subsets of the data.