CAFE: Catastrophic Data Leakage in Federated Learning

1 INTRODUCTION . Federated learning ( FL ) ( Chilimbi et al. , 2014 ; Shokri & Shmatikov , 2015 ) is an emerging machine learning framework where a central server and multiple workers collaboratively train a machine learning model . Most of existing FL methods consider the setting where each worker has data of a different set of subjects but their data share many common features . This setting is also referred to data partitioned or horizontal FL ( HFL ) . Unlike the HFL setting , in many learning scenarios , multiple workers handle data about the same set of subjects , but each has a different set of features . This case arises in financial and healthcare applications ( Chen et al. , 2020 ) . In these examples , data owners ( e.g. , financial institutions and hospitals ) have different records of those users in their joint user base , so , by combining their features , they can establish a more accurate model . We refer to this setting as feature-partitioned or vertical FL ( VFL ) . Compared with existing distributed learning paradigms , FL raises new challenges including the heterogeneity of data and the privacy of data ( McMahan et al. , 2017 ) . To protect data privacy , only model parameters and the change of parameters ( e.g. , gradients ) are exchanged between server and workers ( Li , 2014 ; Iandola et al. , 2015 ) . Recent works have studied how a malicious worker can embed backdoors or replace the global model in FL ( Bagdasaryan et al. , 2018 ; Bhagoji et al. , 2019 ; Xie et al. , 2020 ) . As exchanging gradients is often viewed as privacy-preserving protocols , little attention has been paid to information leakage from public shared gradients and batch identities . In this context , inferring private user data from the gradients has received growing interests ( Fredrikson et al. , 2015 ; Hitaj et al. , 2017 ; Melis et al. , 2018 ) . A popular method that was termed deep leakage from gradients ( DLG ) has been developed in ( Zhu et al. , 2019 ) that infers training data in an efficient way without using any generative models or prior information . However , DLG lacks generalizability on model architecture and weight distribution initialization ( Geiping et al. , 2020 ) . In Zhao et al . ( 2020 ) , an analytical approach has been developed to extract accurate labels from the gradients . Wang et al . ( 2020 ) proposed a novel gradient difference as a distance measure to improve recovery accuracy . However , all of them can not scale up to the large-batch data leakage setting . The contributions of this paper are summarized in the following . 1 ) We develop an advanced data leakage attack that we term CAFE to overcome the limitation of current data leakage attacks on FL . CAFE is able to recover large-scale data both in VFL and HFL . 2 ) Our large-batch data recovery is based on the novel use of data index alignment and internal representation alignment in FL , which can significantly improve the recovery performance . 3 ) The effectiveness and practical risk induced from our data leakage algorithm is justified in the dynamic FL training setting when all parameters in the model are updated every iteration . 2 PRELIMINARY . FL can be categorized into horizontal and vertical FL settings ( Kairouz et al. , 2019 ) . In this section , we provide necessary background of FL in this section . Horizontal FL . In HFL , data are distributed among local workers holding the same feature space . Suppose that there are M workers participating in the FL process and the number of samples in the dataset X is N . The dataset is denoted as X : = [ X1 , . . . , Xm , . . . , XM ] T , where Xm ∈ RNm×p is the local data partitioned to worker m , and p is the dimension of data feature space , Nm is the number of data samples partitioned to local worker m , and ∑M m=1Nm = N . Since all local data share the same feature space , each local worker computes the gradients independently and uploads them to the server . The server receives all gradients from each local worker and uses gradient aggregation methods such as FedAvg ( Konečnỳ et al. , 2016 ) . Let the parameters of the model as θ and the loss function as L. Then the objective function of HFL can be defined as : min θ 1 N M∑ m=1 L ( θ ; Xm ) with L ( θ ; Xm ) : = ∑ n∈Nm L ( θ ; xn ) ( 1 ) Vertical FL . Different from HFL , in VFL , each local worker m is associated with a unique set of features . Each data sample xn in dataset X can be written as xn = [ x T n1 , . . . , x T nm , . . . , x T nM ] T ( 2 ) where xnm ∈ Rpm is the data partitioned to worker m and pm is the data dimension in local worker m. The label space { yn } Nn=1 can be regarded as a special feature and is partitioned to the server or a certain local worker . Similar to ( 1 ) , the objective function of VFL can be written as : min θ 1 N N∑ n=1 L ( θ ; xn1 ; . . . ; xnM ) ( 3 ) 3 CATASTROPHIC DATA LEAKAGE FROM BATCH GRADIENTS . To realize large-scale data recovery from aggregated gradients , we propose our algorithm named as CAFE : Catastrophic dAta leakage in Federated lEarning . While CAFE can be applied to any type of data , without loss of generality , we use image datasets throughout the paper . 3.1 WHY LARGE-BATCH DATA LEAKAGE ATTACK IS DIFFICULT ? . We start by providing some intuition on the difficulty of performing large-batch data leakage from aggregated gradients based on the formulation of DLG ( Zhu et al. , 2019 ) . Assume thatN images are selected as the input for a certain learning iteration . We define the data batch as X = { xn , yn|xn ∈ RH×W×C , n = 1 , 2 , . . . , N } , where H , W , C represents the height , the width and the channel number of each image . Likewise , the batched ‘ recovered data ’ is denoted by X̂ = { x̂n , ŷn|x̂n ∈ RH×W×C , n = 1 , 2 , . . . , N } , which have the same dimension as X . Then the objective function is X̂ ∗ = argmin X̂ ∥∥∥∥∥ 1N N∑ n=1 ∇θL ( θ , xn , yn ) − 1 N N∑ n=1 ∇θL ( θ , x̂n , ŷn ) ∥∥∥∥∥ 2 ( 4 ) Note that in ( 4 ) , the dimensions of the aggregated gradients is fixed . However , as the N increases , the dimension of X̂ and X rise . When N is sufficiently large , it will be more challenging to find the “ right ” solution X̂ of ( 4 ) corresponding to the ground-truth dataset X . On the other hand , CAFE addresses this large-batch issue by data index alignment for batch data recovery , which can effectively exclude undesired solutions . We discuss a specific example in Appendix A . As a motivating example , Figure 1 compares our proposed attack with DLG on a batch of 40 images . The recovery quality of DLG is far from satisfactory , while CAFE can successfully recover all images in the batch . It is worth noting that because DLG is not effective on large-batch recovery , it is suggested in Zhu et al . ( 2019 ) that increasing batch size could be a promising defense . However , the successful recovery of CAFE shows that such defense premise gives a false sense of security in data leakage and the current FL is at risk , as large-batch data recovery can be accomplished . 3.2 CAFE IN VFL . In VFL , the server sends public key to local workers and decides the data index in each iteration of training and evaluation ( Yang et al. , 2019 ; Cheng et al. , 2019 ) . During the training process , local workers exchange their intermediate results with others to compute gradients and upload them . Therefore , the server has the access to both the model parameters and their gradients . Notably , CAFE can be readily applied to existing VFL protocols where the batch data index is assigned . Figure 2 gives an overview of CAFE in the VFL setting . The blue part represents a normal VFL paradigm and the red part represents the CAFE attack . Since data are vertically partitioned among different workers , data index alignment turns out to be an inevitable step in the vertical training process , which provides the server ( the attacker ) an opportunity to control the selected batch data index . Suppose that there are M workers participating FL and the batch size is N . The aggregated gradients can be denoted by ∇θL ( θ , X t ) = 1 Nb Nb∑ n=1 ∇θL ( θ , X tn ) with X tn = [ xtn1 , xtn2 , . . . , xtnM ] . ( 5 ) Algorithm 1 CAFE in VFL ( regular VFL protocol and CAFE protocol ) 1 : Initialize model parameters θ and generate fake data X̂ 2 : for t = 1 , 2 , . . . , T do 3 : Server broadcasts the global model to all local workers ( a total of M workers ) 4 : for m = 1 , 2 , . . . , M do 5 : Worker m takes real batch data 6 : Worker m computes the intermediate results and exchanges them with other workers 7 : Worker m uses the exchanged intermediate results to compute local aggregated gradients 8 : Worker m uploads real local aggregated gradients to the server . 9 : end for 10 : Server computes real global aggregated gradients∇θL ( θ , X t ) 11 : Server computes the fake global aggregated gradients∇θL ( θ , X̂ t ) 12 : Server computes CAFE loss : D ( X t ; X̂ t ) and∇X̂ tD ( X t ; X̂ t ) 13 : Server updates the batch data X̂ t with∇X̂ tD ( X t ; X̂ t ) 14 : Server updates the model parameters θ with∇θL ( θ , X t ) 15 : end for A benign server will perform legitimate computations designed by FL protocol . However , as shown in Figure 2 , a curious server can provide the same legitimate computation as a benign server while simultaneously perform data recovery in a stealthy manner . The server symmetrically generates fake images corresponding to the real ones . Once a batch of original data is selected , the server takes the corresponding fake batch and obtains the fake gradients as ∇θL ( θ , X̂ t ) = 1 Nb Nb∑ n=1 ∇θL ( θ , X̂ tn ) with X̂ tn = [ x̂tn1 , x̂tn2 , . . . , x̂tnM ] . ( 6 ) Algorithm 1 gives a pseudo code that implements our CAFE attack in VFL cases . The key part in our algorithm is aligning the real data batch indices with the fake ones . We define the squared ` 2-norm of the difference between the real and fake aggregated gradients in ( 7 ) . Since the server has the access to the model parameters , the attacker is able to compute the gradient of fake data from the loss in ( 7 ) and optimize the fake data for the purpose of recovering real data . D ( X t ; X̂ t ) = ∥∥∇θL ( θ , X t ) −∇θL ( θ , X̂ t ) ∥∥2 . ( 7 )

Summary: This work introduces CAFE, a novel training algorithm to leak training data in a federated learning setup. Extending from "deep leakage from gradient" fake images are optimised with respect to the difference observed from the client gradients (i.e. with the real images) and the one observed with the current version of the fake image. However, DLG does not work when the mini-batch size increases due to a messy gradient representation. In this work, the authors propose to keep track of the batch index. Indeed, it may happen that the server decides of the batch index corresponding to the training data that will be used by the client during the local training. Within such conditions, a malicious server can easily store fake images corresponding to specific indices and therefore optimise correctly each fake images w.r.t the corresponding real image.