CAFE: Catastrophic Data Leakage in Vertical Federated Learning

1 Introduction . Federated learning ( FL ) [ 8 , 24 ] is an emerging machine learning framework where a central server and multiple workers collaboratively train a machine learning model . Some existing FL methods consider the setting where each worker has data of a different set of subjects but sharing common features . This setting is also referred to data partitioned or horizontal FL ( HFL ) . Unlike the HFL setting , in many learning scenarios , multiple workers handle data about the same set of subjects , but each has a different set of features . This case is common in finance and healthcare applications [ 6 ] . In these examples , data owners ( e.g. , financial institutions and hospitals ) have different records of those users in their joint user base , and so , by combining their features through FL , they can establish a more accurate model . We refer to this setting as feature-partitioned or vertical FL ( VFL ) . Compared with existing distributed learning paradigms , FL raises new challenges including data heterogeneity and privacy [ 20 ] . To protect data privacy , only model parameters and the change of parameters ( e.g. , gradients ) are exchanged between server and workers [ 19 , 15 ] . Recent works have studied how a malicious worker can embed backdoors or replace the global model in FL [ 2 , 3 , 27 ] . Furthermore , as exchanging gradients is often viewed as privacy-preserving protocols , little attention has been paid to information leakage from public shared gradients and batch identities . In the context of data security and AI ethics , the possibility of inferring private user data from the gradients in FL has received growing interests [ 10 , 14 , 21 ] , known as the data leakage problems . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous works have made exploratory efforts on data recovery through gradients . See Section 2 and Table 1 for details . However , existing approaches often have the limitation of scaling up large-batch data recovery and are lacking in theoretical justification on the capability of data recovery , which may give a false sense of security that increasing the data batch size during training can prevent data leakage [ 30 ] . Some recent works provide sufficient conditions for guaranteed data recovery , but the assumptions are overly restrictive and can be sometimes impractical , such as requiring the number of classes to be much larger than the number of recovered data samples [ 29 ] . To enhance scalability in data recovery and gain fundamental understanding on data leakage in VFL , in this paper we propose an advanced data leakage attack with theoretical analysis on the data recovery performance , which we call catastrophic data leakage in vertical federated learning ( CAFE ) . As an illustration , Figure 1 demonstrates the effectiveness of CAFE for large-batch data recovery compared to existing methods . The contributions of this paper are summarized as follows . C1 ) We develop a new data leakage attack named CAFE to overcome the limitation of current data leakage attacks on VFL . Leveraging the novel use of data index and internal representation alignments in VFL , CAFE is able to recover large-scale data in general VFL protocols . C2 ) We provide theoretical guarantees on the recovery performance of CAFE , which permeates three steps of CAFE : ( I ) recovering gradients of loss with respect to the outputs of the first fully connected ( FC ) layer ; ( II ) recovering inputs to the first FC layer ; ( III ) recovering the original data . C3 ) To mitigate the data leakage attack by CAFE , we develop a defense strategy which leverages the fake gradients and preserves the model training performance . C4 ) We conduct extensive experiments on both static and dynamic VFL training settings to validate the superior data recovery performance of CAFE over state-of-the-art methods . 2 Related Work . Recovering private training data from gradients has gained growing interests in FL . Recently , a popular method termed deep leakage from gradients ( DLG ) [ 32 ] has been developed to infer training data in an efficient way without using any generative models or prior information . However , DLG lacks generalizability on model architecture and weight distribution initialization [ 25 ] . In [ 30 ] , an analytical approach has been developed to extract accurate labels from the gradients . In [ 11 ] , another analytical approach has been developed to derive the inputs before a fully connected ( FC ) layer . However , in [ 11 ] , their method only works on a single sample input and fails to extend on a batch of data . In [ 22 ] , a new approach has been developed by recovering the batch inputs before the FC layer through solving linear equations . However , strong assumptions have been made for solving the equations and can not guarantee data recovery in more general cases . In [ 9 ] , it is claimed that a convolutional layer can always be converted to a FC layer . However , the gradients of the original convolutional layer are still different from the gradients of the converted FC layer , which impedes data recovery . Besides the new loss function proposed in [ 11 ] , several previous works design new loss functions or regularizers based on DLG and try to make their algorithms work on more general models and weight distribution initialization . In [ 25 ] , a new Gaussian kernel based gradient difference is used as the distance measure . In [ 31 ] , a recursive method attack procedure has been developed to recover data from gradients . However , in both [ 25 ] and [ 31 ] , the quality of recovery on batch data is degraded . A recent work [ 29 ] proposes an algorithm named GradInversion to reconstruct images from noise based on given gradients . However , their theory and algorithm are mostly built on strong assumptions and empirical observations . Although they successfully reconstruct a batch of training data , the reported batch size is still no larger than 48 . 3 CAFE : Catastrophic Data Leakage in Vertical Federated Learning . In this section , we will introduce some necessary background of VFL and present our novel attack method . We consider the attack scenario where a honest-but-curious server follows the regular VFL protocols but intends to recover clients ’ private data based on the aggregated gradients . Our method is termed CAFE : Catastrophic data leakage in vertical federated learning . While CAFE can be applied to any type of data , without loss of generality , we use image datasets throughout the paper . 3.1 Preliminaries . VFL setting . FL can be categorized into horizontal and vertical FL settings [ 16 ] . In this paragraph , we provide necessary background of VFL . Consider a set of M clients : M = { 1 , 2 , . . . , M } . A dataset of N samples D = { ( xn , yn ) } Nn=1 are maintained by the M local clients , where n is the data index . Each client m in M is associated with a unique features set . A certain data point xn in D can be denoted by xn = [ x > n,1 , x > n,2 , . . . , x > n , M ] > where xn , m is the m-th partition of the n-th sample vector . The label set { yn } Nn=1 can be viewed as a special feature and is kept at the server or a certain local worker . Throughout this paper , we mainly study the VFL setting . CAFE can also be applied to HFL if the data indices of each randomly selected batch are known to workers during training . Use case of VFL . VFL is suitable for cases where multiple data owners share the same data identity but their data differ in feature space . Use cases of VFL appear in finance , e-commerce , and health . For example , in medical industry , test results of the same patient from different medical institutions are required to diagnose whether the patient has a certain disease or not , but institutions tend not to share raw data . Figure 2 gives an example of VFL in medical industry . Batch indices in each iteration . For a given batch size K , we define a set of vectors with binary entries as S = { s1 , s2 , . . . , si , . . . } with |S| = N K . For each vector si 2 RN in S , its n-th element si [ n ] can be either 0 or 1 . There are in total K enires of ‘ 1 ’ in si . In each iteration t , the server randomly selects one element from set S denoted by st , where st [ n ] is the nth element in st . The selected batch samples in the t-th iteration are denoted by D ( st ) = { ( xn , yn ) |st [ n ] = 1 } . Loss function and gradients . We assume that the model is a neural network parameterized by ⇥ , where the first FC layer is parameterized by ⇥1 2 Rd1⇥d2 and its bias is b1 2 Rd2 . The loss function on the batch data D ( st ) and on the entire training data D is , respectively , denoted by L ( ⇥ , D ( st ) ) : = 1 K NX n=1 st [ n ] L ( ⇥ , xn , yn ) and L ( ⇥ , D ) : = 1 N NX n=1 L ( ⇥ , xn , yn ) . ( 1 ) The gradients of losses w.r.t . ⇥ is denoted as r⇥L ( ⇥ , D ( st ) ) : = @ L ( ⇥ , D ( st ) ) @ ⇥ = 1 K NX n=1 st [ n ] @ L ( ⇥ , xn , yn ) @ ⇥ . ( 2 ) And similarly , we define r⇥L ( ⇥ , D ) . 3.2 Why large-batch data leakage attack is difficult ? . We motivate the design of our algorithm by providing some intuition on why performing large-batch data leakage from aggregated gradients is difficult [ 32 ] . Assume that K images are selected as the inputs for a certain learning iteration . We define the selected batch data as D0 = { ( xn , yn ) } . Likewise , the batched ‘ recovered data ’ is denoted by D̂0 = { ( x̂n , ŷn ) } . Then the objective function is D̂ 0 = argmin D̂0 1 K X ( xn , yn ) 2D r⇥L ( ⇥ , xn , yn ) 1 K X ( x̂n , ŷn ) 2D̂0 r⇥L ( ⇥ , x̂n , ŷn ) 2 . ( 3 ) Note that in ( 3 ) , the dimensions of the aggregated gradients is fixed . However , as K increases , the cardinality of D̂0 and D0 rise . When K is sufficiently large , it will be more challenging to find the “ right ” solution D̂0 of ( 3 ) corresponding to the ground-truth dataset D0 . On the other hand , CAFE addresses this issue of large-batch data recovery by data index alignment ( defined in next subsection ) , which can effectively exclude undesired solutions . We discuss a specific example in Appendix B .

Summary:  This paper proposes an information leakage attack (i.e., CAFE) that can recover data samples from aggregated gradients against a  FL setting. The proposed attack first recovers gradients of loss with respect to the outputs of the first fully connected (FC) layer. It then recovers the first FC layer's inputs. Finally, it recovers the original data samples by optimizing an objective that consists of the traditional gradient matching objective and two auxiliary regularizers:  a TV norm term and an internal representation regularization term. The paper provides both theoretical analysis and numerical verification about CAFE's performance. Experiments show that the proposed method outperforms some of the previous works.