Robbing the Fed:  Directly Obtaining Private Data in Federated Learning with Modified Models

1 INTRODUCTION . Federated learning ( Konečný et al. , 2015 ) , also known as collaborative learning ( Shokri & Shmatikov , 2015 ) , is a mechanism for training machine learning models in a distributed fashion on multiple user devices . In the simplest setting , a central server sends out model states to a group of users , who compute an update to the model based on their local data . These updates are then returned to the server , aggregated , and used to train the model . Over multiple rounds , this protocol can train a machine learning model , distributed over all users , without exchanging local data – only model updates are exchanged . Two central goals of federated learning are to improve training efficiency by decreasing communication overhead and to side-step issues of user-level privacy and data access rights that have become a focus of public attention in recent years ( Veale et al. , 2018 ) . Accordingly , many organizations , ranging from large tech companies ( McMahan & Ramage , 2017 ) to medical institutions with especially strict privacy laws , such as hospitals ( Jochems et al. , 2016 ) , have utilized federated learning to train machine learning models . However , in practice , data privacy is not guaranteed in general , but is dependent on a large number of interdependent settings and design choices specific to each federated learning system . In this work , we focus on the user perspective of privacy , and we study federated learning systems in which the central server is not able to directly view user data . The key privacy concern for users is whether model updates reveal too much about the data on which they were calculated . Although Kairouz et al . ( 2021 ) discuss that “ model updates are more focused on the learning task at hand than is the raw data ( i.e . they contain strictly no additional information about the user , and typically significantly less , compared to the raw data ) ” , scenarios can be constructed in which the model updates themselves can be inverted to recover their input user information ( Wang et al. , 2018 ; Melis et al. , 2018 ) . Simple knowledge of the shared model state and model update can be sufficient for such an attack ( Zhu et al. , 2019 ; Geiping et al. , 2020 ) . These inversion attacks are particularly fruitful if a user ’ s model update is based on a single data point or only a small batch . Accordingly , a strong defense against these attacks is aggregation . The user only reports model updates aggregated over a significant number of local data points , and data from multiple users can be combined with secure aggregation protocols ( Bonawitz et al. , 2017 ) before being passed to the server . This ability to aggregate user updates while maintaining their utility is thought to be the main source of security in federated learning . Averaging raw local data in similar amounts would make it unusable for training , but model updates can be effectively aggregated . Previous inversion attacks typically focus on a threat model in which the server ( server here is a stand-in for any party with root access to the server or its incoming and outgoing communication ) is interested in uncovering user information by examining updates , but without modifying the federated learning protocol , a behavior also referred to as honest-but-curious or semi-honest ( Goldreich , 2009 ) . In our case , where the party intending to recover user data is the server , this “ honest ” scenario appears contrived , as the server can modify its behavior to obtain private information . In this work , we are thus interested in explicitly malicious servers that may modify the model architecture and model parameters sent to the user . We focus on scenarios in which an agent obtains data without making suspicious changes to the client code or learning behavior . One scenario which enables this threat model involves recently introduced APIs that allow organizations to train their own models using established federated learning protocols ( Cason , 2020 ) . In this environment , a malicious API participant can change their model ’ s architecture and parameters but can not force unsuspecting edge devices to send user data directly . We introduce minimal changes to model architectures that enable servers to breach user privacy , even in the face of large aggregations that have been previously deemed secure . These changes induce a structured pattern in the model update , where parts of the update contain information only about a fixed subset of data points . The constituent data points can then be recovered exactly , while evading existing aggregation defenses . 2 LIMITATIONS OF EXISTING ATTACK STRATEGIES . A range of possible attacks against privacy in federated learning have been proposed in recent literature . In the simplest case of analytic attacks , Phong et al . ( 2017b ) were among the first to discuss that the input to a learnable affine function can be directly computed from the gradient of its weights and bias , and additional analysis of this case can be found in Qian & Hansen ( 2020 ) ; Fan et al . ( 2020 ) , and in Section 3.2 . However , analytic recovery of this kind only succeeds for a single data point . For multiple data points , only the average of their inputs can be recovered , leading the attack to fail in most realistic scenarios . Recursive attacks as proposed in Zhu & Blaschko ( 2021 ) can extend analytic attacks to models with more than only linear layers - a construction also mentioned in Fan et al . ( 2020 ) . However , these attacks still recover only the average of inputs in the best case . Improvements in Pan et al . ( 2020 ) transform linear layers with ReLU activations into systems of linear equations that allow for a degree of recovery for batched inputs to these linear layers , although preceding convolutional layers still have to be deconvolved by recursion or numerical inversion techniques . Surprisingly , optimization-based attacks turn out to be highly effective in inverting model updates . Wang et al . ( 2018 ) propose the direct recovery of input information in a setting where the users ’ model update is the model parameter gradient averaged over local data . In a supervised learning setting , we define this update by g and the loss function over this model by L with model parameters θ and data points ( x , y ) ∈ [ 0 , 1 ] n×Rm . The server can then attempt recovery by solving the gradient matching problem of min x∈ [ 0,1 ] n ||∇θL ( x , y , θ ) − g||2 ( 1 ) and solve this optimization objective using first-order methods or any nonlinear equation solver . Subsequent work in Zhu et al . ( 2019 ) ; Zhao et al . ( 2020 ) and Wainakh et al . ( 2021 ) proposes solutions that also handle recovery of targets y and variants of this objective are solved for example in Geiping et al . ( 2020 ) with cosine similarity and improved optimization and in Jeon et al . ( 2021 ) with additional generative image priors . Reconstruction of input images can be further boosted by additional regularizers as in Yin et al . ( 2021 ) and Qian et al . ( 2021 ) . Most attacks in the literature focus on the described fedSGD setting ( Konečný et al. , 2015 ) in which the users return gradient information to the server , but numerical attacks can also be performed against local updates with multiple local steps Geiping et al . ( 2020 ) , for example against fedAVG ( McMahan et al. , 2017 ) . In this work , we will discuss both update schemes , noting that gradient aggregation in “ time ” , with multiple local update steps , is not fundamentally more secure than ag- gregation over multiple data points . Previous attacks also focus significantly on learning scenarios where the user data is comprised of images . This is an advantage to the attacker , given that image data is highly structured , and a multitude of image priors are known and can be employed to improve reconstruction . In contrast , data types with weaker structure , such as tabular data , do not lend themselves to regularization based on strong priors , and we will show that our approach , on the other hand , does not rely on such tricks , and is therefore more data-agnostic . The central limitation of these attack mechanisms is the degradation of attack success when user data is aggregated over even moderately large batches of data ( either by the user themselves or by secure aggregation ) . State-of-the-art attacks such as Yin et al . ( 2021 ) recover only 28 % of the user data ( given a charitable measure of recovery ) on a batch size of 48 for a ResNet-50 ( He et al. , 2015 ) model on ImageNet ( ILSVRC2012 ( Russakovsky et al. , 2015 ) ) with unlikely label collisions . The rate of images that can be successfully recovered drops drastically with increased batch sizes . Even without label collisions , large networks such as a ResNet-32-10 ( Zagoruyko & Komodakis , 2016 ) leak only a few samples for a batch size of 128 in Geiping et al . ( 2020 ) . These attacks further reconstruct only approximations to the actual user data which can fail to recover parts of the user data or replace it with likely but unrelated information in the case of strong image priors . Further , although all of the previous works nominally operate under an honest-but-curious server model , they do often contain model adaptations on which reconstruction works especially well , such as large vision models with large gradient vectors , models with many features ( Wang et al. , 2018 ; Zhu & Blaschko , 2021 ) , special activation functions ( Zhu et al. , 2019 ; Zhu & Blaschko , 2021 ) , wide models ( Geiping et al. , 2020 ) , or models trained with representation learning ( Yin et al. , 2021 ; Chen et al. , 2020 ) . These may be seen as malicious models with architectural choices that breach user privacy . In the same vein , we ask , what is the worst-case ( but small ) modification that can be applied to a neural network to break privacy ? 3 MODEL MODIFICATIONS . In this section , we detail an example of a small model modification that has a major effect on user privacy , even allowing for the direct recovery of verbatim user data from model updates . 3.1 THREAT MODEL . We define two parties : the server S and the users U . The server could be a tech company , a third party app using a federated learning framework on a mobile platform , or an organization like a hospital . The server S defines a model architecture and distributes parameters θ for this architecture to the users , who compute local updates and return them to the server . The server can not deviate from standard federated learning protocol in ways beyond changes to model architecture ( within limits imposed by common ML frameworks ) and model parameters . We measure the strength of a malicious modification of the architecture using the number of additional parameters inserted into the model . While it is clear that models with more parameters can leak more information , we will see that clever attacks can have a disproportionate effect on attack success , compared to more benign increases in parameter count , such as when model width is increased .

Summary: This paper explores the privacy of federated learning when the server is malicious, specifically that it is allowed to control the architecture and and set certain weights in adversarial ways. By building on a weakness inherent to the RELU activation function, they show that w.h.p the malicious server can obtain exact reproductions of at least some inputs. They then empirically explore how well this works in practice, showing that the method introduced does pose a significant risk to user data privacy.