Revealing and Protecting Labels in Distributed Training

1 Introduction . Distributed learning paradigms such as federated learning [ 2 ] offer a mechanism for mobile devices to collaboratively train deep learning models via transmission of model updates over the network , while ensuring sensitive training data remains resident , locally , on device . Recent works have demonstrated that information about training data can be revealed from a gradient , or a model update in general . For instance , in the image classification domain , the leaked information ranges from some property of the data like membership inference [ 3 , 4 ] , to pixel-wise accurate image reconstruction [ 1 ] . To reconstruct the samples given to a model , Zhu et al . [ 1 ] propose Deep Leakage from Gradients ( DLG ) , or Gradients Matching ( GM ) , a method that iteratively updates a dummy training sample to minimize the distance of its gradient to a target gradient sent from a client . This has been used to successfully reconstruct large images in a mini-batch [ 5 , 6 , 7 ] . The application of GM has also been demonstrated in other domains , such as language modeling [ 1 ] and automatic speech recognition [ 8 ] . As opposed to model inputs being continuous , training labels being discrete prevents common techniques like gradient descent from being trivially employed for reconstructing them . Instead , training labels can be reconstructed jointly with model inputs by optimizing for continuous `` pseudo '' labels ( a probability distribution on labels ) [ 1 ] . However , this method is not guaranteed to succeed , ∗Work performed while at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for instance , when the weight update is computed from a single sample . Zhao et al . [ 9 ] first observe that a label of a single training image can be analytically derived from a gradient with 100 % success rate . Their method , however , only applies to gradients computed from a single sample , and does not apply to gradients computed from a mini-batch , or aggregated from multiple training steps . Yin et al . [ 7 ] and Wainakh et al . [ 10 ] extend this method to demonstrate that labels of samples in a mini-batch can be recovered . The method leads to high success rate on revealing labels of samples in the batch , but its applicability is restricted to classification models using a non-negative activation function before the fully-connected layer . While many works have focused on sample reconstruction , the knowledge of training labels has been shown to be necessary for high-quality reconstruction [ 7 ] . In sequence models ( e.g. , speech and language domains ) , revealing the labels itself can result in a significant leak of privacy . Speech transcript leakage imposes a significant threat to users ’ privacy and can lead to the exposure of speaker identities [ 8 ] . Text-form data may include secrets such as passwords , social security numbers , or sensitive words that reveal the context of the conversation . However , in sequence models that output natural language , the number of labels ( e.g. , characters , wordpieces , or words ) can be substantially larger than in image classification models . For such large search spaces , the joint optimization approach adopted in [ 1 ] becomes ineffective . In this work , we make the following contributions . 1 . We propose a method to reveal the entire set of labels used in computing a weight update . We refer to the method as Revealing Labels from Gradients ( RLG ) . The method is applicable with any model that uses a softmax activation with cross-entropy loss , thus is applicable to a wide variety of deep learning architectures . We empirically demonstrate that RLG can be applied to a parameter update generated from a mini-batch ( 1-step , N -sample ) , or after several steps ( K-step , 1-sample each ) , or even the most general case of K-step update of N -sample at each step . 2 . We discuss the effects of gradient quantization and sparsification techniques , namely SignSGD and GradDrop , in the prevention of RLG . These methods have been used to reduce the communication cost in distributed training . We discover that they are also effective in mitigating RLG . 3 . We further demonstrate that in sequence models , the knowledge about the set of labels can be used to fully reconstruct a sequence of labels in order , while GM has limited success . In addition to the parameter update , this requires the current state of the model . We demonstrate our method on 1 ) an image classification task and 2 ) an automatic speech recognition ( ASR ) task . For image classification , we present our results on ResNet [ 11 ] , which has often been the target of choice for attacks in previous works [ 5 , 7 ] , and EfficientNet [ 12 ] , the state-of-the-art architecture for image classification . While the approach proposed in [ 7 ] works for the ResNet , only our proposed method works for EfficientNet . For ASR , we demonstrate our method with the attention-based encoder-decoder architecture . We show that our method used in conjunction with GM can perform full transcript reconstruction , while none of prior works succeed . Our code is published at https : //github.com/googleinterns/learning-bag-of-words . 2 Revealing Training Labels from Gradients . 2.1 Background & Notation . We consider a scenario of model training where an adversary gets access to some model updates ( e.g. , gradients ) . This can be possible in distributed learning settings by honest-but-curious participants , or a honest-but-curious server orchestrating the training , or an adversary that compromises some client-to-server communication channel . Note that we do not cover distributed training paradigms that share an intermediate representation , since they usually involve disclosing labels to the training agent [ 13 ] and require different types of defense techniques [ 14 ] . For simplicity , we consider vanilla gradient sharing approach , and assume that no label obfuscation is performed on the client side . We consider model architectures having a softmax activation with cross-entropy loss for classification , which is standard in deep learning based architectures ( e.g . [ 11 , 15 ] ) . The goal is to reveal labels y for labeled samples of the form ( X , y ) . For each training sample , y could be a single label ( e.g. , in classification models ) , or a sequence of labels ( e.g. , in sequence models ) . Let W ∈ Rd×C and b ∈ RC be the weight and bias of the layer prior to the softmax function ( referred to as the projection layer ) , where d is the input dimension ( same as output dimension of the previous layer ) , and C is the number of classes . The projection layer maps the latent representation h ∈ R1×d extracted from the model input to z = hW + b ∈ R1×C , which represents the unnormalized log probabilities ( logits ) on the classes . We assume that the adversary can access ∆W , i.e . the weight update of the projection layer , computed using back-propagation . This could be computed from a single sample , a batch of several samples , or several batches . Note that our setting crucially does not require the adversary to have access to any other part of the update except ∆W ; however , accessing to the full model update can be useful for sequence reconstruction , as discussed in Section 4 . Additionally , the adversary is required to know the id to label mapping , so that columns in ∆W can be associated with actual labels . For the target of the method , we aim to reveal a set of labels from samples used to compute ∆W , i.e. , S = { y | y ∈ Y } . The method does not return the number of samples for each class . In this section , we show that ∆W can be represented as the product of two matrices ( Remark 1 ) , one of which can be related to Y ( Remark 3 ) . Based on this property , we propose a method using singular value decomposition ( SVD ) to reveal the set of labels S given access to ∆W . 2.2 Gradient of the Projection Layer . Before explaining our method , we want to derive some properties of ∆W . We first consider the case when the gradient is computed from a single sample with a single label ( X , y ) . The probability distribution ŷ over the classes is derived by applying the softmax function on z : ŷi = exp ( zi ) ∑ j exp ( zj ) . Training a classification model involves minimizing the cross-entropy loss L = − ∑ i [ y = i ] log ŷi = − log exp zyc∑ j∈C exp zj ( 1 ) Since z = hW + b , we have ∂z∂W = h > . The weight update for the projection layer can be represented as ∆W = ∂L ∂W = ∂L ∂z ∂z ∂W = h > g , where g = ∂L ∂z ( 2 ) ( Note that h and g are row vectors ) . We can generalize this property to the following settings , which involve more than one training samples and labels . Weight update by a mini-batch / sequence of labels The weight update of a N -sample minibatch or a sequence of length N is averaged from weight updates computed from each sample in the batch or each label in the sequence : ∆W = 1N ∑N i=1 h > i gi = H > G , where H = 1N [ h1 , . . . , hN ] , and G = [ g1 , . . . , gN ] Weight update by several training steps The weight update after K training steps is the sum of weight update at each step2 : ∆W = ∑K i=1 αi∆W ( i ) = ∑K i=1 αiH > ( i ) G ( i ) = H > G , where ∆W ( i ) and αi is the softmax gradient and learning rate at the time step i , respectively , H = [ α1H ( 1 ) , . . . , αKH ( K ) ] , and G = [ G ( 1 ) , . . . , G ( K ) ] . More generally , we have the following . Remark 1 . ∆W can be represented as the product of H > ∈ Rd×S and G ∈ RS×C , where S is the number of labels used to compute the weight update . We consider S unknown to the adversary . If the weight update is computed from a batch , S is the batch size . If the weight update is aggregated from several training steps , S is the total number of samples at these steps . In fact , S can be inferred from the rank of ∆W as follows . 2We consider non-adaptive optimization methods , e.g . FedAvg [ 16 ] , which are common in FL training . Remark 2 . Assume that S < min { d , C } and H , G are full-rank matrices . We have rank ( H ) = rank ( G ) = S , and S can be inferred from S = rank ( ∆W ) . Now we want to derive a relation between G and labels Y . Differentiating L w.r.t . z yields gji = ∇z j i = ∂L ∂zji = { −1 + softmax ( zji , zi ) if j = yi softmax ( zji , zi ) otherwise ( 3 ) Since the softmax function always returns a value in ( 0 , 1 ) , each row in G has a unique negative coordinate corresponding to the ground-truth label . Formally , we have Remark 3 . Let Neg ( u ) define the indices of negative coordinates in a vector u . For every row gi in G , Neg ( gi ) = { yi } . More intuitively , in order to minimize the loss , the probability of the ground-truth label should be pushed up to 1 ( by a negative gradient ) , and probabilities of other labels should be pushed down to 0 ( by a positive gradient ) . Hence , signs in the gradient matrix G can be directly linked to labels Y . Though G is not directly observable from ∆W , this property of G is critical for revealing the set of labels from ∆W , as explained in the next section .

Summary: The paper proposes a new approach for revealing the training labels from the gradients of the last layer. The approach relies on the fact that if a label is used for computing the weight update, its corresponding column in the transposed right singular matrix can be linearly separated from other columns. The paper also experimented with how current gradient compression techniques can affect the attack accuracies, and explored how to reveal the sequence of labels, by combining the proposed approach with gradient matching.