Revealing and Protecting Labels in Distributed Training

1 Introduction . Distributed learning paradigms such as federated learning [ 2 ] offer a mechanism for mobile devices to collaboratively train deep learning models via transmission of model updates over the network , while ensuring sensitive training data remains resident , locally , on device . Recent works have demonstrated that information about training data can be revealed from a gradient , or a model update in general . For instance , in the image classification domain , the leaked information ranges from some property of the data like membership inference [ 3 , 4 ] , to pixel-wise accurate image reconstruction [ 1 ] . To reconstruct the samples given to a model , Zhu et al . [ 1 ] propose Deep Leakage from Gradients ( DLG ) , or Gradients Matching ( GM ) , a method that iteratively updates a dummy training sample to minimize the distance of its gradient to a target gradient sent from a client . This has been used to successfully reconstruct large images in a mini-batch [ 5 , 6 , 7 ] . The application of GM has also been demonstrated in other domains , such as language modeling [ 1 ] and automatic speech recognition [ 8 ] . As opposed to model inputs being continuous , training labels being discrete prevents common techniques like gradient descent from being trivially employed for reconstructing them . Instead , training labels can be reconstructed jointly with model inputs by optimizing for continuous `` pseudo '' labels ( a probability distribution on labels ) [ 1 ] . However , this method is not guaranteed to succeed , ∗Work performed while at Google . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for instance , when the weight update is computed from a single sample . Zhao et al . [ 9 ] first observe that a label of a single training image can be analytically derived from a gradient with 100 % success rate . Their method , however , only applies to gradients computed from a single sample , and does not apply to gradients computed from a mini-batch , or aggregated from multiple training steps . Yin et al . [ 7 ] and Wainakh et al . [ 10 ] extend this method to demonstrate that labels of samples in a mini-batch can be recovered . The method leads to high success rate on revealing labels of samples in the batch , but its applicability is restricted to classification models using a non-negative activation function before the fully-connected layer . While many works have focused on sample reconstruction , the knowledge of training labels has been shown to be necessary for high-quality reconstruction [ 7 ] . In sequence models ( e.g. , speech and language domains ) , revealing the labels itself can result in a significant leak of privacy . Speech transcript leakage imposes a significant threat to users ’ privacy and can lead to the exposure of speaker identities [ 8 ] . Text-form data may include secrets such as passwords , social security numbers , or sensitive words that reveal the context of the conversation . However , in sequence models that output natural language , the number of labels ( e.g. , characters , wordpieces , or words ) can be substantially larger than in image classification models . For such large search spaces , the joint optimization approach adopted in [ 1 ] becomes ineffective . In this work , we make the following contributions . 1 . We propose a method to reveal the entire set of labels used in computing a weight update . We refer to the method as Revealing Labels from Gradients ( RLG ) . The method is applicable with any model that uses a softmax activation with cross-entropy loss , thus is applicable to a wide variety of deep learning architectures . We empirically demonstrate that RLG can be applied to a parameter update generated from a mini-batch ( 1-step , N -sample ) , or after several steps ( K-step , 1-sample each ) , or even the most general case of K-step update of N -sample at each step . 2 . We discuss the effects of gradient quantization and sparsification techniques , namely SignSGD and GradDrop , in the prevention of RLG . These methods have been used to reduce the communication cost in distributed training . We discover that they are also effective in mitigating RLG . 3 . We further demonstrate that in sequence models , the knowledge about the set of labels can be used to fully reconstruct a sequence of labels in order , while GM has limited success . In addition to the parameter update , this requires the current state of the model . We demonstrate our method on 1 ) an image classification task and 2 ) an automatic speech recognition ( ASR ) task . For image classification , we present our results on ResNet [ 11 ] , which has often been the target of choice for attacks in previous works [ 5 , 7 ] , and EfficientNet [ 12 ] , the state-of-the-art architecture for image classification . While the approach proposed in [ 7 ] works for the ResNet , only our proposed method works for EfficientNet . For ASR , we demonstrate our method with the attention-based encoder-decoder architecture . We show that our method used in conjunction with GM can perform full transcript reconstruction , while none of prior works succeed . Our code is published at https : //github.com/googleinterns/learning-bag-of-words . 2 Revealing Training Labels from Gradients . 2.1 Background & Notation . We consider a scenario of model training where an adversary gets access to some model updates ( e.g. , gradients ) . This can be possible in distributed learning settings by honest-but-curious participants , or a honest-but-curious server orchestrating the training , or an adversary that compromises some client-to-server communication channel . Note that we do not cover distributed training paradigms that share an intermediate representation , since they usually involve disclosing labels to the training agent [ 13 ] and require different types of defense techniques [ 14 ] . For simplicity , we consider vanilla gradient sharing approach , and assume that no label obfuscation is performed on the client side . We consider model architectures having a softmax activation with cross-entropy loss for classification , which is standard in deep learning based architectures ( e.g . [ 11 , 15 ] ) . The goal is to reveal labels y for labeled samples of the form ( X , y ) . For each training sample , y could be a single label ( e.g. , in classification models ) , or a sequence of labels ( e.g. , in sequence models ) . Let W ∈ Rd×C and b ∈ RC be the weight and bias of the layer prior to the softmax function ( referred to as the projection layer ) , where d is the input dimension ( same as output dimension of the previous layer ) , and C is the number of classes . The projection layer maps the latent representation h ∈ R1×d extracted from the model input to z = hW + b ∈ R1×C , which represents the unnormalized log probabilities ( logits ) on the classes . We assume that the adversary can access ∆W , i.e . the weight update of the projection layer , computed using back-propagation . This could be computed from a single sample , a batch of several samples , or several batches . Note that our setting crucially does not require the adversary to have access to any other part of the update except ∆W ; however , accessing to the full model update can be useful for sequence reconstruction , as discussed in Section 4 . Additionally , the adversary is required to know the id to label mapping , so that columns in ∆W can be associated with actual labels . For the target of the method , we aim to reveal a set of labels from samples used to compute ∆W , i.e. , S = { y | y ∈ Y } . The method does not return the number of samples for each class . In this section , we show that ∆W can be represented as the product of two matrices ( Remark 1 ) , one of which can be related to Y ( Remark 3 ) . Based on this property , we propose a method using singular value decomposition ( SVD ) to reveal the set of labels S given access to ∆W . 2.2 Gradient of the Projection Layer . Before explaining our method , we want to derive some properties of ∆W . We first consider the case when the gradient is computed from a single sample with a single label ( X , y ) . The probability distribution ŷ over the classes is derived by applying the softmax function on z : ŷi = exp ( zi ) ∑ j exp ( zj ) . Training a classification model involves minimizing the cross-entropy loss L = − ∑ i [ y = i ] log ŷi = − log exp zyc∑ j∈C exp zj ( 1 ) Since z = hW + b , we have ∂z∂W = h > . The weight update for the projection layer can be represented as ∆W = ∂L ∂W = ∂L ∂z ∂z ∂W = h > g , where g = ∂L ∂z ( 2 ) ( Note that h and g are row vectors ) . We can generalize this property to the following settings , which involve more than one training samples and labels . Weight update by a mini-batch / sequence of labels The weight update of a N -sample minibatch or a sequence of length N is averaged from weight updates computed from each sample in the batch or each label in the sequence : ∆W = 1N ∑N i=1 h > i gi = H > G , where H = 1N [ h1 , . . . , hN ] , and G = [ g1 , . . . , gN ] Weight update by several training steps The weight update after K training steps is the sum of weight update at each step2 : ∆W = ∑K i=1 αi∆W ( i ) = ∑K i=1 αiH > ( i ) G ( i ) = H > G , where ∆W ( i ) and αi is the softmax gradient and learning rate at the time step i , respectively , H = [ α1H ( 1 ) , . . . , αKH ( K ) ] , and G = [ G ( 1 ) , . . . , G ( K ) ] . More generally , we have the following . Remark 1 . ∆W can be represented as the product of H > ∈ Rd×S and G ∈ RS×C , where S is the number of labels used to compute the weight update . We consider S unknown to the adversary . If the weight update is computed from a batch , S is the batch size . If the weight update is aggregated from several training steps , S is the total number of samples at these steps . In fact , S can be inferred from the rank of ∆W as follows . 2We consider non-adaptive optimization methods , e.g . FedAvg [ 16 ] , which are common in FL training . Remark 2 . Assume that S < min { d , C } and H , G are full-rank matrices . We have rank ( H ) = rank ( G ) = S , and S can be inferred from S = rank ( ∆W ) . Now we want to derive a relation between G and labels Y . Differentiating L w.r.t . z yields gji = ∇z j i = ∂L ∂zji = { −1 + softmax ( zji , zi ) if j = yi softmax ( zji , zi ) otherwise ( 3 ) Since the softmax function always returns a value in ( 0 , 1 ) , each row in G has a unique negative coordinate corresponding to the ground-truth label . Formally , we have Remark 3 . Let Neg ( u ) define the indices of negative coordinates in a vector u . For every row gi in G , Neg ( gi ) = { yi } . More intuitively , in order to minimize the loss , the probability of the ground-truth label should be pushed up to 1 ( by a negative gradient ) , and probabilities of other labels should be pushed down to 0 ( by a positive gradient ) . Hence , signs in the gradient matrix G can be directly linked to labels Y . Though G is not directly observable from ∆W , this property of G is critical for revealing the set of labels from ∆W , as explained in the next section .

Summary: The work shows that a set of labels used in a small batch of the training procedure is identifiable given the gradient of the last layer of the NN model with CE loss and soft-max activation. Also, they propose to modify gradients for defense. There are some theoretical and empirical gaps between this work and real-world settings.