Revealing and Protecting Labels in Distributed Training
Authors: Trung Dang
Om Thakkar
Swaroop Ramaswamy
Rajiv Mathews
Sang Chin
Françoise Beaufays
Venue: NeurIPS 2021 Poster (2021)
Keywords: privacy
distributed training

Abstract: Distributed learning paradigms such as federated learning often involve transmission of model updates, or gradients, over a network, thereby avoiding transmission of private data. However, it is possible for sensitive information about the training data to be revealed from such gradients. Prior works have demonstrated that labels can be revealed analytically from the last layer of certain models (e.g., ResNet), or they can be reconstructed jointly with model inputs by using Gradients Matching [Zhu et al.] with additional knowledge about the current state of the model. In this work, we propose a method to discover the set of labels of training samples from only the gradient of the last layer and the id to label mapping. Our method is applicable to a wide variety of model architectures across multiple domains. We demonstrate the effectiveness of our method for model training in two domains - image classification, and automatic speech recognition. Furthermore, we show that existing reconstruction techniques improve their efficacy when used in conjunction with our method. Conversely, we demonstrate that gradient quantization and sparsification can significantly reduce the success of the attack.

Revealing and Protecting Labels in Distributed Training
Trung Dang"
Om Thakkar Google omthikrr@ooggee. com
Swaroop Ramaswamy
Boston University trungvddbu. edu
Google swaroopran@google com
Rajiy Mathews
Peter Chin Boston University spchin@bu. edu
Françoise Beaufays
Google mathews@google.cmm com
Google fsbEgoogle. com
Abstract
Distributed learning paradigms such as federated learning often involve transmis- sion of model updates, gradients, over network, thereby avoiding transmission private data. However, it is possible for sensitive information about the training data to be revealed from such gradients. Prior works have demonstrated that labels can be revealed analytically from the last layer of certain models (e.g.. ResNet), they can be reconstructed jointly with model inputs by using Gradients Match- ing [1] with additional knowledge about the current state of the model. In this work, propose method to discover the set of labels of training samples from only the gradient of the last layer and the to label mapping. Our method is applicable to wide variety of model architectures across multiple domains. We demonstrate the effectiveness of our method for model training in two domains image clas- sification, and automatic speech recognition. Furthermore, we show that existing reconstruction techniques improve their efficacy when used in conjunction with our method. Conversely, we demonstrate that gradient quantization and sparsification can significantly reduce the success of the attack.
Introduction
Distributed learning paradigms such as federated learning 2 offer mechanism for mobile devices to collaboratively train deep learning models via transmission of model updates over the network, while ensuring sensitive training data remains resident, locally, device. Recent works have demonstrated that information about training data can be revealed from a gradient, or a model update in general. For instance, in the image classification domain, the leaked information ranges from some property of the data like membership inference reconstruct the samples given a model, Zhu al. [ propose Deep Leakage from Gradients (DLG), or Gradients Matching (GM), method that iteratively updates a dummy training sample to minimize target gradient sent from client. This has been used successfully
3 4.. to pixel-wise accurate image reconstruction [I To
the distance of its gradient reconstruct large images in a mini-batch 5 6 77. The application of GM has also been demonstrated in other domains, such as language modeling and automatic speech recognition X
As opposed to model inputs being continuous, training labels being discrete prevents common techniques like gradient descent from being trivially employed for reconstructing them. Instead, training labels can be reconstructed jointly with model inputs by optimizing for continuous "pseudo" labels (a probability distribution labels) T.. However, this method guaranteed succeed,
Work performed while Google.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

for instance, when the weight update is computed from a single sample. Zhao et al. [9] first observe that a label of single training image can be analytically derived from gradient with 100% success rate. Their method, however, only applies to gradients computed from single sample, and does not apply to gradients computed from a mini-batch, or aggregated from multiple training steps. Yin et al. 7 and Wainakh et al. [0] extend this method to demonstrate that labels of samples in a mini-batch can be recovered. The method leads to high success rate on revealing labels of samples in the batch, but its applicabilityy is restricted to classification models using non-negative activation function before the fully-connected layer.
While many works have focused on sample reconstruction, the knowledge of training labels has been shown to be necessary for high-quality reconstruction (7]. In sequence models (e.g., speech and language domains), revealing the labels itself can result in significant leak of privacy, Speech transcript leakage imposes a significant threat to users' privacy and can lead to the exposure of speaker identities [8] Text-form data may include secrets such as passwords, social security numbers, or sensitive words that reveal the context of the conversation. However, in sequence models that output natural language, the number of labels (e.g., characters, wordpieces, or words) can be substantially larger than in image classification models. For such large search spaces, the joint optimization approach adopted in [I becomes ineffective.
In this work, we make the following contributions.
1. We propose a method to reveal the entire set of labels used in computing a weight update. We refer to the method as Revealing Labels from Gradients (RLG). The method is applicable with any model that uses a softmax activation with cross-entropy loss, thus is applicable to wide variety of deep learning architectures. We empirically demonstrate that RLG can be applied to a parameter update generated from a mini-batch (1-step, N N-sample), or after several steps (K-step. I-sample each), even the most general case of K K-step update of N N-sample at each step.
2. We discuss the effects of gradient quantization and sparsification techniques, namely Sign- SGD and GradDrop. in the prevention of RLG. These methods have been used to reduce the communication cost in distributed training. We discover that they are also effective in mitigating RLG.
3. We further demonstrate that in sequence models, the knowledge about the set of labels can be used to fully reconstruct sequence of labels in order, while GM has limited success. In addition to the parameter update, this requires the current state of the model.
We demonstrate method on 1) an image classification task and 2) an automatic speech recognition (ASR) task. For image classification, we present our results on ResNet [1], which has often been the target of choice for attacks in previous works 7,, and EfficientNet [12], the sfate-of-the-ar
architecture for image classification. While the approach proposed in [ works for the ResNet, only our proposed method works for EfficientNet. For ASR, we demonstrate our method with the attention-based encoder-decoder architecture. We show that our method used in conjunction with GM can perform full transcript reconstruction, while none of prior works succeed. Our code is published at httph:ts//// //githbb com/ http://ooooggggo////ooooooo learnninnn/errnnnnnn.ane bag ooogoof-o words
Revealing Training Labels from Gradients
2.1 Background & Notation
We consider scenario of model training where adversary gets access to some model updates (e.g.. gradients). This can be possible in distributed learning settings by bottst-but-curious participants, or honest-but-curiout server orchestrating the training, an adversary that compromises some tlient-to-server communication channel. Note that we do not cover distributed training paradigms that share an intermediate representation, since they usually involve disclosing labels to the training agent [33 and require different types of defense techniques [4]. For simplicity, we consider vanilla gradient sharing approach, and assume that label obfuscation is performed the client side.
We consider model architectures having a softmax activation with cross-entropy loss for classification, which is standard in deep learning based architectures (c.g. (5)..). The goal is to reveal labels

---

for labeled samples of the form (X.y). For each training sample, y could be single label (e.g., in classification models), or a sequence of labels (e.g., in sequence models).
Let W E RRdCC and b E RC be the weight and bias the layer prior to the softmax function (referred as the projection layer), where d is the input dimension (same output dimension of the previous layer), and C' is the number of classes. The projection layer maps the latent representation h E IR Ixd extracted from the model input probabilities (logits) on the classes.
hW + b E gIxC, which represents the unnormalized log
We assume that the adversary can access A W. i.e. the weight update of the projection layer, computed using back propagatatn This could be computed from a single sample, a batch of several samples, or several batches. Note that our setting crucially does not require the adversary to have access to any other part of the update except A AW; however, accessing to the full model update can he useful for sequence reconstruction, discussed in Section 4 Additionally, the adversary is required to know the id to label mapping, so that columns in A AW can be associated with actual labels. For the target of the method, we aim to reveal set of labels from samples used compute A i.e. S = {y y E }. The method does not return the number of samples for each class.
In this section, we show that AW can be represented one of which can be related to Y (Remark 33). Based on this property, we propose a method using singular value decomposition (SVD) to reveal the set of labels S given access to A AW.
the product of two matrices (Remark D
2.2 Gradient of the Projection Layer
Before explaining our method, we want to derive some properties of AW. We first consider the case when the gradient computed from a single sample with a single label (X, y). The probability exp(x.)
distribution ý over the classes is derived by applying the softmax function on 2: 0: Σ, exp(es))
Training a classification model involves minimizing the cross loss
L -= Σ y = i) log 0: = - -log EjEc exp 2y i
exp Zye
(1)
Since = hW + b, we have ow represented as
h.
The weight update for the projection layer can be
Oc
OC dz
OC
AW
h 9,
where g
aW
Oz OW
Oz
(Note that h and are row vectors). We can generalize this property to the following settings, which involve more than one training samples and labels.
Weight update by mini-batch sequence of labels The weight update of N N-sample mini- batch or a sequence of length N is averaged from weight updates computed from each sample in the batch or each label in the sequence: AW- = M. h HH G. where H = [ hy,] and G = .9......IN' INl
Weight update by several training steps The weight update after K training steps is the sum of weight update at each ster AW >k A; (a) >D a;H G H G, where A W (6) and o; is the softmax gradient and learning rate at the time step i, respectively, H [a H (1): (1).....CC H (K)), and G [G G(j)). More generally, we have the following.
Remark 1. AW can be represented as the product of H T E Rdxx and G E SxC, where S is the number of labels used to compute the weight update.
We consider unknown the adversary. If the weight update is computed from a batch, S is the batch size. If the weight update is aggregated from several training steps, S is the total number of samples at these steps. In fact, S can be inferred from the rank of A AW as follows.
We consider non-adaptive optimization methods, e.g. FedAvg [6. which common in FL training.
3

---

Remark 2. Assume that S < minfd, C} and H, G are full-rank matrices. We have rank H) rank(G) S, and can be inferred from = rank( A.).
Now we want to derive a relation between G and labels Y. Differentiating & w.r.t. 2 yields
g.
Oc 021
{
-1 + softmax (2) zi) (a)naaxx(, softmax (z,, zi)
i
otherwise
Since the softmax function always returns a value in (0, 1), each row in G has a unique negative coordinate corresponding to the ground-truth label. Formally, we have
Remark 3. Let Neg(u) define the indices of negative coordinates in vector u. For every row 9: in G, Neg(g,) = {ys)
More intuitively, in order to minimize the loss, the probability of the ground truth label should be pushed up (by a negative gradient), and probabilities of other labels should be pushed down (by positive gradient). Hence, signs in the gradient matrix G can be directly linked to labels Y. Though G is not directly observable from AW, this property of G is critical for revealing the set of labels from A AWW explained in the next section.
2.3 Proposed Method
In this section, we propose method reveal the set of labels S from AW, which we refer to as Revealing Labels from Gradients (RLG). For our method to work, we need to assume that S < minfd, C), i.e., the method works when A W is aggregated from the weight updates computed from a reasonable number of samples. This is justifiable, since when running distributed training on private data on edge devices, is good practice use a small batch size [17]. For sequence models that output natural language, d and C' are usually in the order of thousands or tens of thousands, which is often much larger than
By the singular value decomposition, A AW can be decomposed into PEQ, where P E RR and Q S are orthogonal matrices, and E RSSS is diagonal matrix with non-negative elements on the diagonal. The following remark is a corollary of (3).
Remark (Existence of a hyperplane). Assume that there is sample with label c. There exists vector r E R such that rq < 0 and rq' 0, or Neg(rq) = {c}. where q' is the j-th column in Q. In other words, the point q" can be separated from other points q fe in the S-dimensional space by a hyperplane rx 0.
If label appears in the batch, we have y/ c for some Neg(g,) {c]- Letr 9 then we have rQ = 9,. or Neg(rQ) {c). Therefore, if a label appears in the batch, there exists linear classifier without bias that separates qe from qi The problem of finding a perfect classifier can be solved via linear programming: if there exists classifier that separates of from q² #c, the following problem has a solution.
LP(c): min PERY
rq
s.t.
rg S
0 and
rq 2 V.Vi
(4)
Note that this defines a necessary, but not sufficient condition for e being used to compute the model update. Figure aad Algorithm provide illustration and pseudo-code for approach. In Figure I. the points 02 and ds are separable from the rest by a hyperplane passing through the origin, hence they predieted as a label in S. The other points, however, are not separable. From the remark 4 is certain that they are not included in S.
2.4 Comparison with Related Works
There have been efforts to look at the gradient of the projection layer to infer information about training labels. Zhao et al. [9 are the first to observe that training label can be retrieved from A W. However, they provide method that works only with gradient computed from single data sample. The method is based on the observation that A AW! h g¹. thus AW' VW ||/]ll3'' g¹. Since

---

ope
aga
9
SVD
AW
PXXx
o9
QERSxC
DNN
(2)
Figure I: An illustration of RLG. Black nodes correspond to labels of samples used compute the weight update. (1) The weight update of the projection layer AW decomposed into PEQ by SGD. (2) Each column in is mapped to point in S-dimensional space. If point is separable from the rest, the corresponding label is predicted as being included in
Algorithm 1 Reveal the set of labels from a weight update of the projection layer
Input:
Gradient of the projection layer AW & RRxxC all labels C {c))
S rank(AW)
Find Q E RSC the right singular matrix of AW S
for i to do
if LPcc;) has a solution then
Add label into S end if
end for
Return: Number of labels Y S, set of labels S
ge < O and gا > 0, the training label can be obtained by finding column in AW whose dot product with other columns results in a negative number. Yin al. [7], on the other hand, assume that H > 0, i.e. non-negative non-linear activation function is applied before the projection layer, and demonstrate that labels in a batch can be revealed with high success rate with this assumption. The gradient computed from the batch AW is the super-position of gradients computed from each sample AW (5) Since AW (i) h 9:, Neg(g,)
} and h >> O. the y,-th column in A W (i) has all negative values and other columns have all positive values. They empirically observe that the magnitudes of negative columns dominate those of positive columns, thus a negative element Aw! in AW hints that there exists AWr, (i) with the j-th negative column, or the label j appears in the batch. The set of labels the batch is retrieved by
S s=
-={1 min Aw? 0 sE|d
(5)
Without the O assumption, Yin al.'s method does not work. For H to a non-negative matrix, this approach requires that non-negative activation functions being applied prior to the input of the projection layer. While non-negative activation functions such as ReLU are widely used in an image classification model, other variations such as Leaky RELU [18. elu [19] may return negative values as to address the dying ReLU problem. In sequence modeling, the hyperbolic tangent function (tanh) is usually employed, for example, in LSTM [20], attention mechanism [2]]. Our approach does not assume anything about H. which makes applicable to a variety of deep learning architectures.
However, there are some limitations RLG. First, we require S < min(d, C} for SVD; therefore, the method is not applicable when the number of samples S is large and the number of classes C' is small. Second, since RLGG relies on matrix decomposition and linear programming solutions, slower than (5). and precision issue could arise when AW 0 (e.g. at late training stage).
5

---

Table 1: Precision, Recall, F1, and EM score of set of labels prediction on ResNet50 and Efficient- NetBO. While Equation (5) from Yin et al. [ performs comparably with RLG on ResNet50, only
RLG succeeds on the EfficientNetB0 model.
ResNet50
EfficientNetB00
N
Prec
Recall
F1
EM
Prec
Recall
F1
EM
Yin et (Eq
10
.996
.998
.96
.010
1
.020
0
50
.974
.987
.27
.049
1
.093
0
100
.960
.979
.04
.095
1
.174
0
RLG (ours)
.999
.998
.998
.97
1
1
1
50
.998
.966
.977
.27
.999
1
.999
.95
100
.978
.966
.972
.02
.998
.999
.999
.78
2.5 Experiments
We demonstrate our approach on 1) an image classification task, and 2) an automatic speech recogni- tion task. We consider our task binary classification task, which returns (positive) if label is included in the training data and 0 (negative) otherwise. The Precision, Recall, and FI score can be computed from this setup. We also provide the Exact Match (EM) score to measure the accuracy of the entire set. In the ASR task, we also report the length error (LE, the average distance to the ground length) when inferring the length of transcript from the remark 2
2.5.1 Image Classification
We demonstrate RLG on ResNet [11. which has been chosen as the target model in previous works 7, and EfficientNet [12,, the current state-of-the-art architecture on the ImageNet benchmark
15.
22
We implement the baseline approach in 7 using the equation 5 For EfficientNet, the model
uses the swish activation function 23 24 which may return a negative value. As discussed in
Section 2.4 previous works 7 cannot be adapted this setting. We randomly sample 100 batches 10, 50, 100 from the validation set of ImageNet.
use the ImageNet pre-trained models provided by Tensorflow (Keras) (5 library. The average sizes of the set of labels for batch of 10, 50, and 100 samples are 10.0, 48.8, and 92.5 labels, respectively.
of size
Table T shows the Precision, Recall, FI, and EM score for the baseline and our method. For ResNet50, the baseline method performs well with perfect Precision and near-perfect score. For EfficientNerBO, however, it fails to give the set of labels, while RIG can predict correctly for 95% of the 50-sample batches, and 78% of the 100-sample batches.
2.5.2 Automatic Speech Recognition
We apply RLG on the weight update from encoder-decoder attention-based ASR model 26 15] to demonstrate its application in revealing the Bag of Words (BoW) of spoken utterances. The model word-piece-basee
model, implemented in lingvo (7 using Tensorflow (25). The number of word pieces in the vocabulary is 16k. We demonstrate BoW prediction for utterances from LibriSpeech (28] For each length value of less than equal to 50 characters, we sample at most 10 utterances from the combined test set of LibriSpeech. The total number of sampled utterances is 402.
Equation (5) 7 cannot be used for this architecture, since hyperbolic tangent activation function used in the ASR model. We implement a bascline with Gradients Matching [ to optimize the one-hot matrix of the transcript jointly with the input to the decoder (context vector), by matching the gradient of the decoder. We run the reconstruction with learning rate 0.05, decaying by a factor of 2 after every 4,000 steps. The reconstruction stops after 2,000 steps that the transcript remains unchanged. The method should be able to reconstruct the full transcript; however, we only report the BoW prediction results. In the baseline, weight update is obtained from untrained model.
For RLG, we implement BoW prediction on a weight update from an untrained model, a partially trained model (4k steps), and fully trained model (10k steps). The training is performed on centralized training data, with batch size 96 across GPUs. Since iterating over 16k labels takes time, perform screening round. Consider each column in as data point in a S-dimensional space.

---

Using the Perceptron algorithm (29] the screening round simply uses 500 points with the largest norm filter out all points that are not separable from them. This is faster than solving the full LP problem. Only points that pass the screening round are processed in our algorithm. Each inference takes less than minutes on a CPU.
Table 2 shows LE, Precision, Recall, Fl score of the prediction, along with EM score of the entire BoW. For an untrained model, RLG can recover the exact BoW for 98% utterances. While these results slightly drop on partially trained and trained models, they remain highly accurate with a recall of 1. For the baseline, Gradients Matching fails on most utterances, with only 1.8% EM.
We also demonstrate RLG for a weight update computed from K steps, with a N-sample mini-batch at each step. after one or K training steps. Table 3 shows Precision, Recall, F1, and EM score for the cases of multi-sample single-step (N = 4, 8), single-sample multi-step (K' = 4, 8), and multi-sammle multi-step (N 2, K = 2). While S can be inferred from AW (remark 2). in practice, when S is large, its prediction is erroneous due to precision issue, especially when multi steps are involved (for example, the average length estimate error for the 8-step case 8.18). Here we use the ground-truth length for the value of S. While RLG achieves perfect score on the multi-sample single-step case, there is drop when the weight update is computed from several steps.
LE
Prec
Rec
F1
EM
K
Prec
Rec
F1
EM
GM
.007
.008
.008
.018
4
1
1
1
1
1
RLG
.998
.999
.988
RLG(4k)
.993
.996
.955
.959
.903
.930
.535
RLG(10k)
.092
.989
.994
.933
8
.931
.862
.895
.160
2
2
.914
.955
.330
Table Length Error, Precision, Recall, F1 score of the prediction, along with EM score of the BoW. For RLG, we also run it on a weight update for partially trained model (4k steps), and a trained model (10k steps).
Table 3: Precision, Recall, F1, and EM score of the prediction when the weight update is computed from a mini-batch of N samples, after K steps.
3 Defense Strategies
To defend against our method, distributed training should avoid exchanging precise gradients, so that the linear dependency in (2) does not hold. In this section, we investigate the effect of two common gradient compression techniques, namely gradient quantization (e.g. Sign-SGD) and gradient sparsification (e.g. GradDrop), on RLG.
Gradient Quantization (e.g. Sign-SGD) Using only the sign of gradient for distributed stochastic gradient descent has good practical performance R0 31,, and has been studied in the context of
distributed training to reduce the communication cost with nearly no loss accuracy [32] 33 34 35).
This has also been studied in the context of federated learning with theoretical guarantees [36].
Gradient Sparsification (e.g. GradDrop) Distributed training can be more communication- efficient with gradient sparsification techniques. By proposing GradDrop, which simply maps up to 99% smallest updates by absolute value utility on several tasks. The compression rate can also be self-adapted for better efficiency (38).
Aji et al. [37] demonstrate achieving the same
We use Sign-SGD, and GradDrop with the rate of 50% and 90% the weight update of the projection layer to demonstrate the defensive effects of these techniques on RLG. We report the results of the EfficentNetBO model and the ASR model for the settings that RLG performs best, reported in Section 2.5.1 (i.e., the batch size of 10) and Section 2.5.2 (i.e., the untrained model).
Results Table 4 compares the Precision, Recall, F1, and Exact Match score of RLG for model with and without defense techniques. We can see that Sign-SGD can mitigate RLG in most cases, with 0% EM on EeficientNetB0O and 10.4% EM on ASR. GradDrop, however, does not have a significant effect on the success of RL.G some settings, e-g., the ASR model. Since A AW may already be sparse, applying GradDrop may have little effect on breaking the linear dependency
7

---

Table 4: Precision, Recall, FI, and EM score of different defense strategies on RLG for weight updates computed from an EfficientNetBO model, and an ASR model.
EfficientNetBO
ASR
R
F1
EM
P
R
F1
EM
No Defense
1
.998
1
.999
.988
Sign-SGD
.010
1
.020
0
.504
.181
.266
.04
GradDrop 50%
.999
.843
.914
.17
.857
1
.923
.881
GradDrop 90%
.294
.455
.997
.981
.989
.866
Reveal Sequences of Labels
In sequence models, the algorithm in Section 2 only returns BoW. While the knowledge of BoW, in many cases, can be used to reveal the content of an utterance, many sensitive phrases, such as passwords numbers, may be composed of uncommon word combination. In this section, propose an approach that utilizes the BoW to reconstruct the full transcript.
4.1 Gradients Matching with BoW
We use W model denote the weights of the model. Let (X", ,x""") be the training sample used to compute the gradient of W model- The weight update computed from (X", is AW" model = DC
ZW TW (X y²)). In sequence model, X and y" have variable lengths. A malicious adversary who wants to reconstruct X and yy via Gradients Matching tries to solve the following optimization:
ac & model
min X.U
(X,y)
W model
s.t.
X E cISi
(6)
where T is the length model input, the of labels, and S be the length of the sequence. Besides the model architecture, the current model weights W model- and the weight update W model computed from training sample, the adversary is also required to know the length of the model input 7, and the length of the model output S.
Since y is discrete variable, it cannot be optimized with gradient-based methods. Instead of optimizing for we can optimize for the post-softmax probability distribution P E RXxC which we refer to smooth labels. The loss smooth labels modified from (1)
exp 2
log
j= exp 2/
1= k=
Optionally, regularization term can be added to the objective in (6) to constrain the smooth labels
P. We use the regularization R(P) Σi= ||I|p.1| 1|] to enforce that each row in P sums to 1.
The embedding of a regular label c, i... e E(c,) where È's the lookup dictionary, if required in the network, can be changed to a smooth label embedding e; E k=1 E(cx) Now let + Co2y Cisss} denote the BoW obtained from the Algorithm in Section Since if word included in the ground-truie truth transcript, it is guaranteed to be included in S, our search space can be restricted to words included in More specifically, let P' E Sx denote the smooth labels for words included in the BoW. By setting elements in P that are not associated with word in the BoW
S {Ci,
expa," %i
O, the loss becomes LBow - E' log ES_eep
With the BoW restriction, the search space labels is reduced from S C to (S]. This is significant reduction since the size of the vocabulary C' is much larger than the size of the BoW for a sentence or an utterance. Note that the optimization task (6) also requires knowledge about the length of the model input T", which should not be known by the adversary. In the encoder-decoder

---

Table 5: Number of variables optimize, WER, and EM score of transcript reconstruction by GM with and w/o BoW. For GM with BoW, we also repeat the reconstruction up to times and report the best results in terms of gradients distance.
#vars
WER
EM
GM w/o BoW (I-run)
130k
>1
.018
GM with BoW (I-rum)
20k
.284
.519
GM with BoW (5-run)
20k
.010
.975
1-run
5-run
1.0
1.0
0.5
0.0
0.0
0.5
1.0
0.0
1.0
Checkpoint
1e6
Checkpoint
1eb
EM (BoW)
EM (transcript)
WER (transcriot)
Figure 2: The results for BoW prediction (F1, EM scores) and full transcript reconstruction (EM,
WER scores) from a weight update computed by model checkpoints at 0, 1k, 2k, 4k, and 10k steps.
architecture with attention mechanism, we eliminate the requirement for T by optimizing for the context vectors A SxdA that is given to the decoder via attention mechanism, instead of optimizing for X. Here, S can be inferred from VW in Algorithm [ and d can be inferred from the model architecture, The optimization problem ( becomes
@W min A,
(1
@LBow (A,
VW decoder
+ ++XRP'))
s.t. A A P' R (8)
decoder
where A is the weight of the regularization term.
4.2 Experiments
We conduct full transcript reconstruction on 402 utterances sampled in Section 2.5.2 with and without BoW, by minimizing the objective function in (8) with À = 1. We start with learning rate 0.05, reducing by half after every 4,000 steps until it reaches 0.005. The reconstruction stops when the transcript remains unchanged after 2,000 steps. The reconstruction operates from a few-step trained model. We also run the reconstruction up to five times for GM with BoW and report the results of runs with the lowest gradients distance. The average running time for each reconstruction ranges from 10 minutes to hour on a single GPU, depending on the transcript length.
gradient computed
Table 5 shows the average size of the search space (number of variables to optimize), along with Word Error Rate (WER) and EM score of the experiments. The EM score of GM without BoW only 0.018, which indicates that the reconstruction fails for most utterances. However, GM with BoW can successfully reconstruct more than 50% utterances at the first run, and close to 100% utterances after 5 runs. Note that given BoW, in many cases, it is not difficult task to rearrange the words to form a meaningful sentence. Our experiments, however, do not rely on language model guidance, which demonstrates that reconstruction succeeds with even uncommon word combinations, posing security threat on utterances containing secret sequences.
We also perform reconstruction at different training stages. From 402 sampled utterances, we sample a smaller subset of utterances. We train the ASR model and keep track of checkpoints at 0, 1k, 2k, 4k, and 10k-th step. Figure 2 shows the EM scores, and WER of the reconstructed transcripts. While GM on models at later training stage less efficient, we can still reconstruct the transcript exactly for around 40% of the utterances.

---

Potential Negative Societal Impacts
As discussed in Section 2.1 our method can be potentially used by an adversary that compromises some client-to-server communication channel, by cunest-bu-ccurious host with access model update. The applications range from verifying if a sample of a certain class has been used to compute the update, reconstructing full sequence of labels used for training. Being aware of the existence the techniques, a more secure system could be designed for the privacy of training participants.
Conclusion
We propose RLG, a method to reveal labels from a weight update in distributed training. We demonstrate that RLG can be applied to wide variety of deep learning architectures, and can be used in conjunction with other methods to improve their performance. We also discuss the defensive effects of gradient compression techniques on the success of our method.
References
[1] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances Neural Information Processing Systems, pages 14774-14784, 2019.
[2] Brendan McMahan and Daniel Ramage. Federated learning: Collaborative machine learning without centralized training data. Google Research Blog, 3, 2017.
[3] Luca Melis, Congzheng Song. Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pages 691-706. IEEE, 2019.
[4] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2017.
[5] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how easy is it to break privacy in federated learning? arXiv preprini arXiv:2003.1405: 2020.
[6] Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating gradient leakage attacks in federated learning. arXiv preprint arXiv:2004.10377 2020.
[7] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. arXiv preprint arXiv:2104.00588. 2021.
[8] Trung Dang, Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Peter Chin, and Françoise Beaufays. A method to reveal speaker identity in distributed asr training, and how to counter it. arXiv preprint wrXiv:2104.078.. 2021.
[9] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.0266 2020.
[10] Aidmar Wainakh, Till Müßig, Tim Grube, and Max Mühlhäuser. Label leakage from gradients in distributed machine learning. In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), pages 1-4. IEEE, 2021.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[12] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, PMLR, 2019.
[13] Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents. Journal of Network and Computer Applications. 116:1-8, 2018.
10

---

[14] Praneeth Vepakomma. Abhishek Singh, Otkrist Gupta, and Ramesh Raskar. Nopeek: Informa- tion leakage reduction to share activations in distributed deep learning. In 2020 International Conference Data Mining Workshops (ICDMW), pages 933-942. IEEE, 2020.
[15] W. Chan, N. Jaitly, Q. Le, and O. Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960-4964, 2016.
[16] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication -efficiennteiintn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[17] Khe Chai Sim, Petr Zadrazil, and Françoise Beaufays. An investigation into on-device personal- ization of end-to-end automatic speech recognition models. arXiv preprint arXiv: 19X76,990.00678, 2019.
[18] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Citeseer, 2013.
[19] Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 1. 1.07299, 2015.
[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 1997.
[21] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 0.000.043,,.. 2014.
[22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[23] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv: 1710.05941, 2017.
[24] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.
linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
[25] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th (USENIX) symposium on operating systems design and implementation ( (OSDII) 16), pages 265-283. 2016.
[26] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, volume 28, pages 577-585, 2015.
[27] Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao. Chung-Cheng Chiu, al. Lingvo: a modular and scalable framework for sequence-o--sequence modeling. arXiv preprint arXiv: 190.0099902 2019.
[28] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210. IEEE, 2015.
[29] Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6))386, 1958.
[30] Martin Riedmiller and Heinrich Braun. Rprop-a fast adaptive learning algorithm. In Proc. of ISCIS VII), Universitat. Citeseer, 1992.
[31] Geoffrey Hinton, N Srivastava, K Swersky, T Tieleman, and A Mohamed. Coursera: Neural networks for machine learning. Lecture 9c: Using noise as a regularizer, 2012.
[32] Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-paralled distributed training of speech dnns In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
[33] Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560-569. PMLR, 2018
11

---

[34] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252-3261. PMLR, 2019.
[35] Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint arXiv:2002.08855, 2020.
[36] Richeng Jin, Yufan Huang, Xiaofan He, Tianfu Wu, and Huaiyu Dai. Stochastic-sign sgd for federated learning with theoretical guarantees. arXiv preprint wxXi:2202.109940 2020.
[37] Alham Fikri Aji and Kenneth Heafield Sparse communication for distributed gradient descent. arXiv preprini 170.000000 2017.
[38] Chia-Ye Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrishnan. Adacomp: Adaptive residual gradient compression for data-parallel distributed training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12


Reviews:
[{"review_id": "yt14rHhPm9u", "reviewer": "Reviewer_scvf", "summary": "The paper proposes a new approach for revealing the training labels from the gradients of the last layer. The approach relies on the fact that if a label is used for computing the weight update, its corresponding column in the transposed right singular matrix can be linearly separated from other columns. The paper also experimented with how current gradient compression techniques can affect the attack accuracies, and explored how to reveal the sequence of labels, by combining the proposed approach with gradient matching.", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do not work in this area and I am not familiar with related work. I am not able to judge the originality and significance of the work. The following questions/suggestions are mostly on writing and clarity.\n\n* Related work is discussed both in Section 1 and Section 2.4. Maybe it is better to put them in one place.\n* In Section 2, the problem is defined in a way that the adversary only has access to some model updates. However, the approach in Section 4 needs more information than that (e.g., model weights). This makes the paper less coherent.\n* The approach in Section 2 only needs the update of the weight matrix in the last layer. From a research standpoint, I understand that it is interesting to explore how we can use less information to extract what we want. However, I am curious how common it is in practice that the adversaries only have access to the update of the weight matrix in the last layer? In other words, if in practical scenarios, the adversaries have more information than that (e.g., the updates of all model weights), how your approach can be improved? Maybe it is useful to briefly discuss this in the paper.\n* Line 112: the learning rate assumption does not work for common optimizers which have different adaptive learning rates for different values in the weight matrix (e.g., Adam).\n* Eq 3: since softmax also depends on other values, it might be better to change the notation to explicitly point out which zs are used to in softmax normalization.\n* Remark 4: q (the columns of Q) is not defined in the main text (only illustrated in Figure 1).\n* Remark 4: from what I understand, q^c being linearly separatable from other columns does not necessarily ensure the existence of label c. In other words, the proposed algorithm could generate false positives. Is this correct? If so, this should be discussed in the paper.\n* Besides the above, what other factors contribute to the drop of recall and precision? You mentioned in the experimental sections that the estimation of S could be inaccurate. Please discuss if there are other sources of errors, and show how much each of them contributes to the total errors.\n* Does gradient matching (GM) applicable to the experiments in Section 2.5.1? And if so, why don't you compare with it? I see that you compared with GM in Section 2.5.2 and Section 4 only.\n* Line 230: It mentioned that the prediction of S is inaccurate and therefore you use the ground-truth S for this experiment. This is inconsistent with the problem setup where you do not assume to know S. Please also show the results with the estimated S.\n* Table 1: it is hard to compare the two algorithms as the corresponding numbers are not listed together. \n* The current structure of the paper is attack (Sec. 2) -> defense (Sec. 3) -> attack (sec. 4). This is not a natural flow. Maybe you can consider reordering the sections.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations and potential negative societal impact of their work adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "s2rmsGxiiv", "reviewer": "Reviewer_u2De", "summary": "The work shows that a set of labels used in a small batch of the training procedure is identifiable given the gradient of the last layer of the NN model with CE loss and soft-max activation. Also, they propose to modify gradients for defense.\nThere are some theoretical and empirical gaps between this work and real-world settings. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the work generalizes the recovery scenarios to multiple samples and steps, compared with a single sample and step solution by an earlier pioneering work\n- the logic is clear\n\nWeakness:\n- It seems that the label not used in updating step also has the probability to be identified as \"used\". it is critical to provide rigorous proof for their proposed method.\n- As the authors admit, their method does not work when the batch size is greater than the number of classes, which might be a major concern for application.\n- The notation is unorganized and there are many typos that cause unnecessary troubles for understanding, e.g., the main result Remark 4.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "l0z_xcqRU3l", "reviewer": "Reviewer_Bkhg", "summary": "This paper presents a method, RLG (Revealing Labels from Gradients), from only the gradient of the last layer and the id to label mapping. The method is tested on two domains: (1) image classification; and (2) automatic speech recognition. The results indicate that under a reasonably small batch size, the proposed method is robust to a wider range of data sets while the baseline performs well only on a smaller subset. The paper next shows that gradient quantization and gradient sparsification could be possible defense strategies against the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a new method to uncover potential vulnerabilities in distributed learning systems.\nThe method, its performance and possible defense strategies are described well. \n\nQuestions: \n\n1.  How much accuracy can be lost when the defense strategies are used?  \n\nThe paper is written well. \n\nOne minor typo: a honest-but-curious  -> an honest-but-curious \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "aCpcdNYQLDg", "reviewer": "Reviewer_6fay", "summary": "The primary objective of paper is to propose a method which can reveal labels only from shared gradient information in distributed learning methodologies. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows a clever mathematical derivation of how gradients are connected to labels - as a product of two row matrices h,g. the matrices themselves are derivative of loss function and hidden layer activations - and linear programming and SVD based methods can be used to identify the labels under certain conditions. the paper demonstrates results on ASR and Imagenet. the paper suffers from certain limitations - overlooking other methods in distributed learning and just focusing on federated learning, only looking at label reconstruction, only looking at vanilla version of federated learning, not reporting variance on metrics reported - and fundamental limitation of algorithm itself S < min{d, C}", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper shows a mathematical approach using SVD to predict labels using gradients from final layers in neural network, while this is a novel approach a few improvements can be recommended:\n\n1. Security flaws in federated learning (and split learning) are well known and active research is being done on mitigation. However authors only look at the vanilla gradient sharing approach and then talk about mitigations. As [1] proposes there are methods including gradient noise to mitigate such flaws. This limits the impact of paper. and paper would benefit from analysis of leakage when security measures in papers like [1] are being applied.\n2. Most security work is focused around reconstruction of data and not labels - there is a reason for that. Raw data contains important information about privacy of individuals. While labels are crucial too - its easy to obfuscate the labels and project them in a different embedding - A second crucial factor about impact of a label recovery focused method.\n3. The paper talks about revealing labels in distributed learning. However authors only address it for federated learning. There are other distributed learning methodologies [1, 2] which should be both referenced in related work and studied from point of view of security. \n4. While multiple architectures and datasets are used - the paper doesnt report variance numbers / error metrics around the prediction - making it hard to assess the robustness of method from experiments alone.\n5. The paper is difficult to understand - exposition could be simplified.\n\n\n\n[1] Vepakomma, Praneeth, et al. \"NoPeek: Information leakage reduction to share activations in distributed deep learning.\" 2020 International Conference on Data Mining Workshops (ICDMW). IEEE, 2020.\n\n[2] Gupta, Otkrist, and Ramesh Raskar. \"Distributed learning of deep neural network over multiple agents.\" Journal of Network and Computer Applications 116 (2018): 1-8.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper gives an interesting way to reveal label information from the transmitted gradients in distributed training. Empirically, the paper also shows that techniques for communication reduction mitigate the privacy leakage of label information. Overall, this paper gives new insights into what information can be leaked from the model updates in federated learning. The authors should incorporate the reviewers' suggestions in the next revision.
Decision: Accept (Poster)