Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method, RLG (Revealing Labels from Gradients), from only the gradient of the last layer and the id to label mapping. The method is tested on two domains: (1) image classification; and (2) automatic speech recognition. The results indicate that under a reasonably small batch size, the proposed method is robust to a wider range of data sets while the baseline performs well only on a smaller subset. The paper next shows that gradient quantization and gradient sparsification could be possible defense strategies against the proposed method. 
main_review: The paper presents a new method to uncover potential vulnerabilities in distributed learning systems.
The method, its performance and possible defense strategies are described well. 

Questions: 

1.  How much accuracy can be lost when the defense strategies are used?  

The paper is written well. 

One minor typo: a honest-but-curious  -> an honest-but-curious 
 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The primary objective of paper is to propose a method which can reveal labels only from shared gradient information in distributed learning methodologies. 
main_review: The paper shows a clever mathematical derivation of how gradients are connected to labels - as a product of two row matrices h,g. the matrices themselves are derivative of loss function and hidden layer activations - and linear programming and SVD based methods can be used to identify the labels under certain conditions. the paper demonstrates results on ASR and Imagenet. the paper suffers from certain limitations - overlooking other methods in distributed learning and just focusing on federated learning, only looking at label reconstruction, only looking at vanilla version of federated learning, not reporting variance on metrics reported - and fundamental limitation of algorithm itself S < min{d, C}
limitations_and_societal_impact: The paper shows a mathematical approach using SVD to predict labels using gradients from final layers in neural network, while this is a novel approach a few improvements can be recommended:

1. Security flaws in federated learning (and split learning) are well known and active research is being done on mitigation. However authors only look at the vanilla gradient sharing approach and then talk about mitigations. As [1] proposes there are methods including gradient noise to mitigate such flaws. This limits the impact of paper. and paper would benefit from analysis of leakage when security measures in papers like [1] are being applied.
2. Most security work is focused around reconstruction of data and not labels - there is a reason for that. Raw data contains important information about privacy of individuals. While labels are crucial too - its easy to obfuscate the labels and project them in a different embedding - A second crucial factor about impact of a label recovery focused method.
3. The paper talks about revealing labels in distributed learning. However authors only address it for federated learning. There are other distributed learning methodologies [1, 2] which should be both referenced in related work and studied from point of view of security. 
4. While multiple architectures and datasets are used - the paper doesnt report variance numbers / error metrics around the prediction - making it hard to assess the robustness of method from experiments alone.
5. The paper is difficult to understand - exposition could be simplified.



[1] Vepakomma, Praneeth, et al. "NoPeek: Information leakage reduction to share activations in distributed deep learning." 2020 International Conference on Data Mining Workshops (ICDMW). IEEE, 2020.

[2] Gupta, Otkrist, and Ramesh Raskar. "Distributed learning of deep neural network over multiple agents." Journal of Network and Computer Applications 116 (2018): 1-8.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new approach for revealing the training labels from the gradients of the last layer. The approach relies on the fact that if a label is used for computing the weight update, its corresponding column in the transposed right singular matrix can be linearly separated from other columns. The paper also experimented with how current gradient compression techniques can affect the attack accuracies, and explored how to reveal the sequence of labels, by combining the proposed approach with gradient matching.
main_review: I do not work in this area and I am not familiar with related work. I am not able to judge the originality and significance of the work. The following questions/suggestions are mostly on writing and clarity.

* Related work is discussed both in Section 1 and Section 2.4. Maybe it is better to put them in one place.
* In Section 2, the problem is defined in a way that the adversary only has access to some model updates. However, the approach in Section 4 needs more information than that (e.g., model weights). This makes the paper less coherent.
* The approach in Section 2 only needs the update of the weight matrix in the last layer. From a research standpoint, I understand that it is interesting to explore how we can use less information to extract what we want. However, I am curious how common it is in practice that the adversaries only have access to the update of the weight matrix in the last layer? In other words, if in practical scenarios, the adversaries have more information than that (e.g., the updates of all model weights), how your approach can be improved? Maybe it is useful to briefly discuss this in the paper.
* Line 112: the learning rate assumption does not work for common optimizers which have different adaptive learning rates for different values in the weight matrix (e.g., Adam).
* Eq 3: since softmax also depends on other values, it might be better to change the notation to explicitly point out which zs are used to in softmax normalization.
* Remark 4: q (the columns of Q) is not defined in the main text (only illustrated in Figure 1).
* Remark 4: from what I understand, q^c being linearly separatable from other columns does not necessarily ensure the existence of label c. In other words, the proposed algorithm could generate false positives. Is this correct? If so, this should be discussed in the paper.
* Besides the above, what other factors contribute to the drop of recall and precision? You mentioned in the experimental sections that the estimation of S could be inaccurate. Please discuss if there are other sources of errors, and show how much each of them contributes to the total errors.
* Does gradient matching (GM) applicable to the experiments in Section 2.5.1? And if so, why don't you compare with it? I see that you compared with GM in Section 2.5.2 and Section 4 only.
* Line 230: It mentioned that the prediction of S is inaccurate and therefore you use the ground-truth S for this experiment. This is inconsistent with the problem setup where you do not assume to know S. Please also show the results with the estimated S.
* Table 1: it is hard to compare the two algorithms as the corresponding numbers are not listed together. 
* The current structure of the paper is attack (Sec. 2) -> defense (Sec. 3) -> attack (sec. 4). This is not a natural flow. Maybe you can consider reordering the sections.
limitations_and_societal_impact: The authors discussed the limitations and potential negative societal impact of their work adequately.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The work shows that a set of labels used in a small batch of the training procedure is identifiable given the gradient of the last layer of the NN model with CE loss and soft-max activation. Also, they propose to modify gradients for defense.
There are some theoretical and empirical gaps between this work and real-world settings. 
main_review: Strengths:
- the work generalizes the recovery scenarios to multiple samples and steps, compared with a single sample and step solution by an earlier pioneering work
- the logic is clear

Weakness:
- It seems that the label not used in updating step also has the probability to be identified as "used". it is critical to provide rigorous proof for their proposed method.
- As the authors admit, their method does not work when the batch size is greater than the number of classes, which might be a major concern for application.
- The notation is unorganized and there are many typos that cause unnecessary troubles for understanding, e.g., the main result Remark 4.
limitations_and_societal_impact: addressed
review_title: null

Meta-review: This paper gives an interesting way to reveal label information from the transmitted gradients in distributed training. Empirically, the paper also shows that techniques for communication reduction mitigate the privacy leakage of label information. Overall, this paper gives new insights into what information can be leaked from the model updates in federated learning. The authors should incorporate the reviewers' suggestions in the next revision.
Decision: Accept (Poster)