Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the attack that tries to recover text data from the gradients, this type of attack is a threat particularly for federated learning where the central server may recover the private client data through gradients. Based on previous work that optimises the input to minimise the distance between gradients, this paper further proposes to alternate continuous optimization and discrete optimization that is guided through a language model prior. The discrete optimization is claimed to help obtain text data that is more like fluent language. The resulting approach, LAMP, greatly outperforms previous approaches on three binary-classification benchmarks.

strengths_and_weaknesses: ### Strengths:

1. The proposed approach is novel and technically sound. The motivation and contribution are clear – from the examples in Table 2 and quantitative improvements, it does seem like previous approaches fail to yield grammatical text while the proposed discrete optimization step seems to help it a lot.
2. The empirical results are strong especially with batch size > 1. The ablation study is appreciated as well.
3. The paper is well-written. 



### Weaknesses:

1. *(minor)* The proposed approach adds several additional hyperparameters to tune, for  example, $\alpha_{lm}, \alpha_{reg}, n_c, n_d, n_{init}$. This surely adds complexity of the approach and may make the proposed attack difficult to be practically applied. While I appreciate the detailed hyperparameter paragraph in Line 268-274, I think it would be better to report the hyperparameter selection range as well, so that the readers could have an idea how much effort is imposed to tune these hyperparameters.

2. *(major)* I am a bit worried that the model comparison is only conducted on a randomly selected set of 100 sentences. 100 sentences sounds too few for me, and I am not sure how robust the model rank is based on only 100 random examples. I feel this point should be justified properly, either through constructing a larger test set, or using different random seeds to generate different sets of 100 test examples and testing on each of them.
 
3. *(minor)* While Line 239 mentions to illustrate the generality of the proposed approach with respect to model size, I don’t think this paper really made that goal by only using tiny and base sizes of BERT – at least BERT-large should be included to have a relatively complete coverage. I understand that the authors may be limited by resources to use larger models, which is fine. I just wanted to point out the current experiments are not sufficient to indicate generality with respect to model sizes (this is rather a minor point anyway).  

4. *(major)* Line 257-259 mentions the baselines use 2500 iterations while the proposed approach uses 2000. Is *it* in Algorithm 1 2000 (and what are the values of *n_c* and *n_d*)? How did you choose these numbers? Do the baselines fully converge? I would like to see more justifications to show that the comparison is fair, because LAMP employs a nested for loop (while the baselines do not?) in optimization and I feel the number of total optimization steps (and the cost/time) in LAMP is actually larger than the baselines, right?

5. *(minor)* Line 272 mentions that LAMP additionally adopts a two-step initialization procedure that seems important to me. I would like to see the ablation results on this two-step init in Table 3 to know how much of the improvement over baselines is from this initialization.


```
After author response:  
The author response addressed most of my concerns, and I would like to increase my score to 7 given that the authors will update the paper accordingly as promised.
```

questions: I listed all my questions or suggestions in the weaknesses above, I would like to improve my rating if the authors could address them in the rebuttal, particularly the *major* points.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a model for recovering user data from gradient updates in a federated learning system for text classification. They achieve this by alternating continuous gradient based optimization with discrete heuristic based token-reshuffling. The authors show that the proposed model outperforms methods that use only gradient based updates to the tokens.
strengths_and_weaknesses: The primary contribution of the paper is the alternation between gradient-based updates and token reshuffling for reconstructing user text. I am unaware if such an approach has been attempted for adversarial attacks in text. Hence, this can be assumed to be novel.

The paper is well-written and easy to follow. The ablation studies confirm the importance of token-reshuffling for learning a good attack. The ablation studies also show the importance of L1+L2 loss (although this was proposed in a different paper).

From a novelty perspective, this alternation between token reshuffling and continuous updates for attacking text classifier appears to be novel. However, I am not an expert in this field.

Some parts of the paper are unclear. For instance, the authors mention that they use Adam for learning the embeddings for each input during the continuous optimization phase. However, they do not mention how they compute the gradient of the loss with respect to the input embeddings x*. The gradient computation requires second order derivatives which hasn’t been discussed in the paper at all.

It is unclear how token-reshuffling with continuous gradient-based updates influences the descent direction. Specifically, I am wondering if token-reshuffling is only performed at the end of all gradient-based updates, how will the performance be affected.

questions: How important is to alternate token-reshuffling updates with continuous optimization? What if the token-reshuffling appears only at the end of the continuous updates?
limitations: The limitations haven’t been discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel strategy to attack pretrained models for text models which aims to reconstruct the private user data used to finetune the model through federated learning. The algorithm takes the finetune gradient as well as the target label as the input, then search for the private user input sentence. The general idea is a iterative two-step process, where the first step is a continuous optimization to search for a embedding that leads to similar gradient, and the second step is a discrete optimization that uses GPT models to find a output sentence with the lowest perplexity.

####
Thanks for the authors' response! I think it might be worth it to clarify the main novelty of the paper in the main method section. It would also be nice to clearly discuss the limitation at the end of the paper, as researchers not familiar with this line of research might benefit from some background information about the social impact/limitation of the method.
strengths_and_weaknesses: strengths:
1. the method is very interesting and novel. It also addresses an interesting problem in the pretrain-finetune paradigm that is very popular now.
2. the writing is mostly well structured and easy to understand. 
3. the final performance also seems good compared to previous methods. The ablation study and examples are pretty nice addition to the results.

weakness:
1. the paper didn't really clarify the main difference between the proposed method and the prior work, such as TAG. Therefore, it is hard to tell what are the exact novelty the paper adds and the effect of the novelty. My understanding is that the random transformation and the use of GPT models allow the output more natural. However, it is really hard to tell from the current structure of the paper.
2. the method can only be applied to classification where label is known, which seems pretty limited. It might be more natural to have unlabeled data for federated learning setting, as one might not be able to annotate the user text.
questions: 1. Could you clarify the major difference between your method and prior work?
2. How can this method be extended to the more realistic setting where one does not have labels for the data?
limitations: The authors did not address limitations and potential negative impact. There are several points worth mentioning:
1. Works on adversarial attack could be exploited by hackers.
2. The method is only limited to classification data.
review_title: null

Meta-review: metareview: This paper describes a novel method to recover the input text based on the computed gradient. This is important in the context of federated learning, which promises to enable learning through gradient sharing while keeping the input text secret. The findings of the paper demonstrate that gradients are sufficient to recover significant parts of the input text questioning the federated learning premise at least in the context of large language models.

The approach in novel and technically sound. Empirical results are convincing. The paper is well-written and clear.

Given current trends to growing model size, it will be great if the paper can further scale the experimental results to larger models.
confidence: Certain
Decision: Accept