Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a privacy attack scheme for federated learning, retrieving the private data from the model aggregation techniques.

main_review: The idea looks interesting, but I have some concerns as below.

1. The assumption used in Proposition 1 is not realistic
    1. I’m not sure how come the server knows the CDF of “some quantity associated with user data”. By the way, would you clarify “some quantity associated with user data”? It is vague.
2. Does the reconstruction available for all cases?
    1. It seems like several figures has gray boxes, meaning that the image is not recovered. Do we have some understanding why/when such failure cases occur? If not, it is better to delve into what happens to that failure scenarios and what is the possible fix for that failure.
3. No comparison with related works?
    1. I have no idea what are other schemes handling malicious server scenario, but if we can find some, it is necessary to compare this scheme with conventional schemes. 

summary_of_the_review: This is an interesting paper, but it is better to include more concrete theoretic/empirical analysis on why/when the suggested scheme works. Moreover, comparison with existing schemes is missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new attack on federated learning, demonstrating that model updates shared in a federated setting can still leak user data.  Although federated learning has been used as a privacy technique, this paper shows that other mitigation techniques should also be used.  Attacks before were based on updated from a single data point, and hence aggregating data in an update was used as one approach against this attack.  The adversarial setting here is with a malicious server who may modify the model’s architecture in a seemingly innocuous way.  Allowing models to be changed in ways that are currently allowed with FL APIs, this paper shows that user data can still be reconstructed even with large aggregations.  Another benefit of this approach is that it is more data agnostic, not relying on strong data priors.  

main_review: The result (Proposition 1) in the imprint module section has a strong assumption that the server knows the CDF of a quantity associated with user data.  This seems contradictory to the statement in the paper that claims the results here are more data agnostic.  

Although the imprint module can be inserted in any position in a neural network (subject to a constraint), I am left wondering whether it would make it better for the attack to put the imprint module earlier or later in the neural network.  I would be interested to see results when testing different positions in the neural network.

The experiments show the power of the attack, while demonstrating that not every image is reconstructed.  Regarding Figure 4, I understand that the license plate is blocked for privacy reasons, but I wonder if the reconstructed image still has a readable license plate.  

In the potential defense and mitigation strategies section, I would be interested to know the overall local privacy guarantee, in particular what dimension are the updates?  I would expect sigma = 0.01 to offer better privacy guarantees, but maybe the updates are high dimensional.  I would have liked this section to be more fleshed out.  

### UPDATE ###
I have read over the author feedback and think the suggested changes can improve the work.  I will keep my score unchanged.

summary_of_the_review: Overall a nice attack that shows the keeping data distributed while only sharing model updates does not provide strong privacy guarantees.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces novel attacks for reconstructing training data given the device contribution in the federated learning setting. The attacks exploit the gradient behavior of ReLU and insert carefully tailored feed forward layers into a benign network so as to reconstruct the input from the gradients. The demonstrated examples show the effectiveness of the attacks that even training with large batch size, attacks with certain knowledge about the distribution of the data could recover almost the exact training data. 

main_review: Strengths
1. Well-written and organized paper. The threat models and attacks are clearly described and well-motivated. 
2. The proposed imprint module is very novel and exploits the vulnerabilities of gradient computation in popular network design.
3. The attack experiments are well-executed. The reconstructed results are really close to the original inputs, even for gradients aggregated over a large batch size or multiple local iterations or with local DP, which is much better than reconstructing the averaged inputs as most previous works do.

Weaknesses and questions
1. Stealthiness of the attacks, as acknowledged by the authors, might be a problem and the attacks could be easily detected by inspecting the model architecture or parameters. Could the parameters in the imprint modules themselves reveal the attacks, given their distribution might be very different from other layers? Is it possible to disguise the parameters as normal layer parameters as well?
2. Since the imprint module changes the intermediate activations, how does this affect the overall utility of the models? Unfortunately, I did not see any results on how the architecture might change the model performance. If the performance is surprisingly different for a data on device, then the device owner might know there is something wrong with the particular model received from the server.
3. I did not understand the setup for the federated learning experiments in Section 4.3. How many users are sampled in each iteration? How are their data splitted? Does the attack work even when there are multiple local iterations and a large number of users sampled in the central iteration? I.e. aggregation of model updates from a large number of users (i.e. magnitude of thousands) trained with multiple local SGD (which could be the setup when large tech companies deploy FL)
4. There could be another DP related mitigation where the model updates are first securely aggregated and then added with central DP noise before sending to the server. Could the server still infer anything from the noised aggregate?  

summary_of_the_review: This paper proposed novel and effective data reconstruction attacks in federated learning. Overall, I recommend accepting this paper for its solid technical contribution and convincing results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores the privacy of federated learning when the server is malicious, specifically that it is allowed to control the architecture and and set certain weights in adversarial ways. By building on a weakness inherent to the RELU activation function, they show that w.h.p the malicious server can obtain exact reproductions of at least some inputs. They then empirically explore how well this works in practice, showing that the method introduced does pose a significant risk to user data privacy. 
main_review: Strengths:
- Provides clear and easily understood high level message: Federated Learning is very vulnerable if the server cannot be trusted. 
- The paper does not rely on simplifying and unrealistic assumptions. It would be (reasonably) simple to implement the method in practice. 

Weaknesses:
- Paper's builds heavily on key insights from previous works, such that from the previous works one could deduce the results in this paper without too much difficulty. And this insight came from previous works on privacy in deep learning, meaning this paper is not making a contribution by newly bringing this insight into the community. 
summary_of_the_review: The paper provides another piece of evidence that federated learning on its own is not sufficient to protect user privacy, and explains how even with only a minor malicious change the server can exactly recover user data. And the message is very clear and well exposed in the paper. However method is only a slight variation on ideas from previous work.
review_title: null

Meta-review: The authors provide an interesting improvement on privacy attacks in federated learning, demonstrating the ability to extract individual points even over large batches. While there were some concerns about the technical difficulty of the approach, reviewers were broadly in support of the work. As I tend to agree, this is an interesting strengthening beyond what it appears we were able to do before. This is yet another piece of evidence against the canard in FL that only sharing gradient updates provides privacy guarantees.
Decision: Accept (Poster)