Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: - This work proposed FedRecon, a method that splits the client model into local and global parameters to enable stateless federated learning
- During training, the client trains the local and the global model using separate splits of the client dataset, but only transmit the global data for aggregation
- Clients with no local parameters trained (clients not seen during the training), can reconstruct/train their local parameters after fetching the global parameters from the server using the local training data. Similarly, other clients which might have either stale local parameters or might have lost/forgotten the local parameters can reconstruct them too
- This makes the method stateless since the clients need not store the global parameters (sever stores them) and the local parameters (since they can be reconstructed)
- The authors show good empirical performance on matrix factorization and next word prediction task. Additionally, the authors also the practical utility and scalability of the method through the deployment of the method in a real-life scenario.
main_review: **Strengths**

- The proposed method is a simple extension of the standard federated learning setup, with the difference being only in how the local model is trained. Experiments show good performance with a wide range of hyperparameters (support and query data splits, local update steps, etc), and the method is applicable to a wide array of models.
- I also appreciate the authors providing the prior submission review. It helped me understand how the paper has evolved over multiple submission cycles.
- Open-sourcing of code through the TensorFlow Federated Library for future experimentation and practical use is a good contribution. However, it might be my ignorance of the framework but I would like to understand how is it different from someone open-sourcing the code for any other submission to the conference. Is the open-sourced code the same used by authors to run their experiments, or they modified it especially to release it as a separate framework?
- The deployment to a mobile application proves the practical applicability of the method and is a good contribution. However, the authors mention in the Submission History section that "[practical deployment results are] rare in FL papers which typically achieve only simulation results". I would disagree with this comment. Results on simulated settings do not necessarily mean inapplicability to practical deployments. Recent datasets do have clients in the order of hundreds of thousands, which mimics the practical setting to a certain extent. Additionally, very few researchers have the resources to deploy to millions of devices, and while these results are good to have in the paper, their absence should by no means be negatively looked at.


**Clarifications and Improvements**

- Table 1: What is the Server + Standard Eval and Server + ReconEval training setup? Can the authors describe this somewhere in the paper or refer to a prior work that utilizes this setup?
- Table 1, 2: FedRecon for matrix factorization is trained for 50 gradient steps each for the local and global params, and for NWP it's trained for 100 gradient steps each. How many gradient steps were the baselines trained for? Can the authors please provide the hyperparameters used to train the baselines as well?
- Lines 263 - 265: The authors mention that the results for Server+ReconEval varied widely across users. Since they ran 500 iterations of the experiment, can the authors please provide the standard error numbers along with the reported mean accuracy to help better understand the variation amongst clients, and compare this variation with FedRecon
- I felt that the experiment section was informative but was slightly convoluted to follow and understand. For a future version of the paper, the authors might consider refactoring the section to make it more readable.



Overall, I felt this work:

- tackles a relevant problem in federated learning, 
- proposes a simple, widely applicable to solve the problem, and
- empirically shows good performance in simulated and practical scenarios.

Thus, I would vote for its acceptance.
limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors proposed a new FL framework to address data privacy, communication cost, and non-iid data distribution issues in FL. In the proposed method, clients only send and receives partial models to cloud and reconstruct the other half from the clients' own data. 
main_review: I have the following major concerns about the paper. 

1. What does the model agnostics refer to? Do the authors mean the clients can have different models or the framework can handle different tasks? Please clarify.

2. For the tasks, I agree the task like matrix factorization is suitable for the proposed method. However, for other tasks where each client trains a deep model and aggregates all the clients' models on the cloud, I don't think only compare with FedYogi is enough. FedYogi is an extension from FedAvg which in general only produces a global model and is not suitable for personalization. In my opinion, the authors should compare at least of the SOTA personalized FL methods for their performance, such as 

a. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

b. Personalized Cross-Silo Federated Learning on Non-IID Data

c. Federated Optimization in Heterogeneous Networks

d. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

e. Adaptive Personalized Federated Learning

f. Federated learning ´of a mixture of global and local models

3. The proposed method is a heuristic method without any theoretical convergence guarantee. 

4. I also wonder whether this framework works if the local model is a complex and large model. In addition, how does the partition size affect the proposed framework? Especially, a large model is needed.

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a new approach to partially local federated learning. In contrast to previous works their approach does not require agents to maintain state (i.e. store the local parameters of the model). Instead, they reconstruct the personalized parts of the model on the fly from local data.  Such a state-less approach is crucial for any practical deployment of FL at scale since is enables new clients that have not yet trained their local parameters to participate in training and to perform inference.
Their approach (FedRecon) is inspired by model-agnostic meta learning, where fast reconstruction at test-time is a key design criteria. 

Empirically the performance of FedRecon is evaluated along different axis. The authors show that their approach trains global parameter that are particularly well suited for reconstruction, the approach achieves performance comparable to personalization through fine-tuning of a global model, and they study the qualitative behavior of their method wrt several hyperparameters. 
In addition the authors report on a successful real-world deployment of the method across hundreds of millions of clients and discuss the practical challenge of data sparsity. 

The code for reconstructing the experiments is packaged as a library to allow users to seamlessly integrate other tasks and algorithms. 

main_review: The paper extends partially local federated learning to settings where agent can not maintain state. This scenario seems well motivated from a practical perspective and the authors give several good application examples. The core idea of using a local reconstruction algorithm to obtain the local parts of the model is simple but interesting and to the best of my knowledge it is novel in this context. 

My main criticism is the presentation of the empirical results in Section 5. The baselines are not well introduced and the goal of the individual comparisons is not clear a priori. This makes it hard to follow (in particular the first part in 5.2, Table 1/2) and as a result your baselines seem a bit constructed at a first glance. 

I think it would help if you make clear from the beginning that a direct comparison to existing methods is not possible because there is no other state-less approach that keeps parts of the model parameters entirely local. Then, you could one-by-one outline what aspect of your method you want to study and then introduce the baselines for this purpose. For example, the ReconEval evaluation scheme for Table 1 would make more sense to the reader if they knew that you aim to show that your method learns global parameters that are particularly well suited for reconstruction later on, in comparison to other methods of obtaining these global parameters. 

In this context, a comparison to a global only model might also be interesting, i.e., a model where you ignore the private features completely. This would help clarify the need for local steps in the ReconEval procedure that you introduce. 

In 263 you mention that results vary a lot across users. Could you elaborate more on this? Showing the distribution over accuracy values of users for the different methods (instead of only RMSE) would be useful here and provide more insights into this interesting finding.

It is nice to see that you actually deployed this federated learning method in a real application. You mentioned that data sparsity was an issue and prevented the naïve implementation of your method.  Were there any other practical challenges you were facing? You mention a 29.3% increase in CTR, this claim is not particularly useful without a reference. What method was used in the previous system?  

Overall, I think the authors adequately addressed the concerns of the meta-review from the previous review round. If the authors are willing to make an effort to address my concerns about the presentation of the empirical results, I will recommend acceptance. 

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a partially local federated learning method named FEDRECON based on federated reconstruction.  The experimental results demonstrate the effectiveness of FEDRECON. The authors also deploy this approach in a real-world mobile keyboard application.
main_review: Pros:
1. This paper is well written and easy to follow.
2. The proposed approach is interesting.
3. The authors provide a nice code library.

Cons:
1. The contributions of this work over other partially local federated learning methods are not clearly explained.
2. The motivation of reconstruction is not clarified.
3. Some related works are not cited nor compared.

Detailed comments:
1. The technical contributions of this work are not well elaborated. Partially local federated learning is a widely studied problem [1-5], especially in the recommendation field. In these methods, each client locally learns a user embedding while item embeddings are global parameters. The authors' discussions in Appendix A do not clarify why this setting is not optimal. 
2. The motivation of reconstruction is also not clarified. For example, in federated recommender systems, why personalized parameters can be generated from shared global ones? What is the intuition?
3. Some related works are not cited nor compared. The authors need to compare the methods in the recommendation field that locally keep personalized parameters. In addition, the authors may consider comparing some meta-learning based federated learning methods like [2] to better support the key statements.

[1] Chai, Di, et al. "Secure federated matrix factorization." IEEE Intelligent Systems (2020).
[2] Lin, Yujie, et al. "Meta Matrix Factorization for Federated Rating Predictions." SIGIR 2020.
[3] Ge, Suyu, et al. "Fedner: Privacy-preserving medical named entity recognition with federated learning." arXiv preprint arXiv:2003.09288 (2020).
[4] Ammad-Ud-Din, Muhammad, et al. "Federated collaborative filtering for privacy-preserving personalized recommendation system." arXiv preprint arXiv:1901.09888 (2019).
[5] Wu, Chuhan, et al. "Fedgnn: Federated graph neural network for privacy-preserving recommendation." arXiv preprint arXiv:2102.04925 (2021). 
limitations_and_societal_impact: The authors discussed the privacy-related issues of the proposed approach and related limitations in the appendix. However, it is encouraged to include some discussions in the main content.
review_title: null

Meta-review: I have carefully read all reviews, rebuttals and subsequent discussion. The proposed methodology relies on a sequence of intuitive shortcuts, reformulations and claims, which in totality make intuitive sense, but are not supported by any theory, and as such, the approach remains a heuristic. For example, the original cross device FL formulation is not well motivated, despite the fact that it is used in nearly every paper on the topic. This is not a particular issue with the current paper - this is an issue with all works in this area.  The authors rightfully claim that statelessness can be a problem in practical deployments of FL in the cross device setting. However, statefulness, when present, has a clear purpose: well designed stateful methods can achieve superior theoretical guarantees, and this is the main reason why states are useful. If the authors tested again stateful methods in the regime when states are useful (e.g., states in SCAFFOLD help to reduce client drift, which leads to better theoretical rates in certain regimes; there are dozens of examples like this), I doubt their stateless approach could win. Of course, once a method is applied outside of the scope in which it is theoretically shown to be better than competing approaches, all bets are off, and the empirical behavior can go both ways. I appreciate though that in the setting considered by the authors, storing states is not practical. The authors then proceed to a reformulation of the cross device FL problem in the form of (2), but do not make any attempt to explain what is lost by using this reformulation. Clearly, solving (1) and (2) are two very different tasks. The connection provided between (2) and (1) is intuitive only. I believe one can think of simple counterexamples that show that by solving (2), one does not approach the solution of (1) in any reasonable way. In this sense, their reformulation is a heuristic again. It may work in some regimes, but it is not clear what regimes these are. Some theory is sorely needed here. In totality, the authors provide a heuristic which may or may not work, depending on the problem being solved. This reduces its usefulness / applicability in regimes/applications beyond the ones studied in the paper. Having said that, the approach seems to be useful for the application at hand. The connection of (2) to MAML is helpful. However, outlining this connection alone does not mean that by solving (2) one gets any close to solving (1). In that sense, the authors could have well just started by saying that what they want to solve is (2) to start with. However, why should one formalize a FL problem in the form (2)? Say we can solve this problem exactly. What does it say about the generalization properties of the model we learn that way? Why does this formulation have any connection to generalization in the first place?

In summary, I believe this is a solid empirical work, proposing a certain heuristic approach to address certain practicality issues encountered in cross device FL models of a certain type (while clearly the approach is model-agnostic, it is not clear at all it makes sense to use it for any model type - some models might not be well trainable using alternating minimization steps). 

I think this paper should be seen as borderline. I tend to accept it as the reviews are largely positive. But at the same time, I am very familiar with the related literature, and my own reading of the paper would lead to a score of 5. I will leave the final decision to SAC.

AC
Decision: Accept (Poster)