Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the important problem of quantized personalization in Federated Learning. The paper introduces knowledge distillation across heterogeneous clients requiring different precision/architectures. To that effect a loss function is formulated, and the algorithm QuPeD is proposed. Theoretical converge of QuPed is provided, along with experimental validation. 
main_review: The difference between model x_i and quantization center c_i should be emphasized. I would like to see an example of such quantization with particular values of c_i. 

Motivate the choice of R(x,c). l_1 norm is used, with is suited for proximal algorithms used in QuPeD, but such choice should be motivated well. what not l_2. Is there any sparsity concerns?

As mentioned by the authors themselves, the bounded gradient assumption is too strong.

Further explanation needed on Assumptions A.4 and A.5. Appendix A does not provide any values of the lipschitz/smoothness parameters. Since, the authors are using sigmoid/tanh, I would like to see the parameters derived for such functions. Only A.6 is justified in the appendix, by connecting it to Fallah, et.al. Moreover, A.7 seems very strong. Overall I believe that the set of assumptions are too strong, and more justifications are required. Either show standard instances where these thing hold, or argue that these are standard by citing papers using the same assumption.

Theorem 2 is very hard to parse. There are several parameters, and scaling of those are not clear, so the result right now is non-intuitive. Also, after the theorem, I would request the authors to add implications/remarks about the theorem, maybe get back to some standard results.

I have a question about the 1/\sqrt{T} rate on gradient-norm-squared. In standard distributed learning, with full gradient (which is same as this paper), a 1/T rate on gradient norm squared can be obtained (see https://arxiv.org/pdf/1911.09721.pdf, https://arxiv.org/pdf/1803.01498.pdf with no Byzantine setup). I would request the authors to comment on this. Why is the rate slower? Is this because of the proximal operator? Is it optimal? Why do we need a proximal operator if it hurts the rate of conergence?

limitations_and_societal_impact: Mentioned briefly. For further limitations, see above for detailed comments.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a QuPeD framework personalized FL via knowledge distillation (KD) to alleviate both data and resource heterogeneity. The authors first present a relaxed formulation and an algorithm under the centralized setting. Next, they extend the previous formulation to FL setting to enable different model dimensions and precision requirements for clients. The authors further provide detailed convergence arguments and empirically evaluate the proposed methods on practical datasets.  
main_review: Merits:
-The paper considers both data and resource heterogeneity, which is a fundamental issue in FL. The paper is well written and organized.  Algorithms and their corresponding convergence results are clearly interpreted. Empirical evidence is sufficient and the results seem promising. 

-The use of KD is natural (but not new) as each client may hold models with different dimensions and precision. To formalize the use of KD as an optimization problem is original and important as it facilities the theoretical characterizations. 

-The implications of the coupled effect of data and resource heterogeneity on the convergence rate from Theorem.2 are interesting.

Weakness/potential improvements
-The client's loss function (i.e., Eq.2) seems contrived since it simply combines a compressed objective and an integrated KD loss. Is there a principled way for each client to choose $\lambda$ and $\lambda_p$? 

-Further evaluations on more challenging tasks e.g. Stack Overflow are needed.

limitations_and_societal_impact: Yes, the authors have explicitly addressed the limitations (appendix) and potential negative societal impact (last section in the paper).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies a quantized and personalized algorithm for faster inference and adapting to data/system heterogeneity in federated learning.
Personalization is facilitated by knowledge distillation. Model compression is achieved by end-to-end training to optimize for quantized models and quantization parameters.

main_review: 
Pros:

* The problem of jointly consider personalization and compression is interesting.

* In terms of quantization alone, the formulation (6) is novel to me (to the best of my knowledge), and demonstrates improvements compared with (4) or (5), although it is a direct combination of existing works.

* The observation from Theorem 2 that clients with more representative data can tolerate more aggressive compression is interesting.

* Experiments clearly show the effectiveness of both the quantization method, and the main contribution QuPeD. QuPeD also outperforms existing personalization methods, while allowing for heterogeneous model architecture and precision.

Concerns:


* (a) One bottleneck of federated learning is the computation and memory constraints of on-device training. The proposed method only reduces inference time, but adds much training overhead, which is very expensive. Other directions of compression methods (beyond teacher-student knowledge distillation) that can reduce training cost (computation or memory) while producing a smaller model could be more desirable for federated learning, for example, training factorized layers. (b) If only considering inference, there are other natural and simpler baselines, such as first performing regular personalization training (via KD or other techniques) and then pruning the models as a post-processing step. How does this work empirically?

* The convergence analyses do not support partial client participation.  

* I think the experiments are lacking results on text data. As the gradient distribution of image data and text/language data are very usually different., it not clear if the proposed method could still yield improvements on the new type of data, and how much. 

* Both the quantization method and the personalization method (KD) have been studied by previous literature, and the proposed optimization objective (2) combines the two ideas. I think discussing how potentially these two parts interact with each other (i.e., how to design better personalization methods that are inherently suitable for quantization) would be more interesting.

My major concerns are the first two points.


## update
I have updated my score (see detailed comments in my response below). 
limitations_and_societal_impact: The authors have discussed the limitations of their work, and I don't think there is potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel optimization objective to address the heterogeneity issue in the setting of federated learning. The heterogeneity comes from the fact that different local clients may have different data distribution and resource constraints. The proposed method allows every client to learn a quantized model with possibly different sizes. Locally, each client optimizes for a loss function containing two major terms: 1) loss of the local quantized model; 2) the distance between the quantized model and the global model.

The authors propose an alternating proximal algorithm, and provide convergence analysis under standard assumptions. The authors then empirically show that the proposed algorithm outperforms related works on CIFAR-10 and FEMNIST.
main_review: Strength:
- The paper is logically well structured. The design idea behind the proposed loss function is clearly stated in Section 2. All the theoretical results are followed by remarks that explain the implications, which greatly improves the readability of this paper.
- The idea of allowing every client to learn a personalized quantized model of different size seems new in the federated learning literature. 
- The paper provides convergence analysis of the proposed algorithm.

Weakness:
- This paper is motivated by the cross-device federated learning (as stated Section 1), but the proposed algorithm may be difficult to use in the cross-device setting. Federated learning has two settings (see Table 1 in [1]): 
   - Cross-device: in this setting, a client can be a mobile device (e.g., smartphones), and there can be millions of clients in total, only a few of them (say hundred) participate in each training round, and each client usually participates *at most once* in the entire training session.
    - Cross-silo: in this setting, a client can be an organization (e.g., banks or hospitals), and there can be hundreds of clients in total, and it is ok to assume that all clients are available and participate in every round.

The proposed algorithm (Algorithm 2 in the paper) requires that all clients participate in every training round, and that every client maintains a local model and is responsible to update and carry it from this round to the next round. Furthermore, the proposed algorithm does not support inference on new clients (i.e., clients that do not appear in the training). These constraints make it difficult to use the proposed algorithm in the real-world cross-device federated learning applications.

- The paper does not compare with a natural baseline algorithm FedAvg + fine-tuning [2], i.e., each client fine-tunes the global model (learned by FedAvg) locally to create a personalized model. This simple algorithm can be used in both cross-device and cross-silo settings. If needed, one can add an extra quantization step after local fine-tuning to create a quantized personalized model.

- The hyperparameters are not properly tuned for the baseline algorithms compared in Section 5. For example, in Section 5, the authors said that “For all methods, if applicable, we set τ = 10 local iterations, initialize learning rate of 0.1...”. It is unclear why the authors used the same learning rate for all baseline algorithms, and it would be more convincing if the authors can tune the learning rates (and also epsilons if adaptive optimizer like Adam is used) for different algorithms. 

[1] Kairouz et al., “Advances and Open Problems in Federated Learning”, arXiv: 1912.04977, 2019.

[2] Jiang et al., “Improving Federated Learning Personalization via Model Agnostic Meta Learning”, arXiv: 1909.12488, 2019.
limitations_and_societal_impact: Yes, the authors pointed out the potential societal impact in Section 6.
review_title: null

Meta-review: The paper discusses how to properly design personalized models in Federated Learning. A novel aspect of the paper is to  introduce knowledge distillation across heterogeneous clients, and require that different precision/architectures can be used across different network users.  Theoretical converge of the proposed QuPed is provided, along with extensive experimental validation. Overall the paper is well-written, and the authors have made some interesting observations, such as  how data and resource heterogeneity can affect the convergence rate, and that clients with more representative data can tolerate more aggressive compression are interesting to the community. On the downside, the KD for (personalized) federated learning has already been studied, and the optimization formulation for the proposed algorithm seems to be a straightforward combination of different aspects of the modeling (quantization and KD). 
Decision: Accept (Poster)