Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work studied masked federated learning (FL) for device-heterogeneous settings. The theorems on the convergence of masked FL revealed the limitation of existing masking strategies that induced biases to the model convergence. Therefore, the authors propose to optimize the per-client masks on the server. Empirical results demonstrate that the proposed method can improve multiple existing masked federated learning methods.
strength_and_weaknesses: Strength
* The method is principled and motivated by the theoretic analysis of convergence. It is novel that the authors show the bias induced by heterogeneous masking. The finding reveals the critical upper bound of empirical studies of masked federated learning so far.
* Because of the general non-convex and masking-agnostic assumptions, the method can be integrated with different masked federated learning methods, like HeteroFL, Split-Mix FL, etc. 
* The experiments also demonstrate that optimizing masks could improve different baselines. Therefore, the theorems and methods are generalizable and could be impactful in device-heterogeneous settings.

Weakness
* The method is motivated by the bias on the theoretic convergence bound. But the intuition behind the bias is not well-explained. The authors may elaborate on why the bias occurs. Instead of attributing the bias to the masking, it should be discussed how the bias is related to the heterogeneity. If all clients follow the same mask, how will the bias be? If all clients follow non-overlapped masks, will the bias be maximized? When the mask makes the bias minimized if can all clients afford the same number of parameters?
* Authors provided two techniques for solving (10a-d). More details about how to choose one of the two techniques should be provided to ease the practice.
* How does the method work on feature non-iid datasets? Since Split-Mix FL has shown that the HeteroFL would perform poorly on feature non-iid datasets, the Split-Mix FL based evaluation should be conducted on feature non-iid datasets.
clarity,_quality,_novelty_and_reproducibility: Quality: The method has been compared to multiple baselines under two datasets. It would be better if the authors could provide more datasets.
Novelty: The proposed method is novel, and the insights are also novel through the lens of convergence.
Clarity: The writing could be improved by making the formulations more concise and by elaborating on theoretic results. Some notations should be more meaningful. For example, when *the normalized mask* (k_n)_l is introduced, the meaning of the term should be explained, in addition to its formulation.  Lemma 1 is not explained. Especially how it is used for proving Theorem 1? In Theorem 1, the meaning of each term should be explained more. 

Minor issues with the clarity.
* The heads model should be head models.
* In the first sentence of the 2nd paragraph of Page 2, "may lead to a bias in the obtained solution of the learning model," is confusing. I suppose the bias should be on the convergence error rather than the model itself.
summary_of_the_review: The paper provides a novel insight into masked federated learning, which should be valuable for device-heterogeneous federated learning. Some issues on the clarity and experiments should be addressed during the rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author proposes to replace the random mask in FedBABU with a masked optimized for the device heterogeneity. The proposed masking is layer-wise so that all parameters in a layer will be frozen or not, based on the formulation Eq(10). Empirical evaluation shows better performance, less trainable parameters and FLOPS.
strength_and_weaknesses: Strengths:
1. The motivation to let devices with more computational resources to do more training makes sense.
2. There are empirical benefits of the proposed method regarding the accuracy and FLOPS.

Weaknesses/Questions:
1. How is \phi_g updated?
2. Can you show and compare the training curves?
3. Can you confirm that among those devices with lower computational capacity, a higher computational capacity leads to more trainable parameters?
4. In Eq (1), the \theta_n in f_n(\theta_n) is the concatenation of w and \phi. However, in section 4 the arguments becomes w, contradicting the previous definition. What are the definitions of F_n and f_n exactly? Why theorem1 is not affected by data heterogeneity?
5. In remark 1, minimizing upper bounds does not mean it is the best way.
6. There is a lacking in the empirical validation for the benefits (e.g., time) of the proposed method when applying to heterogeneous devices.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The novelty is a little bit incremental given the existing work FedBABU.
summary_of_the_review: The motivation of applying customized mask to FedBABU seems valid, but it is not surprising to see the empirical improvements. There are multiple aspects to further improve the paper as stated in weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a specific masking strategy on each client that leads to two outcomes; 1) adhering to the computational constraints of each client and 2) improving the personalised model performance. Instead of prior works that “zero out” the masked parameters, this work instead “freezes” the masked parameters to the values communicated by the server. While strictly speaking this leads to less efficiency gains, the benefit is that each hardware constrained client can maintain the flexibility of the server model, while still being able to contribute in training (a part of) it. To optimise the masking assignment, the authors formalise it as a constraint satisfaction problem that targets to minimise the theoretical convergence bound bias coming from masking. As the latter is an NP-hard problem, the authors propose a specific approximation procedure that seems to empirically work. Finally, the improvement to the personalised model performance comes from also adopting the FedBABU strategy, i.e., splitting the learning of the body and head of the network; the head is frozen and kept local while training the (masked) global body and when the body has converged, the entire network is fine-tuned locally with the available data samples. The authors perform extensive experiments and ablation studies to show the performance of PerFedMask in various settings. 
strength_and_weaknesses: Strengths
- Extensive experimental evaluation
- Theoretical analysis of convergence

Weaknesses:
- Server side optimization problem for masking does not consider FLOPs directly (which is a better proxy for computational cost, e.g., convolutional layers are more computationally expensive that linear ones, however the latter have more parameters) 
- Incremental; mostly a combination of FedBABU + freezing methods (such as CoCo-FL)
- No indication of uncertainty / error bars in the results (so unclear whether the gains in some cases are significant) 
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear and relatively straightforward to follow. One thing that could be improved in the exposition is Eq. 1; it is not entirely correct for FL, as this objective can be solved without the devices communicating altogether since the optimization problems are independent. FL seeks consensus between the local models, which is what correlates the optimization problems and creates the global model. I would advise the authors to update it. Other than that, I believe that there is reasonable information to be able to reproduce some of the results, although I would have liked some more information on the optimization specifics for the SCA algorithm implementation (e.g., an algorithm, hyperparameters it have, etc). 

In my opinion, the main drawback of this work is novelty. As far as I understand, the personalization component is just an application of FedBABU whereas the freezing part comes from the likes of works such as CoCo-FL. The novelty of this work thus seems to lie on the convergence analysis which leads to the uncovering of the convergence bound bias and the server side algorithm to fix it. Having said that, there is no comparison against the heuristic strategy proposed at CoCo-FL so it is hard to tell its significance compared to prior art. 

Finally, some more questions from my side:
- It is a bit weird that in Table 3, more finetuning steps can be harmful. Is it due to overfitting on the local dataset that this happens? 
- What is the importance of freezing the local heads on each client? How does the method perform without the local head freezing?

summary_of_the_review: Based on all of the above, and primarily due to my novelty concerns, I cannot recommend acceptance. This work could benefit by better positioning in terms of works such as CoCo-FL and by making more clear the improvements that it brings to the table.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an extension of the personalized federated learning algorithm FedBABU that adds masking vectors associated with each device in order to mitigate device heterogeneity. The contribution over prior masking approaches to coping with device heterogeneity in the literature is that the authors show a bias term in the convergence bound introduced by the use of masking vectors and optimize the masking vectors to address this bias. Extensive experimental evaluation is provided supporting the performance of the method in different settings.
justification_for_why_not_higher_score: One reviewer was a marginal accept.
justification_for_why_not_lower_score: One reviewer was a strong accept, and the other's premise for having a marginal accept is that the authors did not compare to/discuss a work that is actually concurrent with the submitted work.
Decision: Accept: poster