Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Practical federated learning (FL) needs to handle heterogenous resource constraints among participants. Existing solutions mostly rely on pruning a global model in each client, which would cause uneven contributions to the global model weights among different clients. The paper proposes a mechanism that factorizes the global model weights into two low-rank tensors, where one of the tensors is unified across all the clients (called neural basis) and the other tensor varies according to clients’ resource. The paper also proposes a regularization term to encourage the basis vectors to be orthogonal with unit term. The evaluation shows the proposed mechanism outperforms two existing solutions, HeteroFL and FjORD.
strengths_and_weaknesses: Strengths
-------------
1. The paper proposes a reasonable solution to an important problem in FL.
2. The paper proposes a mechanism that generalizes pruning-based solutions to this problem.
3. The proposed regularization term is interesting.

Weaknesses
----------------
1. The paper misses important baselines in this area. Specifically, the paper should also compare its solution against LotteryFL (Li et al., 2020), FedHM (Yao et al., 2021), and FedMask (Li et al., 2021). These three solutions aim to address the same problem, and they report better accuracy than the paper. 
2. The novelty is incremental. The idea of using low-rank factorization to address system heterogeneity has been proposed by FedHM (Yao et al., 2021). The general idea of using low-rank factorization for FL was proposed by FedDLR (Qiao et al., 2021) and FedPara [47]. While the detail of factorization might be different, it is unclear if the proposed factorization technique is better than other existing ones.
3. There is no analysis on computation and communication complexity. Similarly, there is no convergence analysis. The paper should provide an analysis of the proposed mechanism, and it would be much better to do the same against related solutions in this space.

questions: 1. Can you provide more evidence to support the claim that the shared neural basis enables more knowledge sharing than pruning among clients? If the values in the coefficient matrix are small, isn’t it effectively similar to pruning?
2. How will the proposed method work if each device has a unique complexity level? Will the training still converge?

limitations: The paper does not provide meaningful discussions on its limitations, and it would be good to see the discussion on the following aspects:
1. How will the proposed mechanism work with fine-grained complexity levels?
2. What are the privacy implications of sharing the coefficient matrices?
3. What if the resource constraint changes within a communication round (e.g., other applications running)?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents the design of All-In-One Neural Composition, which can formulate networks at different complexities with one unified neural basis that's shared among clients. FL systems have suffered from the problem of system heterogeneity where the service providers would either sacrifice model capacity or data accessibility. This paper provides the FLANC framework that provides systematic support to train complexity adjustable neural networks and evaluated on a wide range of statistical data heterogeneity and system heterogeneity. 
strengths_and_weaknesses: Strength: 
1. Proposed a runtime method to construct the compressed model instead of applying after the training process compared to other low-rank techniques.
2. Conducted extensive experiments to show the effectiveness of the proposed method. 

Weakness:
1. Didn't include error bars in the evaluation results. The results reported may not be statistically confident without any error bars. The authors could add randomness experiments on some small scale experiments. 
questions: 1. In the evaluation section, the authors varied the $p$ to simulate heterogeneous systems. It's good to also include some real cases including the exact CPU/GPU that might result in the different $p$ (i.e. the limitations of computation power on each device to result in the limitation on model size).
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents FLANC, a model decomposition method for the heterogeneous devices in Federated Learning (FL). The basic idea is to change the model size/complexity by varying the ratio of channels. 
strengths_and_weaknesses: Strengths
1.Better supporting of heterogeneous devices (e.g., IoT and mobiles) is an important research topic in FL.
2.FLANC is generally applicable to other models and datasets. 
3.FLANC shows promising results compared to HeteroFL and FjORD.
4.This paper is well-written and easy to follow. 

Weaknesses
1.Only the metric of accuracy is evaluated. It is better to consider other important aspects of FL, e.g., communication overheads. 
2.Evaluation results are from one-run experiment only, which is less convincing.
questions: How does FLANC perform with regards to other FL aspects such as training rounds and communication overheads?
limitations: None
review_title: null

Meta-review: metareview: This paper proposes a method to cope with heterogeneous computation capabilities of clients in federated learning. The initial reviews were positive, but some the high-score reviewers indicated low confidence. The following concerns were raised.
1. Limitations in the experimental baselines
2. Lack of theoretical justification for the convergence and the communication/computation complexity
3. Somewhat incremental novelty
The authors put in significant effort to address the concerns during the rebuttal which led to a slight increase in the average score. Therefore, I recommend acceptance of the paper. I strongly encourage the authors to take the reviewers' constructive feedback into account when revising the paper. 
confidence: Less certain
Decision: Accept