Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a weight shared training framework for FL that maintains a primary network at the server and distributes subnetworks to clients. The authors propose algorithms for efficient selection of the subnetworks and aggregation. This allows efficient federated learning and deployment of the models. 

strength_and_weaknesses: Strengths

- The proposed problem (not directly studied in other works) is relevant
- the approach is relatively straightforward and elegant
- The results are convincing and well analyzed on the presented datasets

Weakness
- Although the datasets are standard for FL the authors primarily focus on variants of CIFAR-10 and convnets. It would be beneficial  to see the generality of the approach to other architectures (e.g. MLP's and ViT's) and more distinct datasets (e.g. text data). 

clarity,_quality,_novelty_and_reproducibility: Clarity - the paper is overall clear 
Quality - the experiments are of strong quality on the presented datasets
Novelty - The paper effectively combines a number of existing ideas for a relevant proposed problem
summary_of_the_review: The paper provides an interesting and relevant problem setting and an effective algorithm for this setting. Validation on additional datasets and broad model classes would  increase the strength of the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in a federated fashion. The paper also proposes a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time. The paper further proposes a mechanism for subnetwork weight aggregation with variable overlaps mitigating intra-network interference. The proposed approach can take an important first step towards co-training model families in a federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction.
strength_and_weaknesses: Strength:
1) The proposed concept of 'training a global family of models' seems to be very well-motivated, novel and appealing.
2) The proposed method clearly outperforms the baseline, independent SOTA training of DNN subnetworks (without weight sharing) using FedAvg.

Weakness:
1) The concept of 'training a global family of models' is well-motivated by the need in on-device inference in real-world applications. However, by multiple model variants, I assume the variances of the performances in each devices can be much more significant, compared to the single model in previous FL approaches. However, only achieving some level of accuracy is not enough to release a real product (e.g, many human evaluations of the user experience are needed). With many different models in many devices, it will incur a lot of efforts to track and maintain these models. Thus, I am not sure whether the concept of 'training a global family of models' is practical in real-world applications.
2) Currently there is only one baseline evaluated in the paper. It is understandable that the paper proposes a very novel setting ('training a global family of models'), so the comparable baselines are limited. However, we may still align the setting and compare the proposed approach to existing FL approaches? For example, assume there are 10 models. For each client, we can train a suitable model by existing FL approaches on all clients with similar (or less) devices constraints. Although there are some benefits of 'training a global family of models', these experiments can still provide some insights about whether the proposed solution outperforms SOTA FL approaches (or still has a gap) in FL problems (e.g., non-iid).
clarity,_quality,_novelty_and_reproducibility: The paper is well-motivated and the proposed solution has good intuitions. Comparisons to other baselines in some aligned setting can be considered.

The paper is clearly presented and very easy to read.

The paper studies a very novel problem setting 'training a global family of models'. However, maintaining many models in real-world applications can introduce a lot of complexities. Not sure whether it is practical in the real world. 
summary_of_the_review: The paper studies a very novel problem setting 'training a global family of models'. However, maintaining many models in a real-world applications can introduce a lot of complexities. Not sure whether it is practical in the real world. Currently there is only one baseline evaluated in the paper. It is understandable that the paper proposes a very novel setting ('training a global family of models'), so the comparable baselines are limited. However, we may still align the setting and compare the proposed approach to existing FL approaches.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a weight-sharing approach for simultaneously training multiple models with different architectures to satisfy different deployment needs in a federated setting. Their method combines existing techniques for weight-sharing with new approaches for distributing architectures and averaging parameters. The resulting algorithm is evaluated on three vision datasets. Note that I have reviewed this paper before and parts of my review are taken from my previous evaluation.
strength_and_weaknesses: ### Strengths:
1. The problem setting is well-motivated and a practical algorithm here would be valuable for applications of federated learning.
2. The authors conduct ablation studies of their solutions for handling interference due to weight-sharing.
3. Experimentally there is a comparison to reasonable baselines (iFedAvg) and the authors demonstrate improvement by the method in obtaining a superior accuracy-complexity tradeoff. There is also evaluation of the method at different heterogeneity levels, albeit synthetically generated via classes.

### Weaknesses:
1. The main weakness of the paper remains the limited evaluation. The proposed algorithm is evaluated on three closely related datasets, all containing natural images with the same size. Most FL papers will at least include a text application alongside evaluation on image data, given that text is one of the dominant applications of FL. There is not a strong reason to know that the proposed method will directly extend beyond image data, especially since the non-FL work this paper is based on (OFA) focuses on vision. I believe to be impactful the paper would at least have to evaluate on one of the text datasets in LEAF (Caldas et al., 2019), or on more recent datasets based on Reddit or StackOverflow.
2. It is unclear if code will be provided for reproducibility.

### References:
1. Caldas, Duddu, Wu, Li, Konecny, McMahan, Smith, Talwalkar. LEAF: A benchmark for federated settings. arXiv 2018.
clarity,_quality,_novelty_and_reproducibility: Clarity: reasonable.  
Quality: the experimental evaluation remains limited to similar datasets that are the same or closely related to the non-federated variants of the work.  
Novelty: while there has been some work on hyper parameter tuning in FL, to my knowledge the work introduces the first federated OFA method.  
Reproducibility: it is unclear if code will be provided.
summary_of_the_review: The problem addressed is well-motivated and the authors obtain reasonable empirical results on the datasets considered, but the evaluation remains limited to CIFAR-like datasets, which makes the applicability to practical FL unclear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a federated learning method that supports training multiple models in various sizes simultaneously without heavy costs and performance loss. The core idea is weight sharing, where different models can share common knowledge. Experimental results show the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:
1. The problem studied in this paper is interesting.
2. The proposed method seems to be effective.
3. The concept of temporal-spatial subnetwork distribution is interesting.

Weaknesses:
1. The relations between this work and existing client-heterogeneous FL works [1-3] are not clarified ([2] and [3] are also not included in this paper). The authors mention that "Diao et al. (2020); Lin et al. (2020a) provide a single global model to every client", which is quite confusing because Diao et al. (2020) distribute different models to different clients. In fact, these methods can choose to distribute models in different sizes to different clients in different rounds. Moreover, [1] and [3] also consider weight sharing mechanisms. Thus, it is not clear what is the difference between the scenario in this paper and existing works, as well as the unique advantage of the proposed method.
2. The authors state that "To our knowledge, there does not exist an FL algorithm which trains a global family of
models using weight-sharing". This may not be the case. Thus, more comparisons with existing baselines are needed.
3. The methodology section is somewhat too brief, and many details are not well clarified. 


[1] HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264.
[2] FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization. arXiv preprint arXiv:2111.14655.
[3] No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices. KDD 2022.
clarity,_quality,_novelty_and_reproducibility: The clarity of this paper is not so satisfactory. The technical quality, novelty, and reproducibility are moderate.
summary_of_the_review: This work is interesting but its insufficient clarity is a major concern. The technical contribution seems to be not very big. Thus, my recommendation is "marginally below the acceptance threshold".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: - Summary

The paper proposes a weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in a federated fashion. The paper also proposes a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time. The authors propose algorithms for efficient selection of the subnetworks and aggregation. This allows efficient federated learning and deployment of the models. The proposed approach can take an important first step towards co-training model families in a federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction.

- Strengths:

1. Reviewers found the idea of 'training a global family of models' well-motivated, and appealing.
2. Experiments show favorable performance for the selected (limited) tasks, supporting the algorithmic claims.
3. After the reviews, the authors have revised parts of the paper, by including additional experiments and code, along with explanations to reviewers' points.

- Weaknesses:

1. Reviewers still find the selection of the experiments provide partial answers to the efficacy and value of the algorithms. All reviewers appreciated the fact that the authors spent time to include experiments on the PTB dataset (which is a different application than the one included in the original version of the paper). Reviewers found this choice i) partially answers the concern of limited experiments (a reviewer mentions that in FL scenarios, more text-based datasets are available), ii) does not provide enough evidence that the methodology generalizes to other datasets + other tasks than the ones originally selected. 

- Recommendation:

First of all, the authors should be stayed assured that there were discussions regarding the paper, and the area chair read carefully i) the paper, ii) the reviews, and iii) the discussions after the rebuttal. We acknowledge that the paper addresses concerns raised, but the reviewers have remaining concerns:

1. We recommend the authors include a more thorough experimental validation for their methodology: given that their method lies at the intersection of NAS and FL applications, including experiments that are common in both fields should be sufficient to answer any questions about the empirical validation of the method. 
2. The paper lacks theoretical justification: while this is not necessary (and in some cases almost impossible), the experimental part should be complete, supporting further the point above. Experiments is what is only left for a reviewer to understand whether there is value in the methodology proposed; 

We recommend the authors to consider these (common over all reviewers) concerns to revise their paper; we strongly support for a resubmission to a near-future ML conference.

justification_for_why_not_higher_score: While the authors propose a method that is considered novel by a subset of reviewers, all reviewers found the evaluation limited and incomplete. Accepting the paper in this form is not possible, unless a major revision is performed. We suggest a more thorough empirical evaluations, including applications both in NAS and FL scenarios.
Decision: Reject