Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically studies the federated learning setting on super-net training.
This paper considers both directions simultaneously to overcome the data privacy issue and data heterogeneous issue in parallel.
strength_and_weaknesses: Pros:

1. The problem is well-motivated. Existing methods either "train a single global model but keeping each local heterogeneous training data decentralized" or "train an overarching network that supports diverse architectural settings to address heterogeneous systems equipped with different computational capabilities". This paper considers both directions simultaneously to overcome the data privacy issue and data heterogeneous issue in parallel.

2. The paper is well written. The challenges of super-net training are well listed and solutions are provided in Section 3.



Cons:

1. This paper is mainly focusing on the empirical part of federated learning and supernet training. However, it would be great if the authors could provide theoretical evidence on why "parametric normalization"  and "in-place distillation" could help alleviate the data heterogeneity issue from either optimization (e.g., convergence analysis) or generalization (e.g., the generalization analysis in [1]). 


[1] Personalized Federated Learning: A Meta-Learning Approach https://arxiv.org/abs/2002.07948

2. Although sampling based on FLOPs is interesting, I am having doubts about whether this will exaggerate the data heterogeneity issue. For example, lower complexity models are more likely to be trained on devices with less computation ability, and the data will be loss similar on these devices.   

3. For experiments, I think the authors also need to compare with single machine supernet training, i.e., all heterogeneous data are gathered onto a single device and used for training. This experiment could be used as a baseline to show the trade-off between efficiency and accuracy.
clarity,_quality,_novelty_and_reproducibility: The paper is well written. 

Although a number of training techniques (e.g., parametric normalization, in-place distillation) are proposed for better training, the backbone of the method sounds to me like a combination of FedAvg and NAS (neural architecture search), which is mostly an incremental engineering effort. 

Code is not provided, not sure about the Reproducibility.
summary_of_the_review: This paper studies a well-motivated interesting problem, paper is well-written and easy to follow. 

This paper mainly focused on the engineering part of federated learning, however for papers that are working on federated learning, some theoretical analysis on either convergence or generalization is usually expected. 

The proposed complexity-based sampling method might exaggerate the data heterogeneously issue and hurt model performance. 

The authors are also expected to compare with centralized training to demonstrate the trade-off between efficiency and accuracy.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: While federated learning has become a popular paradigm for collaborative machine learning, it suffers from data heterogeneity and system heterogeneity. This work proposes FedSup, a federation framework of supernet training to address both heterogeneities. The key idea is to directly train the subnetworks on the device and introduce parametric normalization and self-distillation to improve the training. Experiments show that FedSup improves global and personalized client model accuracies with better communication efficiency.
strength_and_weaknesses: Strength:
+ this work provides an interesting analysis of similarities and differences between federated learning and supernet training in NAS.
+ extensive experiments evaluate both FedSup and E-FedSup performance compared to conventional approaches (FedAvg etc).

Weakness:
- the analysis of the compounding effect in E-FedSup is weak. In E-FedSup, the system heterogeneity and data heterogeneity are entangled since sub-architecture sampling is biased. It would be better if this work could provide more experiments on the generalizability of the trained supernetworks under E-FedAvg.
clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and easy to follow.
summary_of_the_review: This paper proposes an interesting question on training the supernet of neural architecture search in the context of federated learning. However,  In general, I think this paper is marginally above the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies supernet training in the federated setting. Compared to standard supernet training algorithms, various twists have been made to accommodate the constraints in federated learning systems. The experiments have verified the performance of the proposed method. 
strength_and_weaknesses: Strength: 
- Supernet training in federated learning is an interesting topic. 
- This work can potentially have practical value as it explains the detailed steps of the method. 

Weaknesses:
- The paper describes a lot of the details in the approach. The high level picture is somewhat missing, so it is hard to identify a core novel contribution in this work. It seems the contribution is mainly in the detailed tuning of the centralized supernet training algorithm to the federated setting. I think such a work can be useful, but not very interesting from a scientific point of view. 
- It would be nice if non-obvious findings, both in the algorithm design and empirical results, can be highlighted and discussed in the paper. 
- It is not clear what Dirichlet parameter $\beta$ is used to obtain the results shown in the main paper. 
- I wonder why the personalized model gives much better performance than the initial model, and why the initial model's accuracy is so low. Is there an explanation for this?
- There are only a few plots that compares the proposed FedSup algorithm with baselines. To conclude that FedSup performs uniformly better than baseline algorithms, comparisons with baselines need to be made with multiple datasets. I wonder why the appendix does not include such comparisons for the other datasets. 
- What does O and X mean in Table 4?
clarity,_quality,_novelty_and_reproducibility: Clarity: Some parts need clarification as mentioned above. 

Quality: The paper has weaknesses as mentioned above.

Novelty: The novelty is somewhat limited as most contributions are in the detailed steps that appear to be more like tricks instead of something fundamentally new. 

Reproducibility: The code has been provided which is good, although I didn't try to run it. 
summary_of_the_review: The topic is interesting, but the novelty is somewhat limited, and overall the paper should have more insightful discussions on both the technical contributions and empirical results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to improve deployment of deep networks across system heterogeneities. In this regards, author proposed a FedSup framework combining supernet and federated training. The crux of the method is to directly train subnetworks on clients with in-place distillation and substituting batch norm with parametric normalization. On experiments with small datasets, FedSup improves global and personalized client model accuracies for same communication cost. The paper received borderline scores from the reviewers and none of the reviewers were excited to champion for the paper. It would be nice to incorporate reviewer suggestions like:

- Presentation could be improved as in current manuscript the high level picture is somewhat missing, so it is hard to identify a core novel contribution in this work.
- Providing motivation/theoretical analysis on why/how the parametric normalization and in-place distillation could help alleviate the data and system heterogeneity issue from either optimization or generalization could benefit the paper.
- Instead if focusing on being a more empirical paper, then a more comprehensive evaluation is needed: in particular datasets beyond CIFAR and baseline algorithms
justification_for_why_not_higher_score: - Presentation could be improved
- Motivation / theoretical justification for why proposed parametric normalization and in-place distillation can benefit data and system heterogeneities is not clear.
- More datasets beyond CIFAR is needed and baseline algorithms for a more empirical focused paper
Decision: Reject