Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed FedRolex, a model-heterogeneous federated learning framework that breaks the constraint of standard federated learning based on homogeneous model. The authors proposed a rolling technique to extract heterogeneous submodels from a shared global model, which has been shown to achieve much better performance in comparison with other model-heterogeneous federated learning methods such as Federated Dropout and HeteroFL in both high and low data heterogeneity levels.
strengths_and_weaknesses: Strengths:
- The authors did a good job in reviewing the existing works, especially the ones that are recently published. 
- This is a timely and important topic to work on, and the authors have demonstrated that a simple submodel rotation technique can significantly bridge the gap between model homogeneous and model heterogeneous settings.
- The work has a potential to greatly improve the fairness of federated learning. 

Weaknesses:
- The primary weakness of the work is that although the work claims that the proposed FedRolex could train large server models, the model studied in the current version is ResNet18. The work would be much more convincing if FedRolex could train modern large models such as Transformers.

questions: -	Would FedRolex work on modern large models such as Transformer?
-	Can the authors elaborate more on the concept of client drift to make the paper self-contained?

limitations: The authors discussed the limitations in section 5.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors study the model heterogeneous problem in federated learning. The authors propose a simple but effective method, which adopts a rolling window to generate small sub-models for local training. Extensive on real-world datasets verify the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
1. The model heterogeneity problem studied in this paper is important for the applications of federated learning in real-world scenarios.
2. The proposed method is simple to implement.
3. This paper is well-written and easy to follow.

Weakness:
1. The proposed method can only be used to train CNN-based models and is difficult to be applied to train other SOTA models, such as transformers.
2. The proposed method reduces computation costs for low-resource clients by only updating a small part of model parameters in a training round. The proposed method still needs different clients to train an intelligent model with the architecture. However, in some scenarios, it may be difficult for the low-resource clients to save and load the model that can be trained on the rich-resource clients. It seems that the proposed model cannot tackle such scenarios.
3. The computation resources in the rich-resource clients seem not be fully exploited by the proposed method.
questions: Refer to the weakness.
limitations: Refer to the weakness.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors target an important problem in federated learning (FL) and propose a simple partial training (PT)-based technique called FedRolex to enable model-heterogeneous FL. Compared to existing methods, the core innovation in FedRolex is a rolling sub-model extraction scheme, where the sub-model is extracted from the global model using a rolling window that advances in each communication round. As such, parameters of the global model are evenly trained other than trained either randomly or statically in existing methods. This simple innovation has been demonstrated empirically to be the key factor that contributes to the superior performance of FedRolex over state-of-the-art methods. 
strengths_and_weaknesses: Strengths:
- The review of the existing literature is solid. Table 1 summarizes the key differences between the proposed technique and existing works across a number of important dimensions. This is very helpful to understand the contributions of the paper.
- The proposed technique is simple yet powerful. Such simplicity helps the readers to factor out tricks that are not essential.  
- The experiments are designed with careful thought and the conclusions drawn from the experimental results are quite exciting, especially enhancing the inclusiveness of FL in real-world distribution. 

Weaknesses:
- The experiments are limited to vision datasets (i.e., CIFAR-10, CIFAR-10). It is not clear whether the proposed technique can generalize to other data types such as languages.
- The cohort size considered in the experiments is quite small. In practice, a much larger cohort size is normally incorporated in cross-device FL.

questions: - Can the proposed technique generalize to other data types such as languages and the corresponding language models?
- Can the proposed technique generalize to a larger cohort size which is more practical in real-world deployments?

limitations: The authors discussed the limitations in section 5.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose to sample the sub-models in a rolling way and find the simple method improves the performance of the global model compared with baselines.

strengths_and_weaknesses: Strengths:
* Overall, the paper provides an interesting perspective for understanding the limitations of HeteroFL and Federated Dropout. 
* The authors not only evaluate the PT-based methods but also compare **FedRolex** with KD-based methods.
* In the appendix, the authors give statistical analysis to further analyze the proposed method.

Weakness:
* Compared with Federated Dropout and HeteroFL, FedRolex only modifies the sampling method for Federated Dropout. The idea is not novel enough.
* The experimental results do not list the performance of the simplest model in FedRolex. For the heterogeneous methods, the performance of each group should be listed.
* The authors only provide the statistical analysis to prove that the index is chosen equally randomly, the convergence analysis is not provided. 
* The authors should provide the communication cost and computation overhead to reach the target accuracy.
questions: * KD-based baselines perform much worse than PT-based methods on CIFAR-100, could the authors provide the details of the chosen proxy data?
* Though sampling the sub-network in a rolling way outperforms baselines in the given experimental settings, it is not clear whether FedRolex outperforms baselines with a higher client participation rate. 

limitations: This paper does not have a limitation subsection. But I don't think the paper will have potential negative societal impacts.
review_title: null

Meta-review: metareview: To better handle the case that each client is with heterogeneous device resources, this paper presents a model-heterogeneous federated learning algorithm FedRolex. FedRolex rolls the submodel in each federated iteration, in order to train the parameters of the global model on the global data distribution. Experimental results show FedRolex outperforms other model-heterogeneous baselines. Ablation studies on submodel rolling show it is an effective technique. 

However, this paper suffers from several limitations. Firstly, it remains unclear why FedRolex can significantly outperform Federated Dropout and HeteroFL, since they are only different in sampling methods. Secondly, after federated learning, the low-end devices still can only use a sub-model. Will it benefit?
confidence: Less certain
Decision: Accept