Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes DepthFL, an approach that enables resource-constrained clients to participate in Federated Learning (FL) of a global model that is larger than these devices. DepthFL defines shallower versions of the global model for simultaneously training, whereby each client trains a local model of a depth appropriate for its resources.  DepthFL employs self-distillation of the highly trained shallow layers to address the insufficient training of deeper layers. The experimental results show improved results for both the global model and the shallow local models compared to excluding resource-constrained devices. 
strength_and_weaknesses: The paper addresses an important problem by enabling resource-constrained clients in heterogeneous FL. I found the two techniques of varying depths to match client resources and self-distillation of the deeper layers to be quite intuitive. I also found the evaluation section to be well organized, motivated, and presented. 

The only weakness is that I would have liked to see an evaluation of a scenario where 100% of the clients could participate in training the global model, as that could help to understand the upper bound performance of these tasks. 
clarity,_quality,_novelty_and_reproducibility: I found the paper quite easy to read, the evaluation section was good quality, and overall work is quite novel. 
summary_of_the_review: The paper convinced me that local models of varying depths and self-distillation is an effective way to improve performance and usefulness of FL on heterogeneous clients.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose Depthwise Federated Learning (DepthFL) framework (includes mutual self-distillation)  to ensure that the global model accuracy improves compared to exclusive FL (excluding resource-constrained clients which can not train global models)) when performing aggregation of local models in FL on heterogeneous clients, especially, resource-constrained devices. 
- DepthFL is based on depth scaling in contrast to conventional approaches on width scaling, the latter suffers from degrading accuracy when compared to exclusive FL.
- DepthFL constructs a global model that has several classifiers of different depths. It prunes the highest-level layers of the global model to create local models with different depths based on the client's available resources, thus with a different number of classifiers.
- Each client not only trains the classifiers in its local model using local data but also distills the knowledge  across classifiers at the same time
-  For inference, DepthFL uses the ensemble of all internal classifiers while HeteroFL uses the global model with all channels
strength_and_weaknesses: **Strengths:**
- The application of depth-wise pruning to tackle resource heterogeneity in FL
- Distillation from shallow to large classifier models via deep mutual learning
- The main ideas in the paper are easy to understand for the reader
- Extensive experiments with ablation studies that help understand the intuition behind design choices


**Weaknesses:**
- Related study is missing discussion on layer-pruning methods
- Novelty of the proposed method appears to be limited as the application of Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018) for FL to improve upon the layer-pruning method (InclusivFL). It might be better to improve Section 1 to clarify the novelty.

clarity,_quality,_novelty_and_reproducibility: 1. It is important to provide a related study focussing on generic layer-pruning methods as this paper works along the same lines of depth in NN. Some suggested papers (not exhaustive list) in addition to InclusiveFL (reported in paper) are : 
    + S Chen, Q Zhao Shallowing deep networks: Layer-wise pruning based on feature representations (TPAMI'18)
    + Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz,  Importance estimation for neural network pruning (CVPR 2019)
    + TW Chin, C Zhang, D Marculescu , Layer-compensated pruning for resource-constrained convolutional neural networks
2.  It is recommended to provide the comparison with a layer-pruning method (InclusivFL) in the main body as one of the baseline comparisons, more so, because InclusiveFL uses the same baselines such as Exclusive FL and HeteroFL
3. It would be helpful to highlight key modifications to standard FL in Algorithm 1. 
4. Fig. 1 should be redone to improve the quality and illustration. It is also suggested to include illustrations (and a Table) for width-wise (Hetero-FL) and layer-wise pruning methods (InclusiveFL) to highlight the unique properties and benefits of the proposed depth-wise method DepthFL in contrast to other baselines.
5. It is unclear if the experiments used 100 units of Nvidia RTX 2080 Ti GPUs as FL clients or simulations. More discussion on FL setup is required. 

summary_of_the_review: This paper tackles an important problem of FL on heterogeneous clients. The idea is simple to understand which is to have clients with different resource availabilities have local models of varying capacity ( here, a depth-wise pruned version of the global model). However, the paper is written in a way that shows it as a limited novelty by applying Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018). Moreover, an important extensive comparison with InclusiveFL is missing from the main body (though a section exists in Appendix) which is closest in design to DepthFL and has the same baseline comparisons used in this work. 

However, I will be willing to update my score based on overall discussion and clarifications of the issues raised.

---
Edit: I have read the authors' responses and increased my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a new Federated Learning (FL) approach for tackling resource heterogeneity. In their proposed scheme, local models are scaled up horizontally, and FL clients can choose to learn local models with different depths based on their needs. One novelty resides in their design of the reverse direction knowledge distillation, where knowledge is distilled from shallower layers, which were trained by more clients, to deeper layers. As a tradeoff, each layer is attached by a classifier head.
strength_and_weaknesses: Pros:

+ Techically sound approach towards tackling system heterogeneity in FL.
+ Novelty on reverse direction knowledge distillation; authors also provided empirical analysis on its effects.

Cons:
* Extra computation overhead exists in 1) training multiple classifier heads and 2) self-distillation among layers. These classifiers are also required to be transmitted during FL loops.
* Resemblance to early-exit models: The design logic largely reminds me of the early-exit networks (e.g. [1]). Authors are encouraged to cite work along this line and discuss the difference between the proposed work and early exit models.
* Non-comprehensive baselines: authors mentioned different FL approaches of width-based pruning in the related work section, but only FedHetero has been extensively evaluated as the only baseline. When analyzing the lower performance of FedHeteo compared with exclusive learning, I suggest authors study and discuss more related work that enables split learning in FL, such as [2] and [3] which tackled the issue that exists in FedHetero.
 


[1] BranchyNet: Fast inference via early exiting from deep neural networks.  ICPR 2016.
[2] Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. NIPS 2021.
[3] Resilient and Communication Efficient Learning for Heterogeneous Federated Systems. ICML 2022.

clarity,_quality,_novelty_and_reproducibility: This work is clearly written with moderate novelty.
summary_of_the_review: A technically sound work towards addressing system heterogeneity in FL by layer-wise scaling.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the resource challenge of federated learning. Instead of training the same server model across all clients, it proposes scaling the server model along the depth dimension instead of the width dimension to meet the resource limitation of each client. It also found that distilling knowledge from the shallower part to the deeper part is essential to improving the accuracy of the final model ensemble.
strength_and_weaknesses: Strength:
- This paper address an important issue of federated learning that many clients are constrained by the available resource and cannot train the whole server model.
- This paper shows that depth-wise scaling is better than width-wise scaling for federated learning.
- This paper shows that self-distillation from the shallower part to the deeper part can help improve the final accuracy.
- Using the almost free model ensemble (under this paper's experimental setting) to enhance the accuracy is clever.
- The experiments are extensive and informative, and I appreciate the authors' efforts in designing and running them.

==============================================================================

Weakness:
- I noticed that in Table 2 the accuracy of exclusive learning of DepthFL (FedAvg) is consistently higher than that of HeteroFL. The sub-model of HeteroFL is thinner but deeper while the sub-model of DepthFL (FedAvg) is shallower but wider. Empirically, the former will perform better than the latter given the same model size. These counter-intuitive results make me wonder whether the gain comes from other sources instead of doing depth-wise scaling, such as more intermediate supervision points (like SHeteroFL) or unfair comparison (as stated in the last sentence of the first paragraph in Sec. 4.2).
- Does HeteroFL use the same architecture as DepthFL with the intermediate supervision points? If not, DepthFL is actually larger and more complicated than HeteroFL.
- Missing the comparison to Federated Dropout. HeteroFL sounds like a special case of Federated Dropout. HeteroFL drops the neurons in a fixed order while Federated Dropout drops the neurons randomly. I wonder whether the results will change if HeteroFL drops different 25% for each client and for each round instead of the same 25%.
- The justification of why depth-wise scaling is better than width-wise scaling in the last paragraph of Sec. 4.2 sounds weird to me. The claim is that the worse accuracy of width-wise scaling is due to the fact that 25% clients train classifier 1/4 directly and 75% clients train classifier 3/4 indirectly. I think it is the same for the proposed depth-wise scaling. For example, the first 25% layers are also trained by 75% clients indirectly due to backprop. Why can this justify the accuracy difference?

Other comments:
Because it uses a branchynet-style network to resolve the resource issue, it requires adding multiple intermediate supervision points with decoders or classifiers. If the decoders and the classifiers are heavy, adding them may nullify the saving in the reduced encoder and increase the complexity of the global model. Some example applications are image reconstruction or speech recognition.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- The writing of the experiment part is a little bit confusing and hard to understand. It requires me to give this section multiple passes to kind of grab what the authors want to convey. For example, the terms "global model", "global sub-model", and "sub-model" took me a while to differentiate them.
- Other sections look good to me.

========================================================================

Novelty:
I feel the novelty is kind of limited. There are two main contributions of this paper: 1) propose using depth-wise scaling, and 2) use self-distillation to increase the performance of the model. The first one is the special case of techniques like Federated Dropout and only applicable to networks with branchynet-like architectures, and the second has been widely used to improve network training and the quality of sub-models in one-shot neural architecture search. However, the finding that depth-wise scaling is more beneficial than width-wise scaling under the scope of federated learning is interesting to FL audiences, so I increase the rating for the novelty.

========================================================================

Reproducibility:
Yes.
summary_of_the_review: Per the discussion in the other sections, I think the novelty of this paper is somewhat limited, and the concerns mentioned in the weakness section make me skeptical about the correctness of the claim that depth-wise scaling is better than width-wise scaling. I will be happy to change my rating if the concerns can be resolved in the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
This paper addresses an important issue of federated learning that many clients are constrained by the available resource and cannot train the whole server model. The authors propose Depthwise Federated Learning (DepthFL) framework together with mutual self-distillation to do depth scaling.  DepthFL constructs a global model that has several classifiers of different depths, and prunes the highest-level layers of the global model to create local models with different depths based on the client's available resources. Each client not only trains the classifiers in its local model using local data but also distills the knowledge across classifiers at the same time. 


Strengths: 
+ Relevance of the topic: the problem considered is important and timely. 

+ Technical contribution: the paper uses a self-distillation from the shallower part to the deeper part to improve the final accuracy.

+ Strong empirical evaluation: experiments designed and conducted are extensive and informative. 

Weaknesses:

- Limited novelty and position in the existing relevant literature: Several reviewers have expressed concerns about limited novelty as the proposed method is an application of Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018) for FL to improve upon the layer-pruning method (InclusivFL). Related work is missing discussion on layer-pruning methods. Reviewers suggest more extensive discussions on comparison with early-exit models. 

- Extra computation overhead exists in 1) training multiple classifier heads and 2) self-distillation among layers. These classifiers are also required to be transmitted during FL loops. Since the proposed method uses a branchynet-style network to resolve the resource issue, it requires adding multiple intermediate supervision points with decoders or classifiers. If the decoders and the classifiers are heavy, adding them may nullify the saving in the reduced encoder and increase the complexity of the global model. 

- Reviewers have also suggested more baselines for comparison. 

Although there are some concerns regarding limited novelty, the merits of the paper, especially its strong empirical evaluation, outweigh.
justification_for_why_not_higher_score: - Limited novelty and position in the existing relevant literature: Several reviewers have expressed concerns about limited novelty as the proposed method is an application of Deeply-Supervised Nets (Lee et al., 2015) and deep mutual learning (Zhang et al., 2018) for FL to improve upon the layer-pruning method (InclusivFL). Related work is missing discussion on layer-pruning methods. Reviewers suggest more extensive discussions on comparison with early-exit models. 

- Extra computation overhead exists in 1) training multiple classifier heads and 2) self-distillation among layers. These classifiers are also required to be transmitted during FL loops. Since the proposed method uses a branchynet-style network to resolve the resource issue, it requires adding multiple intermediate supervision points with decoders or classifiers. If the decoders and the classifiers are heavy, adding them may nullify the saving in the reduced encoder and increase the complexity of the global model. 

- Reviewers have also suggested more baselines for comparison. 
justification_for_why_not_lower_score: This paper considers an important and timely problem. It has strong
technical contribution, using a self-distillation from the shallower part
to the deeper part to improve the final accuracy. The paper also
demonstrates strong empirical evaluation with experiments designed and
conducted extensively.
Decision: Accept: poster