Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a method to train a network that can run at different depth at test time. The author claimed that a residual block could be considered to perform two functions (1) learning new features (2) refine features. The refinement stage/layers do not change feature semantics thus can be skipped. The proposed method is evaluated on multiple backbones and is demonstrated to achieve better performance than individually trained networks.
strength_and_weaknesses: Strength
1. The paper showed another way to achieve adaptive depth network, that is dropping layers in every stage rather than just the last stages/layers as in previous works.
2. The method achieves better performance than individually trained networks.

Weakness
1. I am not fully convinced by the claim that 'A block with a residual function F is supposed to perform two primary functions (1) learning new higher level features and (2) refining already learned features at the same semantic level'. The paper didn't provide many evidences. The only one maybe Fig.4. But to me, in Fig 4(b), I think the first three blocks didn't change much which the 3rd, 4th, 5th blocks changed a lot, which is against the author's claim.
2. In Table 1, the author only compare the proposed method with some very early works such as slimmable networks. I suggest the author to add some more recent works such as [1, 2]. It seems that the proposed method didn't outperform these works.
3. How did the author build the baseline models ResNet29 and ResNet35. These models are not used in the original paper.
4. The training cost should also be reported since it needs multiple forward pass.
5. I am not sure I understand how the proposed method works. The loss KL(h_{super}|h_{base}) will pull these two features close. The minimum loss will be reached when the skippable layers are optimized to be zero mapping. Thus I am not sure how the network learns.

Refereces

[1] Li, Changlin, et al. "Dynamic slimmable network." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Yang, Taojiannan, et al. "MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations." IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
clarity,_quality,_novelty_and_reproducibility: Some claims in the paper is not well explained. The method part could be further simplified to make it easier to follow. The method seems to be simple and reproducible.
summary_of_the_review: I am not fully convinced by the claims made in the paper. The experimental evaluation is also not complete and missing some references.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel adaptive depth network by inducing a training strategy without the need of additional intermediate gate/classifier. It use a skip-aware BN, with a cost of 0.07% parameter increasing, but reduces the inference cost and achieve better results comparing with the non-adaptive baselines.
strength_and_weaknesses: Pro:
1) The method proposed by the paper to slim a residual network looks reasonable, it changed from a slimmable channels to slimmable sub-paths. It demonstrates in imagenet coco that a wider subnetwork performs better than deeper but thinner network based on resnet-like basenetworks [resnet50/101, mobilenetv2]. 

2) It includes several theoretical side evidence for using skip subpaths rather than skip channels. e.g. [Caramazza & Coltheart 2006], that looks convincing for me. 


Cons:
1) The baselines compared in the paper looks out dated, therefore is less convince for the generalization of the proposed strategy. 
I think the author should also demonstrate the effectiveness over stronger baselines for example: 

Vit-like networks, [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale]
Swin-transformers [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows]
Cvt networks [CvT: Introducing Convolutions to Vision Transformers]

They induces stronger results on all these tasks, it might be better we have baselines for them. 


clarity,_quality,_novelty_and_reproducibility: 1) The paper is easy to read, and understand. [page 6, table 1.  a typo in slimmable widths]

2) For novelty, the method is simply and easy to use, while I think stronger baseline and evidences are necessary to proof its generalization.

3) The author included code, and it should be easy to reproduce. 
summary_of_the_review: Good motivation and direction of research. Simple adaptive skip-paths strategy is proposed, while stronger experiments are needed
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to achieve anytime networks by controlling the network depth during runtime. The method focuses on the typical ResNet-like architectures and proposes to divide the residual blocks in each stage into two parts that are responsible for feature learning and feature refinement, respectively. A new training method is proposed to achieve the above goal, which is implemented as minimizing the difference between the super network and the base network in both final predictions and pooled intermediate features. Experimental results show that the proposed can achieve better trade-offs than the baselines and previous works.

strength_and_weaknesses: Strengths:

- The discussion on the two primary functions of the residual blocks is interesting.
- The paper is well-organized and easy to follow.

Weaknesses:
- The novelty of this paper is questioned. The overall design highly resembles previous work like AlphaNet[1]. The base network in this paper is actually a simplified version of the sampled sub-network in AlphaNet. Besides, while AlphaNet considers multiple scaling dimensions including network depth and width, this paper only considers network depth. As for the training method, both the AlphaNet and this paper adopt KL divergence to minimize the gap between the sub-network and the super-network. 

- Some of the theoretical parts lack insight and are somewhat redundant. For example, by some simple substitution and simplification, one can easily show that Equation (2)-(4) are duplicated. Besides, Equation (6)-(8) are just trivial Taylor expansion of the loss function, which is not strongly related to the proposed method.

- The experiments are not sufficient. According to the paper, the method can be applied to any architectures that are built with residual blocks, including the recent prevailing Vision Transformers [2]. I highly recommend that the authors should perform more experiments using ViT as their baseline to show the effectiveness of the proposed method.
 

[1] Wang, Dilin, et al. "AlphaNet: improved training of supernets with alpha-divergence." International Conference on Machine Learning. PMLR, 2021

[2] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).


clarity,_quality,_novelty_and_reproducibility: - **Clarity:** The paper is clearly written and easy to follow. There are some grammar errors (like “Slimmable widhts” in Table 1) that should be further corrected.

- **Quality.**  The quality is relatively low. The theoretical analysis lacks insight. The experiments are only performed with ResNet and MobileNet, which is not very convincing.

- **Novelty.** The novelty of the proposed method is low. I think it is a simplified version of AlphaNet.

- **Reproducibility.** Code is attached in this submission. 


summary_of_the_review: This paper presents a new method to achieve controllable depth during runtime. However, the proposed method highly resembles previous work AlphaNet. Besides, the lack of experiments makes the paper not very convincing. As a result, I lean towards rejecting this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work develops a method to reduce computational costs of a deep residual network without substantial detriment to its prediction accuracy. Specifically, it considers the second half of the layers in each residual block as disposable, i.e. one could skip those layers at inference time to reduce the runtime. To demonstrate this, the work experiments with minimising the KL-divergence between the output of a residual block and the features produced only by the first half of the layers. The experiments on image classification, object detection and instance segmentation indicate that network efficiency can be almost doubled only at a moderate loss in the prediction accuracy.
strength_and_weaknesses: Strengths 
- Parameter redundancy in neural networks is well known empirically, and it is certainly a natural idea to exploit it by learning to skip groups of subsequent layers.
- The implementation of the approach seems sufficiently simple.
- There is some effort in providing formal justification to motivate the approach (Eq. 2-5).
- The scope of the experiments is good and encompasses classification, instance segmentation and object detection in addition to the ablation study.

Weaknesses

- Overall, the demonstrated empirical improvements are very incremental. For example, on image classification they are somewhat comparable to Wang et al. 2021 (c.f. Tab. 2).
- The baseline does not seem sufficiently strong, in fact 2-3% lower than that reported in the original work (He et al., 2015). A short training schedule may actually disadvantage the baseline, since it has a larger number of parameters. (For the proposed method the convergence may happen faster, since due to the KL-divergence the number of training parameters is implicitly reduced).
- Isn’t the “super-net” the exact copy of the baseline, but with additional KL-divergence term? Why would the baseline have lower classification accuracy?
- There is generally lack of technical rigour which leaves some room for ambiguity and impedes the flow of the presentation. For example, Eq. 2-5 equate the expectation of an n-dimensional vector with a scalar value, which does not seem meaningful. Moreover, the implementation of these constraints (if one follows them somewhat intuitively) is specifically with the KL-divergence. I believe the equations should reflect that. Another example is Eq. (5): is it assumed to follow from Eq. (4), or is it a standalone assumption?

Somewhat minor:
- Inconsistent notation, e.g. h_super in Eq. 1 (with or without parenthesis?); same for h_base.
- Eq. 7 seems sloppy (what is in O(.), partial derivative vs. total derivative).
- the special treatment of the 23 blocks in ResNet-101 seems to be rather contrived.
- I do not see much support in the KL-divergence to "preserve the semantic level of input features".

Further notes:
- Table 1: Please provide details of the evaluation protocol, e.g. what data and criterion was used to select the best model for evaluation?
- Fig 4c does not appear very convincing: the features learned by the base network are somewhat reminiscent of the “super-net”.
- Table 3 unclear: if neither of the two components are used, this should be the baseline setting, shouldn’t it?
- The implementation seems to use a hyperparater T set to 4 in the KL-divergence terms. It is not mentioned in the paper. What is its use?
clarity,_quality,_novelty_and_reproducibility: - The presentation is fairly clear, apart from the technical aspects discussed above.
- The quality is sufficient, but suffers primarily due to the lack of careful notation and rigour in experimentation.
- There is some novelty in skipping the computation in the residual blocks as proposed here. However, it could still be considered as a particular implementation of model pruning, which is a well-studied strategy.
- The code is provided for review. There is some discrepancy w.r.t. the paper description, hence I'd assume the work could be diffuclt to reproduce exactly without the original code.
summary_of_the_review: This work achieves reduction in model efficiency with tolerable loss in accuracy. These results are encouraging. However, the experimental protocol requires more elaboration, much like the technical aspects of the presentation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a method for anytime networks with adaptive depth without relying on intermediate gates/classifier decisions. The problem addressed by the authors is very important and timely. However, all reviewers raised major concerns about the paper, including incremental novelty over prior methods (e.g., AlphaNet) and outdated baselines.
justification_for_why_not_higher_score: All four expert reviewers consider the paper does not pass the acceptance bar of ICLR, unfortunately. The AC agrees with this decision. While the problem addressed by the authors is very important, the reviewers raised concerns about incremental novelty and outdated baselines. The AC agrees the method is similar to AlphaNet as pointed out by one of the reviewers. The author response provided new experiments, including more recent baselines, such as ViT. However, considering inference efficiency, as opposed to training, it is not clear that the proposed approach has significant advantages over AlphaNet and other approaches. It would also be desirable to see experiments with more modern ViT architectures (and shoe curves depicting latency vs accuracy at multiple operating points). The authors are encouraged to revise this work and submit the paper to another top conference.
Decision: Reject