Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper tackles the problem of dynamic inference where the forward graph will depend on the input data. It focuses on real speedups instead  of theoretical one, this makes the work more valuable. There is a latency prediction model that can estimate the latency by considering the algorithms, scheduling and hardware properties. Experiments are performed on image classification and demonstrate latency reduction of 23 to 45% depending on the hardware superiority. 
strengths_and_weaknesses: Strengths:

+ Dynamic inference is a field of great interest. Naturally, people spent different effort to perform vision tasks so should deep network. 
+ The method considers the real latency and not FLOPs. Additionally, demonstrating real speed-up is greatly appreciated. 
+ Multiple paths are considered during training, and a single path is executed during inference.  
+ Pare is well written and the content comes smoothly. 

Weaknesses:

- Using distillation during fine-tuning will increase accuracy and might be not fair comparing to the original model.
- When compared to the previous work like Conv-AIG, the training recipes might be different (authors start from trained model and do 100 more epochs). The over training pipeline is not described as well, things like augmentations, learning rate scheduler etc should be mentioned. If those are not the same (as number of epochs) then authors should perform a fait comparison with those being the same. 
- There are questions in the section before that are not clear. Understanding them will help to evaluate paper better. 
questions: - How does hardware parameter vector H look like?
- The activation factor $r$, is it preselected or computed automatically? What would be con\pros of both solutions?
- What is the architecture of masker? 
- Is the mask the same for all input/output channels? Or it does vary? 
- Why the case of S=56 is not considered? If my understanding is correct then the entire feature map will be removed which is helpful to reduce latency. 
- Explain how sparsity is set of different layers and why S is different for different models and layers in section 4.3.1 
- Implementation of the 2 branch model in Figure 2 is not clear. If the mask is adaptive then there might be more than 1 candidate with sparse convolution, how would Gumbel softmax sample them? How M is regularized? 
- How is latency computed in Figure 7? Was the setup the same (operator fusion etc) for all models considered? Those questions raise because skipping the entire block as in Convolutional-AIG should be more efficient in terms of latency reduction. 
- Where was the framework implemented? If it was C++ then comparing with cuDNN and TesnorRT would be valuable as there is additional model optimization. 
- Will the code be released? The latency prediction model might be useful to community for future research. 

Minor:

- In the first section, the coarse granularity is mentioned. It would be helpful to explain it once it appears in the text.  
limitations: - Limitations are not listed by authors
- There are seem to be a possible a set of hyper-parameters that need to be tuned like S etc. 
- Some details are missing and clarifying them will be helpful.
- It is not clear how different convolution implementations (like Winograd) will benefit from the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a dynamic inference paradigm based on selective inference of convolutions on the spatial dimension. Specifically, for each convolutional block, it uses a masker layer to predict the masked region of the feature map and only conduct convolution on the masked region. Such convolution has less computation than a full convolution and improves efficiency of the model. Moreover, it utilizes a latency prediction model to estimate the latency of different mask operations. With a differentiable training framework, the model can be optimized according to configurations different hardware platform.
strengths_and_weaknesses: Strength:
-	Although masked inference is not a new idea, this work targets at a more practical inference speed optimization compared to existing dynamic-masking approaches. Several critical modifications for the real efficiency of this approach are adopted in the inference pipeline: fused operations, granularity choices of the mask etc.
-	Experiments with ResNet and RegNet validates the effectiveness of this method on several tasks: ImageNet classification and object detection. It reduces inference time by 23% and 45% on V100 and TX2 platform respectively without significant drop on accuracy.
Weaknesses:
-	The framework can only be applied to models with ResNet block whose major computations are located in the 3x3 conv of the network blocks. I don’t think it work well for other efficient models such as MobileNet and ShuffleNet that does not have heavy 3x3 convs.
-	Only GPU platforms are tested. Does this approach generalize to other types of hardware such as CPU? 
-	For the speed comparison, what kind of cuda runtime environment are you using for the experiments? What are the cuda version, cudnn version? are you using TensorRT? These details are not given in the main text.

questions: See above.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission introduces a spatially dynamic neural network approach, developed in a latency-aware manner that can generate realistic speed-ups during inference. The proposed methodology allows to dynamically skip computation at different spatial regions of the feature maps at a coarser granularity (of blocks of pixels), directly generalising the widely studied pixel-level approaches from the literature. This leads to more regular computation pattern and reduces the overheads, to maximise the attainable inference speed gains. Additionally, the granularity of the pixel blocks is tuned in a latency-aware manner to balance the speed-accuracy trade-off. Results indicate that the proposed approach is effectively improving efficiency, across hardware platforms and models for image classification and object detection tasks.
strengths_and_weaknesses: Strengths:
- The paper addresses a very important real-world issue, which is well-motivated and demonstrated experimentally.
- The proposed approach, although rather simple, is able to generate realistic speed-ups with no effect in accuracy, showcasing improved efficiency compared to baselines.
- Useful insights are provided in the experiments analysis.

Weaknesses (see questions for more details):
- The novelty of the proposed approach is limited, as it directly generalises a widely studied problem, going from pixel-level to block-level. Similar generalisation in the context of early-exit networks has recently been studied in the literature:
  - Liu, Zhuang, et al. "Anytime Dense Prediction with Confidence Adaptivity." International Conference on Learning Representations. 2022.
-  The description of the latency prediction model (Sec 3.3) is very high-level, making it hard to justify its contribution . 
- Some interesting aspects of this work (eg the generic formulation combining spatial computation skipping and layer skipping - Sec 3.2) are only theoretically mentioned, but not experimentally studied.
- Some of the claimed contributions and ablation studies are mostly comprise technical/implementation aspects (Sec 3.4).
- Evaluation is focused on Classification/detection, but it is unclear how the proposed methodology can be applied to more dense CV tasks, such as instance/semantic segmentation. The reviewer believes that this mapping won't be intuitive and this should be discussed in the limitations of the proposed approach.

Presentation
- Syntax/grammatical errors exist sporadically across the manuscript, mostly in Sec1, 3.3 and 4.5
- Some aspects of the proposed methodology are described in rather high-level, making it impossible to reproduce.
questions: 1. How is the latency predictor implemented? Is it trainable, or based on an analytical performance model/ cycle accurate simulator? Can it capture the effect of more complex operations, such as dilated or separable convs, skip connections etc?
2. The case where a large S can emulate layer skipping should be considered in the ablation.
3. How would the proposed approach be applied to dense tasks, such as semantic segmentation, dense prediction etc ?
4. Is there any benefit on allowing overlapping pixel blocks to be considered during masking? Qualitative results indicate that such a setting may allow for larger values of S. 
limitations: Some limitations are discussed in the conclusion section. The authors are encouraged to enhance this discussion.
review_title: null

Meta-review: metareview: The paper proposes latency-aware spatial-wise dynamic neural networks under the guidance of a latency prediction mode. reviewers arrived at a consensus to accept the paper. 
confidence: Certain
Decision: Accept