Latency-aware Spatial-wise Dynamic Networks
Authors: Yizeng Han
Zhihang Yuan
Yifan Pu
Chenhao Xue
Shiji Song
Guangyu Sun
Gao Huang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Spatial-wise dynamic convolution has become a promising approach to improving the inference efficiency of deep networks. By allocating more computation to the most informative pixels, such an adaptive inference paradigm reduces the spatial redundancy in image features and saves a considerable amount of unnecessary computation. However, the theoretical efficiency achieved by previous methods can hardly translate into a realistic speedup, especially on the multi-core processors (e.g. GPUs). The key challenge is that the existing literature has only focused on designing algorithms with minimal computation, ignoring the fact that the practical latency can also be influenced by scheduling strategies and hardware properties. To bridge the gap between theoretical computation and practical efficiency, we propose a latency-aware spatial-wise dynamic network (LASNet), which performs coarse-grained spatially adaptive inference under the guidance of a novel latency prediction model. The latency prediction model can efficiently estimate the inference latency of dynamic networks by simultaneously considering algorithms, scheduling strategies, and hardware properties. We use the latency predictor to guide both the algorithm design and the scheduling optimization on various hardware platforms. Experiments on image classification, object detection and instance segmentation demonstrate that the proposed framework significantly improves the practical inference efficiency of deep networks. For example, the average latency of a ResNet-101 on the ImageNet validation set could be reduced by 36% and 46% on a server GPU (Nvidia Tesla-V100) and an edge device (Nvidia Jetson TX2 GPU) respectively without sacrificing the accuracy. Code is available at https://github.com/LeapLabTHU/LASNet.

Latency-aware Spatial-wise Dynamic Networks
Yizeng Han' Zhihang Yuan Yifan Pul
Chenhao Xue²
Shiji Song' Guangyu Sun Gao Huang
Department of Automation, BNRist, Tsinghua University, Beijing, China School of Electronics Engincering and Computer Science, Peking University, Beijing, China (hanyz18, pyf20} @mails. tsinghua. edu. en, {shijis, Stsahuannggtssnnguua edu. cn (yuanzhihang, xch927027, gsun}Opku. edu.
Abstract
Spatial-wise dynamic convolution has become promising approach improving the inference efficiency of deep networks. By allocating more computation to the most informative pixels, such adaptive inference paradigm reduces the spatial redundancy image features and saves a considerable amount unnecessary computation. However, the theoretical efficiency achieved by previous methods can hardly translate into a realistic speedup. especially on the multi-core processors (e.g. GPUs). The key challenge is that the existing literature has only focused on designing algorithms with minimal computation, ignoring the fact that the practical latency can also be influenced by scheduling stralegies and hardware properties. To bridge the gap between theoretical computation and practical ef- ficiency, we propose latency-aware spatial-wise dynamic network (L.ASNet), which performs coarse- graiiee spatially adaptive inference under the guidance of novel latency prediction model. The latency prediction model can efficiently estimate the inference latency of dynamic networks by simultancously considering algorithms,, scheduling strategies, and hardware properties. We use the latency predictor to guide both the algorithm design and the scheduling optimization on various hardware platforms. Experiments on image classification, object detection and instance segmentation demonstrate that the proposed framework significantly improves the practical inference efficiency of deep networks. For example, the average latency of a ResNet- 101 on the ImageNet validation set could be reduced by 36% and 46% on a server GPU (Nvidia Tesla-V V V100) and an edge device (Nvidia Jetson TX2 GPU) respectively without sacrificing the accuracy. Code is available at htttps://githiii com/LeepLabTTU/LLAS
1 Introduction
Dynamic neural networks have attracted great research interests in recent years. Compared to static models [1 can allocate computation
which treat different inputs equally during inference, dynamic networks data-dependen manner. For example, they can conditionally skip the computation of network layers [15, 32 30 or convolutional channels [19 D. perform spatially adaptive inference on the most informative image regions (e.g. the foreground areas) 5 31 35 33 8. Spatial-wise dynamic networks, which typically decide whether to compute (Figure ! (a)), have shown promising results in improving the inference efficiency of convolution neural networks (CNNs).
17
13 23
each feature pixel with masker modules [5] 35|
Despite the remarkable theoretical efficiency achieved by spatial-wise dynamic networks 31 35), researchers have found it challenging to translate the theoretical results into realistic speedup.
"Equal contribution.
I Cooresponniig author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Masker
selected pixel locations
Influence
guidance
Algorithm
Dynamic Convolution(S)
Algorithm: Spatially adaplive inference with masker.
Latency
Hardware
gather
convalution(s)
scalter
Scheduling
(b) Scheduling: the practical inference pipeline dynamic convolutions.
(c) The general idea of our work.
Figure 1: An overview of our method. (a) illustrates the spatially adaptive inference algorithmr; (b) is the scheduling strategy: and (c) presents the three key factors to the practical latency. For given hardware, the latency is used to guide our algorithm design and scheduling optimization.
especially on some multi-core processors, e.g., GPUs [35 3 8]. The challenges are two-fold: 1) most previous approaches [5] (5] perform spatially adaptive inference at the finest granularity: every pixel is flexibly decided whether to be computed or not. Such flexibility induces non-contiguous memory access (35) and requires specialized scheduling strategies (Figure (b)); 2) the existing literature has only adopted the hardware-agnoste FLOPs Eloaaiin-poiit operations) an inaccurate proxy for the efficiency, lacking latency-awe aware guidance on the algorithm design. For dynamic networks, the adaptive computation with sub-optimal scheduling strategies further enlarges the discrepancy between the theoretical FLOPs and the practical latency. Note that it has been validated by previous works that the latency CPUs has strong correlation with FLOPs B 35). Therefore, we mainly focus on the GPU platform this paper, which is more challenging and less explored.
We address the above challenges by proposing latency-aware spatial-wise dynamic network (LASNet). Three key factors to the inference lateney are considered: the algorithm, the scheduling strategy, and the hardware properties. Given target hardware device, we directly use the latency, rather than the FLOPs, to guide our algorithm design and scheduling optimization (Figure (c)).
Because the memory access pattern and the scheduling strategies in our dynamic operators differ from those in static networks, the libraries developed for static models (e.g. cuDNN) are sub-optimal for the acceleration of dynamic models. Without the support of libraries, each dynamic operator requires scheduling optimization, code optimization, compiling, and deployment for each device. Therefore, it is laborious to evaluate the network latency on different hardware platforms. To this end, we propose novel latency prediction model to efficiently estimate the realistic latency of network by simultaneously considering the aforementioned three factors. Compared to the hardware-agnosse FLOPSs, our predicted latency can better reflect the practical efficiency of dynamic models.
Guided by this latency prediction model, we establish our latency spatial-wise dynamic network (LASNet), which adaptively decides whether to allocate computation on feature patches instead of pixels (5 31 355 (Figure 2 top). We name this paradigm as spatially adaptive inference at coarse granularity. While less flexible than the pixel-level adaptive computation previous works 5,31 it facilitates more contiguous memory access, benefiting the realistic speedup on hardware. The scheduling strategy and the implementation are further ameliorated for faster inference.
It is worth noting that LASNet is designed as a general framework in two aspects: 1) the coarse- grained spatially adaptive inference paradigm can be conveniently implemented in various CNN backbones, e.g., ResNets [1]] DenseNets [17) and RegNets (25): and 2) the latency predictor is an off-the-shell tool which can be directly used for various computing platforms (e.g. server GPUs and edge devices).
We evaluate the performance of LASNet on multiple CNN architectures on image classification, object detection, and instance segmentation tasks. Experiment results show that our LASNet improves the efficiency of deep CNNs both theoretically and practically. For example, the inference latency of ResNet- 101 on ImageNet 4 is reduced by 36% and 46% on an Nvidía Tesla V100 GPU and an Nvidia Jetson TX2 GPU, respectively, without sacrificing the accuracy. Moreover, the proposed method outperforms various lightweight networks in a low-FL.PS, regime.
Our main contributions are summarized as follows:

---

1. We propose LASNet, which performs moarse-graines spatially adaptive inference guided by the practical latency instead of the theoretical FLOPs. To the best of our knowledge. LASNet is the first framework that directly considers the real latency in the design phase of dynamic neural networks;
2. We propose a latency prediction model, which can efficiently and accurately estimate the latency of dynamic operators by simultaneously considering the algorithm, the scheduling strategy, and the hardware properties;
3. Experiments on image classification and downstream tasks verify that our proposed LAS- Net can effectively improve the practical efficiency of different CNN architectures.
Related works
Spatial-wise dynamic network is common type of dynamic neural networks 77. Compared to static models which treat different feature locations evenly during inference, these networks perform spatially adaptive inference the most informative regions (e.g., foregrounds), and reduce the unnecessary computation on less important areas (e.g., backgrounds). Existing works mainly include three levels of dynamic computation: resolution level [36 371, region level [33 and pixel level
5 31 35 The former two generally manipulate the network inputs 33 37 or require special
architecture design [36]. In contrast, pixel-level dynamic networks can flexibly skip the convolutions
on certain feature pixels in arbitrary CNN backbones [5] 3131355 Despite its remarkable theoretical efficiency, pixel-wise dynamic computation brings considerable difficulty achieving realistic speedup on multi-core processors, e.g., GPUs. Compared to the previous approaches (5) 31 35 which only focus on reducing the theoretical computation, we propose directly use the latency to guide our algorithm design and scheduling optimization.
Hardware-awarg network design. To bridge the gap between theoretical and practical efficiency of deep models, researchers have started to consider the real latency in the network design phase. There two lines of works in this direction. One directly performs speed tests on targeted devices, and summarizes some guidelines to facilitate hand-designing lightweight models [23]. The other line of work searches for fast models using the neural architecture search (NAS) technique
29 34). However, all existing works try to build static models, which have intrinsic computational redundancy by treating different inputs in the same way. However, speed tests for dynamic operators on different hardware devices can be very laborious and impractical. In contrast, our proposed latency prediction model can efficiently estimate the inference latency on any given computing platforms by simultaneously considering algorithm design, scheduling strategies and hardware properties.
3 Methodology
In this section, we first introduce the preliminaries of spatially adaptive inference, and then demon- strate the architecture design of our LASNet. The latency prediction model is then explained, which guides the granularity settings and the scheduling optimization for LASNet. We further present the implementation improvements for faster inference, followed by the training strategies.
3.1 Preliminaries
Spatially adaptive inference. The existing spatial-wise dynamic networks are usually established by attaching masker M in each convolutional block of a CNN backbone (Figure (a)). Specifically, let x H xC denote the input of a block, where H and W are the feature height and width, and Cs the channel number. The masker M takes x input, and generates a binary-valued spatial mask = (x) [0, H Each element M determines whether to perform convolution operations on the corresponding location of the output feature. The unselected regions will be filled with the values from the input 5 31 or obtained via interpolation [35]. We define the activation rate
E., M
of block
representing the ratio of the calculated pixels.
HxW
Scheduling strategy. During inference, the current scheduling strategy for spatial-wise dynamic convolutions generally involve three steps (26) (Figure (b)): 1) gathering, which re-organizes the selected pixels (if the convolution kernel size is greater than x 1, the neighbors are also required) along the batch dimension; 2) computation, which performs convolution on the gathered input; and 3) scattering, which fills the computed pixels on their corresponding locations of the output feature.
3

---

Training
Convixt
Comadr3
Convtxt
Gumbel
Noise
Mouzzz
Input
H
Oulput
Masker
Input
Inference
Masker
Gather-
Qulput
Convixt
Convaxa
Upsemple
Element-wise
Regular convolution
Operator fusion
Selected pixel
multipiication
Figure 2: Our proposed LASNet block. Top: we first generate low-resolution spatial mask Meoarse- which is then upsampled to obtain the mask M with the same size the output feature. Gumbel Softmax [18 24] used for end-to-end training (Sec. 3.5). Bottom: the scheduling optimization is performed to decrease the memory access for faster inference (Sec. 3.40.
Limitations. Compared to performing convolutions on the entire feature map, the aforementioned scheduling strategy reduces the computation while bringing considerable overhead to memory access due to the mask generation and the non-contiguous memory access. Such overhead would increase the overall latency, especially when the granularity of dynamic convolution is at the finest pixel level.
3.2 Architecture design
Spatial granularity. As mentioned above, pixel-level dynamic convolutions 5 31 35 stantial challenges to achieving realistic speedup multi-core processors due to the non-contiguous memory access. To this end, we propose to optimize the grunularity of spatially adaptive inference. Specifically, take the commonly used bottleneck structure in 11] as example, our coarse coarse-grained spatial-wise dynamic convolutional block illustrated Figure 2 Instead of directly producing
raise sub-
a mask with the shape of H x W, we first generate a low-resolution mask Mcoarse E (0, 1 1 * where 5 is named as the spatial granularity. Each element in Mcoarse determines the computation of an S S S-sized feature patch. For instance, the feature size the first ResNet stage 56 x 56. Then the possible choices for S are (1, 4, 8, 14, 28, 56) The mask Mcearse is then upsampled to the size of H x W. Notably, = means that the granularity still at the pixel level as previous methods (5) Note that the other extreme situation (S = 56) is not considered in this paper, when the masker directly determines whether to skip the whole block (i.e. layer skipping [30] (32). Such an overly aggressive approach will lead considerable drop of accuracy, as we presented in Appendix C.2 The masker is composed of a pooling layer followed by 1 convolution.
Differences to existing works. Without using the interpolation operation BS the carefully designed two-branch structure [8), the proposed block architecture is simple and sufficiently general to be plugged into most backbones with minimal modification. Our formulation is mostly similar to that in [31], which could be viewed as a variant of our method with the spatial granularity S=1 for all blocks. Instead of performing spatially adaptive inference at the finest pixel level, our granularity is optimized under the guidance our latency prediction model (details are presented in the following Sec. 4.2 to achieve realistic speedup on target computing platforms.
3.3 Latency prediction model
As stated before, it is laborious to evaluate the latency of dynamic operators on different hardware platforms. To efficiently seek preferable granularity settings arbitrary hardware devices, propose latency prediction model G, which can directly predict the delay of executing dynamic operators any target devices. For a spatial-wise dynamic convolutional block, the latency predictor takes the hardware properties H. the layer parameters P. the spatial granularity S. and the activation rate input and predicts the latency of dynamic convolutional block: = ( S.
Hardware modeling. We model a hardware device as multiple processing engines (PEs), and parallel computation can be executed these PEs. As shown in Figure 3 we model the memory system as
Here refer to stage the cascading of multiple blocks which process features with the same resolution.

---

Computation Units
Prodictd
sip-spppoee Memory
Cn-ch Thr-cheMenne Memery
[LLC/ Burler)
Memeny
Compulation
FE
Figure 3: Our hardware model.
Figure 4: Latency prediction results.
three-level structure [2]
: 1) off-chip memory, 2) on-chip global memory, and 3) memory in PE. Such hardware model enables us accurately predict the cost both data movement and computation.
Latency prediction. When simulating the data movement procedure, the efficiency of non- con-contiguous memory accesses under different granularity S settings is considered. As for the computation latency, it important to adopt a proper scheduling strategy to increase the parallelism of computation. Therefore, search for the optimal scheduling (the configuration of tiling and in-PE parallel computing) of dynamic operations to maximize the utilization of hardware resources. A more detailed description of our latency prediction model is presented in Appendix A
Empirical validation. We take the first block in ResNet-101 as an example and vary the activation rate to evaluate the performance of our prediction model. The comparison between our predictions and the real testing latency on the Nvidia V100 GPU illustrated Figure from which we can observe that our predictor can accurately estimate the real latency in a wide range of activation rates.
3.4 Implementation details
We use general optimization methods like fusing activation functions and batch normalization layers into convolution layers. We also optimize the specific operators in our spatial-wise dynamic convolutional blocks as follows (see also Figure 2 for an overview).
Fusing the masker and the first convolution. As mentioned in Sec. 3.1 the masker in each block consumes very little computation, but it takes the whole feature map as input. Therefore, it is a memory-bounded operation (the inference time is mainly spent memory access). Since the masker and the first convolution in the block share the same input, there is an opportumity to fuse these two operations avoid the repeated access of the input data. Note that spatial-wise dynamic convolution requires the output of the masker. If we fuse the two layers, the first convolution will be changed to a static operation, which may increase the inference latency. There exists threshold of activation rate Tth, when > Tth, the overall latency can be reduced. We decide whether to fuse them according to the average activation rate. See more details in Appendix B
Fusing the gather operation and the dynamic convolution. Traditional approaches first gather the input pixels of the first dynamic convolution in block (Figure (b)).. The gather operation is also a memory-bounde -bounedd operation. Furthermore, when the size of the convolution kernel exceeds I x 1, the area of input patches may overlap, resulting in repeated memory load/store. We fuse the gather operation into the dynamic convolution to reduce the memory access.
Fusing the scatter operation and the add operation. Traditional approaches scatter the output pixels of the last dynamic convolution, and then execute the element-wiss addition (Figure (b)). We fuse these two operators to reduce the memory access. The ablation study in Sec. 4.4 validates the effectiveness of the proposed fusing methods.
3.5 Training
Optimization of non-differentiable maskers. The masker modules are required to produce binary- valued spatial masks for making discrete decisions, and cannot be directly optimized with back we adopt straight-through Gumbel Softmax [8] 24
propagation. Following [35] 31 8)
to train the
network in an end-to-en fashion. Specifically, let M e / xxxX2 x2 denote the output of the mask generator. The decisions are obtained with the argmax function during inference. In the training phase, a differentiable approximation is defined by replacing the argmax operation with a Softmax:
exp (1o8 (M..)) + G...o) G..10//))
M
E [0, NKWW
(1)
Σ1 k exp (log (M...) + G...k) /T]
5

---

232
Adlacon mi
Addesion rea
(a) Relationship between 7t and r for blocks on the Nvidia Tesla V100 GPU.
1.5
$32
26 30
24
04 06 6.
04
85
8.4
(b) Relationship between Fy and r for LAS-RegNatY-SOOMMP blocks on the Nvidia Jetson TX2 GPU.
Figure 5: Latency prediction results of LAS- blocks V100 (a) and LAS- blocks TX2 (b). For both networks, we plot the relationship between the latency ratio r; and the activation rate for the blocks in 4 stages with the convolutional stride 1. The practical efficiency is only improved when ry < 1. Note that S = 1 can harm the practical latency even for a small r (reduced computation), while a larger S will alleviate this problem. See detailed analysis in Sec. 4.2
where t is the Softmax temperature. Following the common practice [3 8l. we let 7 decrease exponentially from 5.0 to 0.1 in training to facilitate the optimization of maskers.
Training objective. The FLOPs of each spatial-wist dynamic convolutional block be calculated based on our defined activation rate [31]. Then we can obtain the FLOPs of the overall dynamic network Fayn. Let Fstat denotes the FLOPs of its static counterpart. We optimize their ratio to approximate a target < t < 1:LFLOPs = ( Fayn
Fast - +)2 addition, we define loss item Lounds as in 3T to constrain the upper bound and the lower bound of activation rates in early training epochs.
We further propose to leverage the static counterparts of our dynamic networks as "teachers" to guide the optimization procedure. Let y and y' denote the output logits of dynamic model ("student") and its "teacher", respectively, Our final loss can be written as
L = L + aLLLOOP + Lhounds) + BT (L(oy/TT)loo) (T)/T)),
(2)
where Ltask represents the task-related loss, e.g., cross-entropy loss in image classification. KL(-II-) denotes the Kullback-Leible divergence, and az, B are the coefficients balancing these items. We use o to denote the log- Soff-ax function, and the temperature for computing KL-divergence.
Experiments
In this section, first introduce the experiment settings in Sec. 4.1 Then the latency of different granularity settings are analyzed Sec. 4.2 The performance of our LASNet on ImageNet is further evaluated in Sec. [4.3 followed by the ablation studies in Sec. Sec.[4.4] Visualization results are illustrated in Sec. 4.5 and we finally validate our method on the object detection task (Sec. (46)). The results on the instance segmentation task are presented in For simplicity, add "LAS-" as a prefix before model names to denote our LASNet, e.g., LAS-ResNet-50..
4.1 Experiment settings
Latency prediction. Various types of hardware platforms are tested, including server GPU (Tesla V100), a desktop GPU (GTX1080) and edge devices (e.g.. Nvidia Nano and Jetson TX2). The major properties considered by our latency prediction model include the number of processing engines (#PE), the floating point computation in processing engine (#FP32), the frequency and the bandwidth. can be observed that the server GPUs generally have larger #PE than the loT devices. The batch size is set as for all dynamic models and computing platforms.
Image classification. The image classification experiments conducted the ImageNet dataset. Following [1,, we initialize the backbone parameter from pre-trained checkpoin and
Wee use the torchvision pre- pre-trained models at htttpp///ppportt...oo////sssaaaa

---

F=0.05
7=0.00
1,0
0:30
Fx023
25
025
2.0
3.5
155
1.0
Spatia granubrity 19)
Sparii wwnttttgn is)
Sparts granuattty (R1
(a) Relationship between and S for LAS-ResNet blocks on V100.
1.75
Weeng tol
1.52
7-335
7-210
14
0.8
2.75
130
5.29
17
Sparaa groneriiyy (5)
grenat 191
Sparii grew by
(b) Relationship between Ty and S for LAS-RegNN blocks on Nvidia Jetson TX2 GPU.
Figure 6: The relationship between the latency ratio ry and the spatial granularity S.
finetune the whole network for 100 epochs with the loss function in Eq. 2 We fix a 10, 0.5 and T = 4.0 for all dynamic models. More details are provided in Appendix B
4.2 Latency prediction results
In this subsection, we present the latency prediction results of the spatial-wise dynamic convolutional blocks in two different models: LAS-ResNet-101 [I] (on V100) and LAS- SeSS-ReNNeeY--OO00 (25 (on TX2). All the blocks have the bottleneck structure with different channel numbers and convolution groups, and the RegNetY is equipped with Squeeze aud-Exxitation (SE) [14] modules.
We first define layn the latency of spatial-wise dynamic convolutional block, and Estat as that of
We investigate the relationship between r. and the activation rate (cf. Sec. 3.5) for different granularity settings. The results in demoossteee that: 1) even equipped with our special optimization on the scheduling strategies, pixel-level spatially adaptive inference (S=1) cannot always improve the practical efficiency. Such fine-grained adaptive inference is adopted by most previous works [31 35 and our result can explain the reason why they can only achieve realistic speedup on less powerful CPUs (35) or specialized devices (3]; 2) proper granularity S > 1 effectively alleviates this problem on both hardware devices. By setting S > 1, realistic speedup could be achieved with larger activation rates.
static block without a masker. Their ratio is denoted as rę Estas
Eayn.
The latency prediction results are further used to seek for preferable granularity setting for the first stages (we fix S for the last stage, where the feature resolution is 7 x 7). Therefore, we plot the relationship between rt and S in Figure I0 It can be observed that: 1) r generally decreases with S increasing for given r; 2) an overly large S (less flexible adaptive inference) brings insignificant improvement on both devices. Especially. enlarging S from 8 to 28 in the first stage of a Resseet brings very little improvement on V100. Based on the results in Figure TO, we can trade off between flexibility and efficiency by selecting appropriate S for different models and hardware devices. For example, we can simply set = in a L.AS-ResNet-Ill 101 to achive realistic speedup. The accuracy-latency plots in Figure 7 also validate this observation. More results of our latency prediction model on the desktop-level GPU, GTX1080, are presented Appendix C.I
4.3 ImageNet classification results
We now empirically evaluate our proposed LASNet on the ImageNet dataset. The network perfor- mance is measured in terms of the trade-off between classification accuracy and inference efffciency. Both theoretical (i.e. FLOPs) and practical efficiency (i.e. latency) are tested in our experiments.
'e use this form to represnet the S settings for the 4 stages of a network.
7

---

134
741
713
Accursoy 154
Accurscy
FLOPS (Kle8)
Latensy on Nuida jasson TX2 (PS))
Latency on lvida Tesa V100 GPU (mms)
(a) LAS -eAS-Reeeett results.
18.0
25.4
75.
rolku Locl
FLOPs (cte9)
Latency on Nvidia Jetson TX2 GPU ims)
Latency on Nvidds nano GPU ims)
(b) LAS-RegNet results.
Figure Experimental results on ImageNet. The proposed coarse- goarse-grained spatially adaptive inference is tested on standard ResNets (a) and lightweight RegNets (b).
4.3.1 Standard baseline comparison: ResNets
We first establish our L.ANNet based on the standard ResNets 11. Specifically, we build LAS- ResNet-50 and LAS-ResNet-101 by plugging our maskers in the two common ResNet structures.
The baselines include various types of dynamic inference approaches: I) layer skipping (SkipNet 32 and Conv-AIG (30); 2) channel skipping (BAS M); and 3) pixel-level spatial-wise dynamic
network (DynConv 31). For our LASNet, we compare various settings of the spatial granularity Snet- We set training targets (cf. Sec. 3.5 EE {0, 4, 0.5, 0.6, 0.7} for our dynamic models to evaluate their performance in different sparsity regimes. We apply the same operator fusion (Sec. 3.4 for both our models and the compared baselines 31 for fair comparison.
Results are presented in Figure 7 (a). On the left we plot the relationship of accuracy v.s. FLOPs. It can be observed that our -es-ResNNets with different granularity settings significantly outperform the competing dynamic neural networks. Surprisingly, coarse graiined spatially adaptive inference -4--2=444-21 and = for the 4 stages) can achieve even higher accuracy when consuming similar FLOPs on ResNets, despite the sacrificed flexibility compared to --er=I-----, that a larger S is also beneficial to the optimization of maskers.
We conjecture
We compare the practical latency of three granularity settings in Figure 7 (a) predicted by latency prediction model (middle on TX2 and right on V100). We can witness that although they achieve comparable theoretical efficiency (Figure (a) left), larger S is more to the finest granularity. For example, the inference latency of is significantly higher than the ResNet- 101 baseline on V100 (Figure 7 (a) right), even though its theoretical computation is much smaller than that of the static model. However, larger granularities 4--211444-2-1 and Spet can effectively improve the inference latency due to its lower burden on the memory access. Remarkably, the latency of ResNet-101 could be reduced by 36% and 46% on V100 and TX2 respectively without sacrificing the accuracy when (=0.4. The classification accuracy is increased by 1.9% with similar inference efficiency. It can be observed that the realistic speedup ratio rę is more close to the theoretical FLOPs ratio target t on the less powerful TX2, because the latency is computation-bounded (i.e. the latency is mainly spent on computation) on such IoT devices. In contrast, there a larger gap between practical and theoretical efficiency on the more powerful V100, as the latency is bounded by the memory access cost.
compared (Sue==1-1-1-1)
4.3.2 Lightweight baseline comparison: RegNets
We further evaluate our LASNet in lightweight CNN architectures, i.e. RegNets- Y (25). Two different sized models are tested: RegNerY-400ME and RegNetY-800MFF Compared baselines include other types of efficient models, e.g., MobileNets-v2 288. ShuffletNets-v2 (23) and CondenseNets [16].
The results are presented 7 (b). The x-axis for the three sub-figures are the FLOPPs the latency TX2, and the latency the Nvidia nano GPU, respectively. We can observe that our
8

---

Noddl Tesio V100 Inst
Figure 8: Ablation studies on S.
Figure 9:
Visualization results.
method outperforms various types of static models in terms of the trade-off between accuracy and efficiency. More results on image classification are provided in Appendix
4.4 Ablation studies
We conduct ablation studies to validate the effectiveness of our coarse -rriiee spatially adaptive inference (Sec. B.2 and operator fusion operations (Sec. 3.4)
More granularities settings. We test various granularity settings on LAS-ResNet-101 to examine the effects of S in different stages. The results the Tesla-V100 GPU presented in Figure 8 It can be found that the finest granularity (Sme:=1-1-1-1) leads to substantial inefficiency despite the reduced FLOPs (cf. Figure (a) left). Coarse-grained spatially adaptive inference in the first two stages (Snet 14-4-1-1 effectively reduces the inference latency. We further increase S in the third stage 2 and 7, and this procedure consistently improves the realistic efficiency the V100 GPU, This trend also holds on the GTX 1080 GPU (see the results in Appendix C.2
Operator fusion. We investigate the effect of our operator fusion introduced in Sec. Sec.3.4 One convolu- tional block in the first stage of (S=4, r=0.6) is tested. The results Table vali- date that every step of operator fusion benefits the practical latency of block, as the overhead on memory access effectively reduced. Especially, the fusion of the masker operation and the first con- volution is crucial reducing the latency.
Table 1: Ablation studies operator fusion.
Masker- Conv Ixl
Gather- Conv3x3
Scatter- Add
Latency (uS)
X
x
163.2
X
X
90.1
/
X
86.7
/
71.4
4.5 Visualization
We visualize the masks generated by our masker in the third block LAS -eAS-ResNe--101 =4 =4- 4-2-1) in Figure 12, The brilliant areas correspond to the locations of elements in mask, and the computation on the dimmed regions is skipped by our dynamic model. We can witness that the masker accurately locate the most task-related regions (even the tiny aircraft at the corner), which helps reduce the unnecessary computation on background areas. These resultsalso suggest that for the first stage, the granularity S=4 is sufficiently flexible to recognize the important regions, and a win-win can be achieved between accuracy and efficiency. Interestingly, the masker could select some objects that are not labeled for the sample, e.g. the flower beside the hummingbird and the human holding the camera. This indicates that our spatial-wise dynamic networks can automatically recognize the regions with semantics, and their capability is not limited by the classification labels. This property is helpful in some downstream tasks, such as object detection (Sec. (46)) and instance segmentation (Sec. 4.7h. which require detecting multiple classes and objects in image. More visualization results could be found in Appendix C.3
4.6 COCO Object detection
We further evaluate our LASNet on the COCO 22 object detection task. The mean average precision (mAP), the average backbone FLOPs, and the average backbone latency on the validation set are used to measure network performance. We test two commonly used detection frameworks: Faster R-CNN 27 with Feature Pyramid Network [20] and RetinaNet (1 Thanks to the generality of our method, we can conveniently replace the backbones with ours pre-trained on ImageNet, and the whole models are finetuned on COCO with the standard setting for 12 epochs (see detailed

---

Table 2: Object detection results on the COCO dataset.
Detection
Backbone
Backbone FLOPs (G)
Backbone Latency (ms)
mAP (%)
Framework
V100
GTX1080
TX2
ResNet-101 (Baseline)
141.2
39.5
119.3
729.4
39.4
JAS-ResNNt-ID 101 (Sncs -4.4-21,, ta0.6)
90.7
33.8
90.6
$24.9
40.3
Faster R-CNN
LAS-ResNet-100
(5mx=4-4-2-1,,
79.3
30.7
82.5
477.1
39.8
LAS-ResNet-101
79.5
29.0
79.9
464.8
40.0
LAS-ResNet-100
67.9
25.3
69.1
401.3
39.5
ResNet-101 (Baseline)
141.2
39.5
119.3
729.4
38.5
RetinaNet
LAS-ResNet-100 ::4,,
77.8
30.4
472.6
39.3
J.A-RResNet-n (..
79.4
28.9
79.9
464.8
39.3
LAS Res-ResNNet-100 (8 (8m==4-4---1,1al 1=0.4)
66.4
25.3
69.1
401.3
38.9
Table 3: Instance Segmentation results on the COCO dataset.
Segmentation
Backhone
Backbone FLOPs (G)
Backbone Latency (ms)
A A (5r)
Aplex (S)
Framework
V100
GTX1080
TX2
ResNet- 101 (Baseline)
141.2
39.5
119.3
729.4
36.1
40.0
LAS -AS-ResNet1101 (Sac m4.4-2-1, tw0.5)
80.5
31.1
83.3
481.9
37,0
41.0
Mask R-CNN
LAS- LAS-ResNet-101 (5mm==44-2-1, t:44)
69.2
27.9
74.8
431.6
36.1
40.0
ResNet--N11 (...,-.--))))
25.8
70.9
36.2
40.0
setup in Appendix [.3). The input images are resized short side of 800 and long side not exceeding 1333. The results of our LAS- AAS-ResNet-101 101 with different Shet settings are presented in Table We can observe from the results that when setting training target as 0.4, the latency of our LAS-ResNet- 101 with 4--7---4-71 is significantly lower than the static baseline all devices without sacrificing mAP in both detection frameworks. With a larger training targets, our LASNet increase the mAP by 0.9% and 0.8% in Faster R-CNN and RetinaNet respectively, while still being faster than the baseline method.
4.7 COCO instance segmentation
We also present the results of instance segmentation on COCO, which demonstrate the effectiveness of our LASNet on the dense prediction task. From the results in Table we can observe that when setting the training target as 0.4, the Mask R-CNN [10] models (S 44-4-2-1111 and Sue:=4-4-7- runs faster on all tested hardware devices without sacrificing the performance. With a training target of 0.5, the AP pmask and A Abox of the Mask R-CNN model could increased by 0.9% and 1.0% respectively while still running faster than the baseline method.
Conclusion
In this paper, we propose to build latency-aware spatial-wise dynamic networks (LASNet) under the guidance of latency prediction model. By simultaneously considering the algorithm, the scheduling strategy and the hardware properties, we can efficiently estimate the practical latency of spatial- wise dynamic operators on arbitrary computing platforms. Based on the empirical analysis on the relationship between the latency and the granularity of spatially adaptive inference, we optimize both the algorithm and the scheduling strategies to achieve realistic speedup on many multi-core processors, e.g., the Tesla V100 GPU and the Jetson TX2 GPU. Experiments object detection and instance segmentation tasks validate that the proposed method significantly improves the practical efficiency of deep CNNs, and outperforms various competing approaches.
image classification,
Acknowledgement
This work is supported in part by the National Key R&D Program of China under Grant A0001500 the National Natural Science Foundation of China under Grants 62022048 and the Tsinghua University-Chie China Mobile Communications Group Ltd, Joint Institute..
10

---

References
[1] Babak Ehteshami Bejnordi, Tijmen Blankevoort, and Max Welling. Batch-shaping for learning conditional channel gated networks. In ICLR, 2020.
[2] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. In OSDI, 2018.
[3] Steven Colleman, Thomas Verelst, Linyan Mei, Tinne Tuytelaars, and Marian Verhelst. Proces- sor architecture optimization for spatially dynamic neural networks. In VLSI-SoC, 2021.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[5] Xuanyi Dong, Junshi Huang, Yi Yang. and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In CVPR, 2017.
[6] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In CVPR, 2017.
[7] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. TPAMI, 2021.
[8] Yizeng Han, Gao Huang, Shiji Song. Le Yang. Yitian Zhang, and Haojun Jiang. Spatially adaptive feature refinement for efficient inference. TIP, 2021.
[9] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfen Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic carly-exiting networks. In ECCV, 2022.
[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
[13] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0480 04100, 2017.
[14] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exxitation networks. In CVPR, 2018.
[15] Gao Huang. Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Wein- berger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018.
[16] Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using leameed group convolutions. In CVPR, 2018.
[17] Gao Huang, Zhuang Liu, Geoff Pleiss, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. TPAMI, 2019.
[18] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel- In ICLR, 2017.
[19] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS, 2017.
[20] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[21] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.

---

[22] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[23] Ningning Ma, Xiangyu Zhang, Hai-Tao Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018.
[24] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.
[25] Iljaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, 2020.
[26] Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. SBNet: Sparse Blocks Network for Fast Inference. CVPR, 2018.
[27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[29] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-awart neural architecture search for mobile. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[30] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In ECCV, 2018.
[31] Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting Spatial Sparsity for Faster Inference. CVPR, 2020.
[32] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV, 2018
[33] Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification. In NeurIPS, 2020.
[34] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[35] Zhenda Xie, Zheng Zhang, Xizhou Zhu, Gao Huang, and Stephen Lin. Spatially adaptive inference with stochastic feature sampling and interpolation. In ECCV, 2020.
[36] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In CVPR, 2020.
[37] Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang. Ying Nie, Zhenzhong Lan, and Yunhe Wang. Dynamic resolution network. NeurlPS, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A]. . You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 4.T and Section 4.6
the public ImageNet and COCO datasets are cited.
12

---

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix D
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix D
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code. data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL.)? [Yes] See the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Most important hyperparameters are given in the main paper, and more details are presented in the supplementary material (Appendix B).
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] Since the experiment results the large-scale dataset, ImageNet and COCO, are generally stable, we perform several repeat experiments in the early exploring phase, and dismiss the error bars in main results as in most previous works.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The details are presented in the supplementary material (Appendix B).
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license the assets? [Yes] See Section 4.1 We use the torchvision pre-trained models.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yGC04AvrBF4", "reviewer": "Reviewer_3HuP", "summary": "Paper tackles the problem of dynamic inference where the forward graph will depend on the input data. It focuses on real speedups instead  of theoretical one, this makes the work more valuable. There is a latency prediction model that can estimate the latency by considering the algorithms, scheduling and hardware properties. Experiments are performed on image classification and demonstrate latency reduction of 23 to 45% depending on the hardware superiority. ", "questions": "- How does hardware parameter vector H look like?\n- The activation factor $r$, is it preselected or computed automatically? What would be con\\pros of both solutions?\n- What is the architecture of masker? \n- Is the mask the same for all input/output channels? Or it does vary? \n- Why the case of S=56 is not considered? If my understanding is correct then the entire feature map will be removed which is helpful to reduce latency. \n- Explain how sparsity is set of different layers and why S is different for different models and layers in section 4.3.1 \n- Implementation of the 2 branch model in Figure 2 is not clear. If the mask is adaptive then there might be more than 1 candidate with sparse convolution, how would Gumbel softmax sample them? How M is regularized? \n- How is latency computed in Figure 7? Was the setup the same (operator fusion etc) for all models considered? Those questions raise because skipping the entire block as in Convolutional-AIG should be more efficient in terms of latency reduction. \n- Where was the framework implemented? If it was C++ then comparing with cuDNN and TesnorRT would be valuable as there is additional model optimization. \n- Will the code be released? The latency prediction model might be useful to community for future research. \n\nMinor:\n\n- In the first section, the coarse granularity is mentioned. It would be helpful to explain it once it appears in the text.  ", "limitations": "- Limitations are not listed by authors\n- There are seem to be a possible a set of hyper-parameters that need to be tuned like S etc. \n- Some details are missing and clarifying them will be helpful.\n- It is not clear how different convolution implementations (like Winograd) will benefit from the work.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ Dynamic inference is a field of great interest. Naturally, people spent different effort to perform vision tasks so should deep network. \n+ The method considers the real latency and not FLOPs. Additionally, demonstrating real speed-up is greatly appreciated. \n+ Multiple paths are considered during training, and a single path is executed during inference.  \n+ Pare is well written and the content comes smoothly. \n\nWeaknesses:\n\n- Using distillation during fine-tuning will increase accuracy and might be not fair comparing to the original model.\n- When compared to the previous work like Conv-AIG, the training recipes might be different (authors start from trained model and do 100 more epochs). The over training pipeline is not described as well, things like augmentations, learning rate scheduler etc should be mentioned. If those are not the same (as number of epochs) then authors should perform a fait comparison with those being the same. \n- There are questions in the section before that are not clear. Understanding them will help to evaluate paper better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "OG1T3DO0pWf", "reviewer": "Reviewer_MLaG", "summary": "This submission introduces a spatially dynamic neural network approach, developed in a latency-aware manner that can generate realistic speed-ups during inference. The proposed methodology allows to dynamically skip computation at different spatial regions of the feature maps at a coarser granularity (of blocks of pixels), directly generalising the widely studied pixel-level approaches from the literature. This leads to more regular computation pattern and reduces the overheads, to maximise the attainable inference speed gains. Additionally, the granularity of the pixel blocks is tuned in a latency-aware manner to balance the speed-accuracy trade-off. Results indicate that the proposed approach is effectively improving efficiency, across hardware platforms and models for image classification and object detection tasks.", "questions": "1. How is the latency predictor implemented? Is it trainable, or based on an analytical performance model/ cycle accurate simulator? Can it capture the effect of more complex operations, such as dilated or separable convs, skip connections etc?\n2. The case where a large S can emulate layer skipping should be considered in the ablation.\n3. How would the proposed approach be applied to dense tasks, such as semantic segmentation, dense prediction etc ?\n4. Is there any benefit on allowing overlapping pixel blocks to be considered during masking? Qualitative results indicate that such a setting may allow for larger values of S. ", "limitations": "Some limitations are discussed in the conclusion section. The authors are encouraged to enhance this discussion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a very important real-world issue, which is well-motivated and demonstrated experimentally.\n- The proposed approach, although rather simple, is able to generate realistic speed-ups with no effect in accuracy, showcasing improved efficiency compared to baselines.\n- Useful insights are provided in the experiments analysis.\n\nWeaknesses (see questions for more details):\n- The novelty of the proposed approach is limited, as it directly generalises a widely studied problem, going from pixel-level to block-level. Similar generalisation in the context of early-exit networks has recently been studied in the literature:\n  - Liu, Zhuang, et al. \"Anytime Dense Prediction with Confidence Adaptivity.\" International Conference on Learning Representations. 2022.\n-  The description of the latency prediction model (Sec 3.3) is very high-level, making it hard to justify its contribution . \n- Some interesting aspects of this work (eg the generic formulation combining spatial computation skipping and layer skipping - Sec 3.2) are only theoretically mentioned, but not experimentally studied.\n- Some of the claimed contributions and ablation studies are mostly comprise technical/implementation aspects (Sec 3.4).\n- Evaluation is focused on Classification/detection, but it is unclear how the proposed methodology can be applied to more dense CV tasks, such as instance/semantic segmentation. The reviewer believes that this mapping won't be intuitive and this should be discussed in the limitations of the proposed approach.\n\nPresentation\n- Syntax/grammatical errors exist sporadically across the manuscript, mostly in Sec1, 3.3 and 4.5\n- Some aspects of the proposed methodology are described in rather high-level, making it impossible to reproduce.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E5r8zz_pc-y", "reviewer": "Reviewer_BmLK", "summary": "This paper presents a dynamic inference paradigm based on selective inference of convolutions on the spatial dimension. Specifically, for each convolutional block, it uses a masker layer to predict the masked region of the feature map and only conduct convolution on the masked region. Such convolution has less computation than a full convolution and improves efficiency of the model. Moreover, it utilizes a latency prediction model to estimate the latency of different mask operations. With a differentiable training framework, the model can be optimized according to configurations different hardware platform.", "questions": "See above.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n-\tAlthough masked inference is not a new idea, this work targets at a more practical inference speed optimization compared to existing dynamic-masking approaches. Several critical modifications for the real efficiency of this approach are adopted in the inference pipeline: fused operations, granularity choices of the mask etc.\n-\tExperiments with ResNet and RegNet validates the effectiveness of this method on several tasks: ImageNet classification and object detection. It reduces inference time by 23% and 45% on V100 and TX2 platform respectively without significant drop on accuracy.\nWeaknesses:\n-\tThe framework can only be applied to models with ResNet block whose major computations are located in the 3x3 conv of the network blocks. I don’t think it work well for other efficient models such as MobileNet and ShuffleNet that does not have heavy 3x3 convs.\n-\tOnly GPU platforms are tested. Does this approach generalize to other types of hardware such as CPU? \n-\tFor the speed comparison, what kind of cuda runtime environment are you using for the experiments? What are the cuda version, cudnn version? are you using TensorRT? These details are not given in the main text.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept