Dynamic Resolution Network 

1 Introduction . Deep convolutional neural networks ( CNNs ) have achieved remarkable success in various computer vision tasks , under the development of algorithms [ 6 , 22 , 37 ] , computation power , and large-scale datasets [ 2 , 17 ] . However , the outstanding performance is accompanied by large computational costs , which makes CNNs difficult to deploy on mobile devices . With the increasing demand for CNNs on real-world applications , it is imperative to reduce the computational cost and meanwhile maintain the performance of neural networks . Recently , researchers have devoted much effort to model compression and acceleration methods , including network pruning , low-bit quantization , knowledge distillation , and efficient model design . ∗Equal contribution . † Corresponding author . This research has been supported by the Key R & D program of Zhejiang Province ( Grant No . 2021C03139 ) . This work was supported by NSFC ( 62072449 , 61632003 ) , Guangdong-Hongkong-Macao Joint Research Grant ( 2020B1515130004 ) , and Macao FDCT ( 0018/2019/ AKP ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Network pruning aims to prune the unimportant filters or blocks that are insensitive to model performance through a certain criterion [ 31 , 15 , 18 , 25 ] . Low-bit quantization methods represent weights and activation in neural networks with low-bit values [ 11 , 13 ] . Knowledge distillation transfers the knowledge of the teacher models to the student models to improve the performance [ 7 , 38 , 40 ] . The efficient model design utilizes lightweight operations like depth-wise convolution to construct some novel architectures successfully [ 8 , 42 , 5 ] . Orthogonal to those methods that usually focus on the network weights or architectures , Guo et.al . [ 4 ] and Wang et.al . [ 33 ] study the redundacy that exists in the input images . However , the resolutions of the input images in most of existing compressed networks are still fixed . Although deep networks are often trained using an uniform resolution ( e.g. , 224× 224 on the ImageNet ) , sizes and locations of objects in images are radically different . Figure 1 shows some samples that the required resolution for achieving the highest performance are different . For the given network architecture , the FLOPs ( floating-number operations ) of the network for processing image will be significantly reduced for images with lower resolution . panda bee damselfly R eso lu tio n lo w h igh understood easily just in blurry mode while the others need to be seen in clear mode . In this paper , we propose a novel dynamic-resolution network ( DRNet ) which dynamically adjusts the input resolution of each sample for efficient inference . To accurately find the required minimum resolution of each image , we introduce a resolution predictor which is embedded in front of the entire network . In practice , we set several different resolutions as candidates and feed the image into the resolution predictor to produce a probability distribution over candidate resolutions as the output . The network architecture of the resolution predictor is carefully designed with negligible computational complexity and trained jointly with classifier for recognition in an end-to-end fashion . By exploiting the proposed dynamic resolution network inference approach , we can excavate the reduncancy of each image from its input resolution . Thus , computational costs of easy samples with lower resolutions can be saved , and the accuracy for hard samples can also be preserved by maintaining higher resolutions . Extensive experiments on the large-scale visual benchmarks and the conventional ResNet architectures demonstrate the effectiveness of our proposed method for reducing the overall computational costs with comparable network performance . 2 Related Works . Although deep CNN models have shown excellent accuracy , they often contain millions of parameters and FLOPs . Thus the model compression techniques are becoming a research hostpot for reducing the computational costs of CNNs . Here , we revisit existing works in two parts , i.e. , static model compression and dynamic model compression . We classify model compression methods that are not instance-aware as the static . Group-wise Convolution ( GWC ) , Depth-wise Convolution and Point-wise Convolution are widely used for efficient model design , such as MobileNet [ 8 ] , ResNeXt [ 36 ] , and ShuffleNet [ 42 ] . Revealing pattern redundancy among feature maps , Han [ 5 ] proposes to generate more feature maps from intrinsic ones through some cheap operations and Zhang et.al . [ 41 ] adopts relatively heavy computation to extract intrinsic information while tiny hidden details are processed with some light-weight operations . The methods above achieve model acceleration to some extent . However , they treat all input samples equally , whereas the difficulty for CNN models or humans to recognize each sample is unequal . So instance-aware model compression can be further explored . Dynamic model compression takes the unequal difficulty of each sample into consideration . Huang [ 9 ] proposes multi-scale dense networks with multiple classifiers to allocate uneven computation across `` easier '' and `` harder '' inputs . Wu et.al . [ 35 ] introduces BlockDrop which learns to dynamically execute the necessary layers so as to best reduce total computation . Veit et.al . [ 28 ] proposes ConvNetAIG to adaptively define their network topology conditioned on the input image . Except for dynamic adjustment on model architectures , recent works pay more attention to input images . Verelst et.al . [ 29 ] proposes a small gating network to predict pixel-wise masks determining the locations where dynamic convolutions are evaluated . Uzkent et.al . [ 27 ] proposes PatchDrop to dynamically identify when and where to use high-resolution data conditioned on the paired low-resolution images . Yang et.al . [ 39 ] dynamically utilize sub-networks of the base network to process images with different resolutions . Wang et.al . [ 33 ] proposes GFNet with patch proposal networks that strategically crop the minimal image regions to obtain reliable predictions . There exists many works on multiple resolutions and dynamic mechanisms . ELASTIC [ 30 ] uses different scaling policies for different instances and it learns from the data how to select the best policy . Hydranets [ 20 ] chooses different branches for different inputs by a gate and aggregates their outputs with a combiner . RS-Net [ 32 ] utilizes private BNs , shared convolutions , and fully-connected layer and to train input images with different resolutions . Different from dynamic adjustment on model architectures and dynamic modification on input images with reinforcement learning , we consider the whole image and propose a resolution predictor to dynamically choose the performance-sufficient and cost-efficient resolution for a single model to obtain reliable prediction with end-to-end training . 3 Approach . In this section , we first introduce the overall framework of the proposed Dynamic Resolution Network ( DRNet ) , then describe the resolution predictor , resolution-aware BN , and optimization algorithm in detail , respectively . 3.1 Dynamic Resolution Network . Inspired by the fact that different sample requires different resolution to achieve the least accurate prediction , we propose to develop an instance-aware resolution selection approach for a single large classifier network . As shown in Figure 2 , the proposed method mainly consists of two components . The first is the large classifier network with both high performance and expensive computational costs , such as the classical ResNet [ 6 ] and EfficientNet [ 24 ] . The other is a resolution predictor for finding the minimal resolution so that we can adjust the input resolution of each image to have a better trade-off on the accuracy and efficiency . For an arbitrary input image , we first forecast its suitable resolution r using the resolution predictor . Then , the large classifier will take the resized image as inputs and the required FLOPs will be reduced significantly when r is lower than that of the origional resolution . To achieve better performance , the resolution predictor and the base network are optimized end-to-end during training . Resolution Predictor . The resolution predictor is designed as a CNN-based preprocessing operation before input samples are fed to the large base network . It ’ s from the inspiration that our well-trained large models can also predict a relative amount of samples correctly though they are in relatively small resolutions while large amounts of computation cost can be saved . On the one hand , the goal of the resolution predictor is to find an appropriate instance-aware resolution by inferring a probability distribution over candidate resolutions . Note that there are a vast number of candidates from 1×1 to 224×224 , which makes it difficult , also meaningless , for the resolution predictor to explore such a long-range of resolutions . As a simplification strategy and practical requirement , we choose m resolution candidates r1 , r2 , · · · , rm to shrink the exploration range . On the other hand , we have to keep the model size of the proposed resolution predictor as small as possible since it will bring extra FLOPs , otherwise , it becomes impractical to implement such a module if its extra introduced computation exceeds the saved one from the low resolution . In this spirit , we design the resolution predictor with a few convolutional layers and fully-connected layers to complete a resolution classification task . Then the preprocessing of the proposed resolution predictor R ( · ) can be given as follows : pr = [ pr1 , pr2 , ... , prm ] = R ( X ) , ( 1 ) where X is the input samples fed to the resolution predictor , m is the total number of candidate resolutions , and pr ∈ Rm is the outputs of the resolution predictor which represents the probability of each candidate . Then the resolution corresponding to the highest probability entry is selected as the resolution fed to the large classifier . Since the process from the soft outputs of resolution predictor to the discrete resolution resizing operation does not support end-to-end training , here we adopt Gumbel-Softmax [ 14 ] module G to turn soft decisions pr into hard decisions h ∈ { 0 , 1 } m by applying Gumbel-Softmax trick to solve the non-differentiable problem : h = G ( pr ) = G ( R ( X ) ) , ( 2 ) where the Gumbel-Softmax trick will be described in the next subsection . For validation , the resolution predictor makes decisions first then the input with the selected resolution only is fed to the large classifier in the normal way as shown in the left part of Figure 2 . Resolution-aware BN . Our framework is proposed to use only a single large classifier for the sake of storage pressure and loading latency , which results in that the single classifier has to process multi-resolution inputs and raises two problems . One obvious problem is that the first fully-connected layer will fail to work with a different input resolution and can be solved with global average pooling . Thus we can process multiple resolutions in one single network . The other hidden problem exists in Batch Normalization ( BN ) [ 12 ] layers . BN is used to make deep models converge faster and more stable through channel-wise normalization of the input layer by re-centering and re-scaling . However , activation statistics including means and variances under different resolutions are incompatible [ 26 ] . Using shared BNs under multiple resolutions leads to lower accuracy in our experiments as shown in section 4.3 . Since the batch normalization layer contains a negligible amount of parameters , we propose resolution-aware BNs as shown in Figure 2 . We decouple the BN for each resolution and choose the corresponding BN layer to normalize the features : xj = γj xj − µj√ σj2 + + βj , j ∈ { 1 , 2 , ... , m } , ( 3 ) where is a small number for numerical stability , µi and σi are private averaged mean and variance from the activation statistics under separate resolutions ; βi and γi are private learnable scale weights and bias . Since shared convolutions are insensitive to performance , the overall adjustment for the original large classifier is shown in the right part of Figure 2 .

Summary: The paper presents a method to reduce FLOP requirements by resizing images dynamically using a resolution prediction network, prepended to an image classifier network.  The resolution is selected from one of a handful of candidates, and resized before being passed to the classifier.  A training-time loss encourages smaller resolutions to be selected, effecting an accuracy vs resolution tradeoff controlled by two parameters.  Since selected resolution for inference is a one-hot categorical, the method uses Gumbel softmax to enable backpropagation during training.  A resolution-aware batchnorm layer is also described, which tracks batchnorm stats separately for each resolution.  The system performs very well, with over 40% reduction in average operations resulting in just one or two percent accuracy loss on imagenet, and small accuracy gains with 10% reduction in computation.